What is Steampipe?

Steampipe is open source software for interrogating your cloud. Run SQL queries, compliance controls and full governance benchmarks from the comfort of your CLI.

Steampipe’s codified operations framework gives you the power to test your cloud resources against security, compliance and cost benchmarks, and to build your own custom control frameworks.

Our multi-threaded Golang CLI makes your custom SQL controls blazing fast with unlimited integration options via our embedded PostgreSQL database.

{` select region, instance_state as state, instance_type as type from aws_ec2_instance; `} {` +-----------+---------+-----------+ | region | state | type | +-----------+---------+-----------+ | eu-west-1 | running | t3.medium | | eu-west-2 | running | m5a.large | | us-east-1 | running | t3.large | +-----------+---------+-----------+ `}

## tl;dr ** → ** **[Connection aggregators](#connection-aggregators)** for AWS multi-account & more.
** → ** AWS **[multi-region wildcard](#multi-region-wildcards)** support.
** → ** Steampipe **[support for Docker](#docker-support)**.
** → ** **[Fast startup](#fast-startup-and-smoother-services)** and smoother services.
** → ** **[7 new plugins and mods](#new-plugins-and-mods)**.
** → ** **[Join our new Slack Channel](https://steampipe.io/community/join)**!
** → ** Even more goodies in the [full release notes](https://github.com/turbot/steampipe/blob/main/CHANGELOG.md#v070-2021-07-22).
## Connection Aggregators Connection aggregators are a simple new way to combine results from multiple connections into a single schema. With this multi-account setup for AWS: ```hcl connection "all_dmi" { plugin = "aws" type = "aggregator" connections = ["dmi*"] } connection "dmi_prod" { plugin = "aws" profile = "profile1" regions = ["us-east-1", "us-east-2"] } connection "dmi_dev" { plugin = "aws" profile = "profile2" regions = ["us-west-1", "us-west-2"] } ``` We can run a single query to instantly combine results from all accounts and regions: ```sql select title, instance_type, region, account_id from all_dmi.aws_ec2_instance; ``` ```text +-----------------+---------------+-----------+--------------+ | title | instance_type | region | account_id | +-----------------+---------------+-----------+--------------+ | DMI IT Bastion | t3.large | us-east-1 | 111111111111 | | DMI Sales DB | m5.xlarge | us-east-2 | 111111111111 | | DMI ActiveDir | m5.large | us-west-1 | 222222222222 | | wufph.com | m3.medium | us-west-2 | 222222222222 | +-----------------+---------------+-----------+--------------+ ``` Aggregator connections works like all the connections you are used to, so can be used immediately to run benchmarks and controls across multiple accounts!

### Multi-region wildcards Accompanying the new multi-connection support in Steampipe, the AWS mod has also added support for using wildcards in the region field of the AWS connection configuration. Previously, multiple regions would need to be specified individually, which can be unwieldly in the configuration file if teams want to support all regions. Using wildcards greatly simplifies the configuration and allows for graceful addition of new regions without need for changing your config. Here are a few examples: ```hcl # All US regions connection "dmi_us" { plugin = "aws" profile = "default" regions = ["us-*"] } # All EMEA connection "dmi_emea" { plugin = "aws" profile = "default" regions = ["eu-*", "af-*", "me-*"] } # All Regions connection "dmi_global" { plugin = "aws" profile = "default" regions = ["*"] } ``` ### More than just AWS While AWS multi-account is a common case, connection aggregators work with any plugin type so you can easily query across credential boundaries. Maybe you ended up with multiple GitHub accounts for work and personal projects and want to query them together: ```hcl connection "gh" { plugin = "github" type = "aggregator" connections = ["dwight_dmi", "schrute_farms"] } connection "dwight_dmi" { plugin = "github" token = "30a98976e098798c9807609876c789675a" } connection "scrute_farms" { plugin = "github" profile = "30a4242424d8c983076093876c7238942c" } ``` ### (Un)Paralleled Performance The quality of life change to allow for aggregate connections & queries is cool in itself, but the developers didn’t stop there. Steampipe’s FDW layer is designed to run all these queries in parallel, so when you execute an aggregate query, Steampipe uses its multi-threaded core to begin fetching results from all connections in parallel and immediately streaming back the results. In practice multi-connection queries will have the same performance as the slowest single connection in the group.

Caching behavior is just as smart. Running an aggregate query pre-loads the cache for subsequent multi-connection queries and for each of the individual connections as well. ## Fast startup and smoother services "Time to first query" is something we focus on a lot. Too many people are stuck building warehouses, or installing databases, etc when all they want to do is run a query. That's why Steampipe automatically detects your credentials when it can and works with any data source with no synchronization steps required. Startup is faster in this release, noticably faster. We've worked hard to drop you into the SQL prompt as fast as possible. Other service improvements in this release: - Run multiple `steampipe query` terminals in parallel. - Improved cancellation handling for queries you regret. ## Docker Support Once you get the feel for running queries interactively with Steampipe it is a natural progression to want to automate queries that you run frequently or share preconfigured workspaces with other members of your team. Containers are a great way to isolate applications from the environment they run in, giving you the ultimate flexibility to consistently run Steampipe on your desktop or in your favorite cloud. While some members of the community have had success getting Steampipe to run in Docker, we wanted to implement some core changes to make use of Steampipe in containerized environments more elegant, as we added support to query Docker container configuration. So, we've published an image to [Docker Hub](https://hub.docker.com/r/turbot/steampipe)! ```bash docker run -it --rm --name steampipe turbot/steampipe ``` **[Using Containers →](/docs/develop/containers)** Because of the Steampipe configuration, embedded database, etc there are a number of different scenarios for using Steampipe in containers. Please check out our [Using Containers doc](/docs/develop/containers) for all the details and share your experiences in the [Steampipe Slack Community](https://steampipe.io/community/join). Psst: We've also published a [Docker plugin](https://hub.steampipe.io/plugins/turbot/docker)! ## New plugins and mods Beyond the CLI, since [our last release](/blog/release-0-6-0), we've added 6 new plugins: - [AbuseIPDB](https://hub.steampipe.io/plugins/turbot/abuseipdb) - query IP address security data - [Bitbucket](https://hub.steampipe.io/plugins/turbot/bitbucket) - query projects, issues, branches, etc - [Docker](https://hub.steampipe.io/plugins/turbot/docker) - query Dockerfile data, containers, etc - [Stripe](https://hub.steampipe.io/plugins/turbot/stripe) - query customers, invoices, etc - [urlscan.io](https://hub.steampipe.io/plugins/turbot/urlscan) - query website metadata - [VirusTotal](https://hub.steampipe.io/plugins/turbot/virustotal) - query virus signatures We've also expanded our mod coverage, including: - [AWS Compliance mod](https://hub.steampipe.io/mods/turbot/aws_compliance) controls for [NIST CSF](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/benchmark.nist_csf) and [NIST 800-53](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/benchmark.nist_800_53_rev_4) - [GCP Thrifty mod](https://hub.steampipe.io/mods/turbot/gcp_thrifty) — Cost/usage checking ## Let’s get building! Steampipe now delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST… and the best way to get started is to [join our new Slack workspace](https://steampipe.io/community/join) and raise your hand; we would love to talk to you! For even more good stuff in v0.7.0, checkout the [full release notes on GitHub](https://github.com/turbot/steampipe/blob/main/CHANGELOG.md#v070-2021-07-22).

v0.7.0: AWS multi-account queries & Docker support