v0.7.0: AWS multi-account queries & Docker support

select
  region,
  instance_state as state,
  instance_type as type
from
  aws_ec2_instance;

+-----------+---------+-----------+
| region    | state   | type      |
+-----------+---------+-----------+
| eu-west-1 | running | t3.medium |
| eu-west-2 | running | m5a.large |
| us-east-1 | running | t3.large  |
+-----------+---------+-----------+
        

tl;dr

→ Connection aggregators for AWS multi-account & more.
→ AWS multi-region wildcard support.
→ Steampipe support for Docker.
→ Fast startup and smoother services.
→ 7 new plugins and mods.
→ Join our new Slack Channel!
→ Even more goodies in the full release notes.

Connection Aggregators

Connection aggregators are a simple new way to combine results from multiple connections into a single schema. With this multi-account setup for AWS:

connection "all_dmi" {
  plugin      = "aws"
  type        = "aggregator"
  connections = ["dmi*"]
}
 
connection "dmi_prod" {
  plugin      = "aws"
  profile     = "profile1"
  regions     = ["us-east-1", "us-east-2"]
}
 
connection "dmi_dev" {
  plugin      = "aws"
  profile     = "profile2"
  regions     = ["us-west-1", "us-west-2"]
}

We can run a single query to instantly combine results from all accounts and regions:

select
  title,
  instance_type,
  region,
  account_id
from
  all_dmi.aws_ec2_instance;

+-----------------+---------------+-----------+--------------+
| title           | instance_type | region    | account_id   |
+-----------------+---------------+-----------+--------------+
| DMI IT Bastion  | t3.large      | us-east-1 | 111111111111 |
| DMI Sales DB    | m5.xlarge     | us-east-2 | 111111111111 |
| DMI ActiveDir   | m5.large      | us-west-1 | 222222222222 |
| wufph.com       | m3.medium     | us-west-2 | 222222222222 |
+-----------------+---------------+-----------+--------------+

Aggregator connections works like all the connections you are used to, so can be used immediately to run benchmarks and controls across multiple accounts!

Multi-region wildcards

Accompanying the new multi-connection support in Steampipe, the AWS mod has also added support for using wildcards in the region field of the AWS connection configuration. Previously, multiple regions would need to be specified individually, which can be unwieldly in the configuration file if teams want to support all regions. Using wildcards greatly simplifies the configuration and allows for graceful addition of new regions without need for changing your config. Here are a few examples:

# All US regions
connection "dmi_us" {
  plugin  = "aws"
  profile = "default"
  regions = ["us-*"]
}
 
# All EMEA
connection "dmi_emea" {
  plugin  = "aws"
  profile = "default"
  regions = ["eu-*", "af-*", "me-*"]
}
 
# All Regions
connection "dmi_global" {
  plugin  = "aws"
  profile = "default"
  regions = ["*"]
}

More than just AWS

While AWS multi-account is a common case, connection aggregators work with any plugin type so you can easily query across credential boundaries. Maybe you ended up with multiple GitHub accounts for work and personal projects and want to query them together:

connection "gh" {
  plugin      = "github"
  type        = "aggregator"
  connections = ["dwight_dmi", "schrute_farms"]
}
 
connection "dwight_dmi" {
  plugin      = "github"
  token       = "30a98976e098798c9807609876c789675a"
}
 
connection "scrute_farms" {
  plugin      = "github"
  profile     = "30a4242424d8c983076093876c7238942c"
}

(Un)Paralleled Performance

The quality of life change to allow for aggregate connections & queries is cool in itself, but the developers didn’t stop there. Steampipe’s FDW layer is designed to run all these queries in parallel, so when you execute an aggregate query, Steampipe uses its multi-threaded core to begin fetching results from all connections in parallel and immediately streaming back the results. In practice multi-connection queries will have the same performance as the slowest single connection in the group.

Caching behavior is just as smart. Running an aggregate query pre-loads the cache for subsequent multi-connection queries and for each of the individual connections as well.

Fast startup and smoother services

"Time to first query" is something we focus on a lot. Too many people are stuck building warehouses, or installing databases, etc when all they want to do is run a query. That's why Steampipe automatically detects your credentials when it can and works with any data source with no synchronization steps required.

Startup is faster in this release, noticably faster. We've worked hard to drop you into the SQL prompt as fast as possible.

Other service improvements in this release:

• Run multiple steampipe query terminals in parallel.
• Improved cancellation handling for queries you regret.

Docker Support

Once you get the feel for running queries interactively with Steampipe it is a natural progression to want to automate queries that you run frequently or share preconfigured workspaces with other members of your team. Containers are a great way to isolate applications from the environment they run in, giving you the ultimate flexibility to consistently run Steampipe on your desktop or in your favorite cloud.

While some members of the community have had success getting Steampipe to run in Docker, we wanted to implement some core changes to make use of Steampipe in containerized environments more elegant, as we added support to query Docker container configuration.

So, we've published an image to Docker Hub!

docker run -it --rm --name steampipe turbot/steampipe

Using Containers →

Because of the Steampipe configuration, embedded database, etc there are a number of different scenarios for using Steampipe in containers. Please check out our Using Containers doc for all the details and share your experiences in the Steampipe Slack Community.

Psst: We've also published a Docker plugin!

New plugins and mods

Beyond the CLI, since our last release, we've added 6 new plugins:

• AbuseIPDB - query IP address security data
• Bitbucket - query projects, issues, branches, etc
• Docker - query Dockerfile data, containers, etc
• Stripe - query customers, invoices, etc
• urlscan.io - query website metadata
• VirusTotal - query virus signatures

We've also expanded our mod coverage, including:

• AWS Compliance mod controls for NIST CSF and NIST 800-53
• GCP Thrifty mod — Cost/usage checking

Let’s get building!

Steampipe now delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST… and the best way to get started is to join our new Slack workspace and raise your hand; we would love to talk to you!

For even more good stuff in v0.7.0, checkout the full release notes on GitHub.