v0.17.0: Snapshots, share with cloud, workspace profiles →
Announcement

v0.16.0: Major memory reduction, new benchmarks, secure AWS regions

Discover the great new features in Steampipe's open source v0.16.0 release!

Steampipe Team
6 min. read - September 1, 2022

tl;dr

Major memory reduction
New AWS Compliance benchmarks
New plugins
New mods
Secure AWS regions
Live connection refresh
net_http_request
Even more goodies in the full release notes.

Major memory reduction

Users and organizations are embracing Steampipe for increasingly large environments, hundreds of connections (e.g. AWS accounts) is now common. This is more than we ever imagined possible, so for v0.16 we challenged ourselves to scale much further through better memory management.

Steampipe now launches only one process per plugin version, instead of one process per connection. So if you have 100 AWS accounts, they'll now share a single plugin process. That makes startup much faster and reduces memory overhead.

Streaming thousands of results back from those connections would previously grow temporary memory to uncomfortable levels. It was released back to the OS (of course), but created unnecessary pressure while running. We now set and reuse a reasonable size memory buffer while streaming, capping our memory growth regardless of result size.

Finally, the in-memory query result cache now supports an optional STEAMPIPE_MAX_CACHE_SIZE environment variable. Set it as a simple way to control the memory used by the query cache.

The impact of these changes is dramatic.

Here's a comparison of plugin memory used by v0.15 vs v0.16 for tests that stream 5,000 rows per connection:

v16 memory improvements for 5k rows

On a similar chart, for 100,000 rows per connection, there's no v0.15 data for the 50- and 100-connection tests. That's because those tests were unable to complete in v0.15!

v16 memory improvements for 100k rows

Time to first row is now much faster, especially when working with many connections:

v16 query time improvements

If you're pushing the limits on Steampipe's scalability, this new architecture creates plenty of headroom to expand.

New AWS Compliance benchmarks

The AWS Compliance mod gained 5 new benchmarks during this cycle!

And the AWS Foundational Security Best Practices benchmark added 32 new controls.

On the Azure front, Azure Compliance added the CIS 1.4 benchmark.

New plugins

Since our last release, we've added 5 new plugins.

  • crt.sh - query certificates, issuers, logs, etc

  • CrowdStrike - query detection events, hosts, users, vulnerabilities, etc.

  • FreshService - query agents, assets, tickets, etc

  • Trivy - query scan artifacts, secrets, vulnerabilities, etc.

  • Vercel - query DNS records, domains, projects, teams, etc.

Thanks to Ciaran Finnegan for help with CrowdStrike, and Graza Andersson for contributing FreshService.

During this cycle we added 14 new tables to the AWS plugin. And the GitHub plugin acquired one new table, github_audit_log, thanks to Jelmer Borst who added the table and Narayan Gowraj who tested it.

New mods

  • AWS Perimeter - check across all your AWS accounts for resources that are insecurely configured, publicly accessible, or shared with untrusted accounts. The mod comprises 10 benchmarks and 52 controls.

  • Hacker News Insights - see which languages, companies, databases, and clouds are popular right now on Hacker News.

  • GitHub Compliance - run the CIS Supply Chain Security compliance benchmark to check source code, build pipelines, dependencies, artifacts, and deployment.

Secure AWS regions

Steampipe now runs in the AWS C2S and SC2S regions. That means users in the U.S. Department of Defense, and in the intelligence community, can now run queries, benchmarks, and dashboards against Top Secret and Secret workloads on these airgapped networks.

Live connection refresh

When you revise an aggregator connection in a plugin config file, or update expired credentials, Steampipe now auto-detects the changes and instantly applies them.

net_http_request

With the addition of the net_http_request table, the Net plugin now enables Steampipe to work as an HTTP client. Queries can use GET or POST, specify HTTP headers, and send POST requests with headers and body.

Let’s get building!

Steampipe delivers tools to build, execute, and visualize and share cloud configuration, compliance, and security frameworks using HCL + SQL and a little elbow grease. To support those tools, it maps a growing suite of APIs to tables that you can query, and join across, in Postgres.

Do you want to help us expand the open source documentation and control coverage for CIS, PCI, HIPAA, and NIST? Add tables to existing plugins? Create plugins to bring new APIs into the mix? Create dashboards to visualize those APIs? The best way to get started is to join our Slack workspace and raise your hand. We would love to talk to you!

For even more good stuff in v0.16.0, check out the full release notes on GitHub.