tl;dr
→ Major memory reduction
→ New AWS Compliance benchmarks
→ New plugins
→ New mods
→ Secure AWS regions
→ Live connection refresh
→ net_http_request
→ Even more goodies in the full release notes.
Major memory reduction
Users and organizations are embracing Steampipe for increasingly large environments, hundreds of connections (e.g. AWS accounts) is now common. This is more than we ever imagined possible, so for v0.16 we challenged ourselves to scale much further through better memory management.
Steampipe now launches only one process per plugin version, instead of one process per connection. So if you have 100 AWS accounts, they'll now share a single plugin process. That makes startup much faster and reduces memory overhead.
Streaming thousands of results back from those connections would previously grow temporary memory to uncomfortable levels. It was released back to the OS (of course), but created unnecessary pressure while running. We now set and reuse a reasonable size memory buffer while streaming, capping our memory growth regardless of result size.
Finally, the in-memory query result cache now supports an optional STEAMPIPE_MAX_CACHE_SIZE environment variable. Set it as a simple way to control the memory used by the query cache.
The impact of these changes is dramatic.
Here's a comparison of plugin memory used by v0.15 vs v0.16 for tests that stream 5,000 rows per connection:
On a similar chart, for 100,000 rows per connection, there's no v0.15 data for the 50- and 100-connection tests. That's because those tests were unable to complete in v0.15!
Time to first row is now much faster, especially when working with many connections:
If you're pushing the limits on Steampipe's scalability, this new architecture creates plenty of headroom to expand.
New AWS Compliance benchmarks
The AWS Compliance mod gained 5 new benchmarks during this cycle!
GxP 21 CFR Part 11: 113 controls
GxP EU Annex 11: 18 controls
NIST 800-171 Revision 2: 126 controls
CISA Cyber Essentials: 119 controls
Federal Financial Institutions Examination Council (FFIEC): 117 controls
And the AWS Foundational Security Best Practices benchmark added 32 new controls.
On the Azure front, Azure Compliance added the CIS 1.4 benchmark.
New plugins
Since our last release, we've added 5 new plugins.
crt.sh - query certificates, issuers, logs, etc
CrowdStrike - query detection events, hosts, users, vulnerabilities, etc.
FreshService - query agents, assets, tickets, etc
Trivy - query scan artifacts, secrets, vulnerabilities, etc.
Vercel - query DNS records, domains, projects, teams, etc.
Thanks to Ciaran Finnegan for help with CrowdStrike, and Graza Andersson for contributing FreshService.
During this cycle we added 14 new tables to the AWS plugin. And the GitHub plugin acquired one new table, github_audit_log, thanks to Jelmer Borst who added the table and Narayan Gowraj who tested it.
New mods
AWS Perimeter - check across all your AWS accounts for resources that are insecurely configured, publicly accessible, or shared with untrusted accounts. The mod comprises 10 benchmarks and 52 controls.
Hacker News Insights - see which languages, companies, databases, and clouds are popular right now on Hacker News.
GitHub Compliance - run the CIS Supply Chain Security compliance benchmark to check source code, build pipelines, dependencies, artifacts, and deployment.
Secure AWS regions
Steampipe now runs in the AWS C2S and SC2S regions. That means users in the U.S. Department of Defense, and in the intelligence community, can now run queries, benchmarks, and dashboards against Top Secret and Secret workloads on these airgapped networks.
Live connection refresh
When you revise an aggregator connection in a plugin config file, or update expired credentials, Steampipe now auto-detects the changes and instantly applies them.
net_http_request
With the addition of the net_http_request table, the Net plugin now enables Steampipe to work as an HTTP client. Queries can use GET or POST, specify HTTP headers, and send POST requests with headers and body.
Let’s get building!
Steampipe delivers tools to build, execute, and visualize and share cloud configuration, compliance, and security frameworks using HCL + SQL and a little elbow grease. To support those tools, it maps a growing suite of APIs to tables that you can query, and join across, in Postgres.
Do you want to help us expand the open source documentation and control coverage for CIS, PCI, HIPAA, and NIST? Add tables to existing plugins? Create plugins to bring new APIs into the mix? Create dashboards to visualize those APIs? The best way to get started is to join our Slack workspace and raise your hand. We would love to talk to you!
For even more good stuff in v0.16.0, check out the full release notes on GitHub.