Claranet France, a managed service provider for AWS, Azure, and GCP, began using Steampipe in 2021 when Patrick Decat, Lead Cloud Engineer, was looking for a better way to populate Claranet's in-house CMDB (configuration management database). The existing solution, which relied on Python scripts to query AWS and Azure, and then push results into a database, needed improvement on two fronts: scalability and ease of use. With 600+ AWS accounts and 300+ Azure subscriptions, the Python scripts were struggling to perform well, combine cross-cloud queries, reason over their outputs in a consistent way, and easily acquire new data sources.
Turbocharging the Python scripts
Steampipe addresses both challenges. Its highly-concurrent architecture makes quick work of intensive scanning at large scale. And with SQL as the universal language for reasoning over diverse data sources it was straightforward to combine query outputs from different clouds to produce the same the format that was already feeding the existing backend.
The Azure component of this solution is now running in production, with the AWS version in staging. Python still drives the process, but now by using Steampipe as a component. Scans occur daily; there are plans to increase the frequency and also run on demand when new accounts appear. And it all scales well. "It's surprising how many requests can be handled when running benchmarks and queries against multiple backends," Patrick says.
Better security and compliance reports
In early 2022 Claranet reached for Steampipe again to improve its security and compliance posture. They'd been using AWS Compliance to run benchmarks, export results to JSON, and convert the JSON to PDF reports. But those PDFs weren't ideal for the teams building cloud infrastructure. So the new approach runs benchmarks, reformulates the results, and feeds them into an internal portal that's organized by customer, with per-account summaries of defects and anomalies. It's currently a proof-of-concept; teams are providing feedback; the approach looks promising so far.
To customize the benchmark outputs, the new system will make extensive use of Steampipe's ability to reuse and remix. "It's easy to create a custom benchmark," Patrick says, "by composing queries and controls that already exist, along with new ones you write from scratch."
A multi-plugin offboarding tool
Here's a problem every company will recognize: when somebody leaves, it's a hassle to check that they were removed from various SaaS tools. Claranet solves it by having Steampipe automate the checks, with the help of a suite of plugins: Azure ActiveDirectory, BitBucket, GitHub, Google Directory, and PagerDuty.
In each case, the solution answers the question: "Does this person known to ActiveDirectory still exist in SaaS tool X?" In the case of GitHub Organizations another plugin, Terraform, comes into play. Claranet uses Terraform to maintain the map of realnames to GitHub identities that it provisions into GitHub Organizations. So the Terraform plugin is a great way to assure that offboarded identities are properly removed.
Scaling Steampipe with AWS Lambda
Along the way, Patrick found himself reaching for a better way to schedule Steampipe jobs. AWS Batch on Fargate wasn't just a cold-start headache: a 60-second delay times 600 AWS accounts really adds up. It was also a rate-limit headache because Steampipe jobs could only appear from a limited range of IP addresses, so would get throttled. (This was specific to the setup of their NAT gateway, and not normally a problem with Fargate.) AWS Lambda looked like the way to cure both headaches.
There’s always a catch, though, and in this case it was that in Lambda, localhost can randomly resolve to 127.0.0.1 (IPv4) or [::1] (IPv6). Patrick raised a pull request to address that problem, has been running with the patch since March, and is seeing a 10x gain from 20 parallel requests in Fargate to 200 in Lambda.
A Steampipe plugin for the in-house CMDB
The in-house database that's fed by Steampipe provides an API, so it was natural to imagine a Steampipe plugin to query that API. Patrick built one, and enjoys the leverage that it provides. "Postgres enables complex queries that we cannot do with our CMDB API," he says. The API is private, so the plugin is too, but the project motivated another PR, this one extending the plugin SDK to enable hydrate functions to pass the Postgres like operator to plugins, so they can in turn push filtering down into APIs where it's most efficiently done.
Next steps
All this adds up to lots of use of Steampipe within Claranet where, we were delighted to learn, there's an internal Steampipe Slack channel with 34 members!
Patrick's other contributions include the Azure plugin's new azure_private_dns_zone table, a raft of documentation fixes, and ongoing proposals to improve Steampipe's scalability.
What's next? Claranet would like to make it easier for everyone to run Steampipe in AWS Lambda, and we look forward to Patrick’s ongoing work on that front.
Meanwhile, we're grateful for Claranet’s broad and deep engagement with Steampipe. If others would like to explore the methods discussed here in more detail, our Slack community is the right place to be.