Paul Solomon, Senior SRE / Platform Engineer at a Chicago based fintech company, is responsible for the company's AWS infrastructure. As the company's AWS infrastructure grew, it became increasingly painful to answer common questions like:
Which IAM keys are more than 90 days old?
What resources are inactive that should be removed?
Is the Owner tag applied to all my resources?
Given a list of IP addresses from the security team, which machines do they represent in which AWS accounts?
When he discovered Steampipe he was intrigued by its ability to inventory cloud assets and join across diverse APIs. Like many Steampipe newcomers he wasn't sure his rusty SQL skills would enable him to make effective use of the tool, but soon set those concerns aside.
How Steampipe simplifies the process
"The docs are great," he says. With thousands of examples to copy, paste, and run, he quickly became productive with Steampipe. Paul and his team appreciate Steampipe's versatility as a component that can integrate into almost any environment. For example, that list of IP addresses is provided as a CSV file that he joins with AWS queries by way of the CSV plugin. Steampipe wraps all that complexity and provides a standard SQL interface in the form of a component that their scripts can orchestrate.
Building upon Steampipe for cost savings
For many key questions there's no need to develop queries because existing dashboards and benchmarks meet the need, notably AWS Tags and AWS Thrifty. Thanks to Thrifty, the team achieved a 30% reduction in their monthly AWS bill. Excessive use of load balancers in test environments was one piece of low-hanging fruit. Thrifty's ability to flag underutilized resources also helped them cut costs.
Things have gotten even easier now that Thrifty enables you to specify common dimensions, like account_id and region, so you can understand control outputs in a more granular way, and organize exports more effectively. You could add those dimensions to the controls yourself, and Paul did in some cases, but he's happy that Steampipe makes it even easier with flexible options to output the right data for his team's recurring cloud reporting.
Another initiative focuses on Datadog, which the dev teams use to gather performance data from their services. That had required scripts to query the Datadog API, another chore that's now being delegated to Steampipe. Using the Steampipe Datadog plugin, Paul takes the opportunity to scoop up all the tag information that he can, with a view toward aligning tag namespaces across cloud services. "You could do all this the hard way," Paul says, "but Steampipe makes it so much easier."
Broadening scope towards security and compliance
Now that the team can answer all kinds of questions with SQL queries instead of scripts that wrangle the AWS API, they’re exploring how AWS Compliance can help assess their security posture and track progress toward compliance. The team consistently uncovers new opportunities to leverage Steampipe further for their SRE use cases to ensure their cloud environments are operationally sound, cost optimized, and secure.
Paul and his team are not alone, they are among a growing number of SREs who benefit from Steampipe in the community. If you have a similar story to tell, please let us know, we're always eager to know how people are using Steampipe within their organizations.