Daemon is a technology consultancy, based in the UK and South Africa, that helps customers assess their alignment with the best practices defined by Amazon's AWS Well-Architected Framework. Their search for ways to streamline and accelerate the process turned up several commercial tools, including Turbot, but their preference was for a free and open-source solution. Steampipe turned out to be just what they were looking for. "We saw a massive opportunity to use it, and also customize it," says Jean-Pierre Pienaar, senior consultant at Daemon.
The process begins with "pre-discovery": they gain access to a customer's AWS environments, identify the workloads to be reviewed, and explore them using a variety of vendor-provided and in-house tools. The goal here is to identify issues to be addressed by a Well-Architected assessment; this was a manual process that they realized Steampipe could help automate.
Collaborative innovation
Because great minds think alike, they saw — before we announced our AWS Well-Architected mod — that it would be possible to build a custom mod that maps controls from AWS Compliance to the questions asked in Well-Architected Security pillar. Their first version of the custom mod mapped controls from AWS Compliance and added tags to align exporated outputs to the Well-Architected framework's questions and answers.
When they saw our announcement they contacted us to to see how we might collaborate. Inspired by their work, we released v0.8 of our Well-Architected mod. It incorporates their control mappings, and expands the tagging scheme to enable deeper integration with the Well-Architected tool's API which they're using to upload answers. We then wrote a post showing how anyone can extend the mod with new mappings. Following that model, Daemon have now successfully replaced their custom mod with the official one. They, and others, are now in a position to add new mappings to existing controls, create new controls based on new queries, and address additional Well-Architected pillars.
Facts versus emotions
Nathan Webster, principal consultant with Daemon, says the benefit of this approach isn't just automation of a manual checklist, though that is valuable. The Well-Architected benchmark delivers facts related to the questions asked, and that "helps take the emotion out of these conversations." In one case, a consulting client who needed to answer a question about whether backups are encrypted, and who believed that the answer was yes, learned otherwise. The benchmark revealed that while encryption was currently enabled, there were unencrypted legacy resources. That's need-to-know information, not a criticism of anyone's ability to configure AWS properly. And this data-driven approach is more likely to enagage technical partners who may otherwise tend to regard the assessment process as just a subjective checklist exercise.
Next steps
To close the loop, Daemon have created a script that converts the benchmark's exported JSON to a mapping file and uploads the results directly into the AWS Well-Architected API. A copy of the results is also exported to HTML and stored in S3. For controls in the alarm state, links to the failures in the report are noted in the Well-Architected review so they can be referenced in the review session with the client. The script is currently in customer trials; once proven, it will be shared with the community via Daemon's partnership with AWS.
Meanwhile, the Daemon team encourages others to help expand the Well-Architected mod. You'll need to apply your domain knowledge in order to map queries and controls to the framework, says Jean-Pierre Pienaar, but Steampipe makes it possible to do that "in a nice, complete way that's repeatable and scalable." If you dive in and give it a try, please let us know how it goes!