v0.12.0: Templates for control outputs, two new formats

select
  region,
  instance_state as state,
  instance_type as type
from
  aws_ec2_instance;

+-----------+---------+-----------+
| region    | state   | type      |
+-----------+---------+-----------+
| eu-west-1 | running | t3.medium |
| eu-west-2 | running | m5a.large |
| us-east-1 | running | t3.large  |
+-----------+---------+-----------+
        

tl;dr

→ New output formats.
→ Templates for control outputs.
→ 4 new plugins.
→ Even more goodies in the full release notes.

New output formats

We've added two new formats for control outputs: asff and nunit3.

ASFF

ASFF is the Amazon Security Finding Format:

AWS Security Hub consumes, aggregates, organizes, and prioritizes findings from AWS security services and from the third-party product integrations. Security Hub processes these findings using a standard findings format called the AWS Security Finding Format (ASFF), which eliminates the need for time-consuming data conversion efforts. Then it correlates ingested findings across products to prioritize the most important ones.

Here's an example of asff output for an AWS Compliance run. It's the finding for the foundational_security_ec2_2 control (The VPC default security group should not allow inbound and outbound traffic).

{
    "SchemaVersion": "2018-10-08",
    "Id": "aws_compliance.control.foundational_security_ec2_2",
    "ProductArn": "arn:aws:securityhub:ap-south-1:453319552164:product/453319552164/default",
    "ProductFields": {
        "ProviderName": "Steampipe",
        "ProviderVersion": "0.12.0-rc.1"
    },
    "GeneratorId": "steampipe-foundational_security_ec2_2",
    "AwsAccountId": "453319552164",
    "Types": [
        "automated"
    ],
    "UpdatedAt": "2022-01-20T22:33:52+05:30",
    "CreatedAt": "2022-01-20T22:33:52+05:30",
    "Title": "2 The VPC default security group should not allow inbound and outbound traffic",
    "Description": "This control checks that the default security group of a VPC does not allow inbound or outbound traffic. The rules for the default security group allow all outbound and inbound traffic from network interfaces (and their associated instances) that are assigned to the same security group.",
    "Severity": {
        "Label": "HIGH"
    },
    "Resources": [
        { 
            "Type": "Other",
            "Id": "arn:aws:ec2:ap-south-1:453319552164:security-group/sg-79245001"
        }
    ],
    "Compliance": {
        "Status": "FAILED"
    }
}

Here's one way to integrate this finding into Security Hub.

aws securityhub batch-import-findings --findings "{\"SchemaVersion\":\"2018-10-08\", ... {\"Status\":\"FAILED\"}}"

Now the finding appears in Security Hub under the Findings section.

nunit3

This is the XML format for NUnit 3.0 test results. Here's an example of nunit3 output for the CIS v1.40 benchmark.

<test-case id="cis_v140_1_1::0" name="aws_compliance.control.cis_v140_1_1::0" result="Passed">
<properties>
    <property>
     <key>steampipe:status</key>
     <value>info</value>
    </property>
    <property>
     <key>steampipe:reason</key>
     <value>Manual verification required.</value>
    </property>
    <property>
    <key>steampipe:dimension:account_id</key>
    <value>899206412154</value>
    </property>
</properties>
<reason>
<message><![CDATA[Manual verification required.]]></message>
</reason>
</test-case>

Here is a visualization made with AWS CodeBuild.

Templates for control outputs

This release adds a templating mechanism for control outputs, based on the golang text/template package. For each output format there's now a directory, in ~/.steampipe/check/templates/, which minimally includes a file called output.tmpl. The name of the directory defines the name of an output format. The content of output.tmpl defines how to unpack and format a golang struct, called Data, that's passed to the template.

For example, let's create ~/.steampipe/check/templates/summary/output.tmpl like so.

{{ define "output" }}

{{ range .Data.Root.Groups }}
Summary for {{ .Title }}
{{ end }}
total: {{ .Data.Root.Summary.Status.TotalCount }}
passed: {{ .Data.Root.Summary.Status.PassedCount }}
failed: {{ .Data.Root.Summary.Status.FailedCount }}
skipped: {{ .Data.Root.Summary.Status.Skip }}

{{ end }}

Now there's a new summary output format.

steampipe check --output=summary all

Summary for Zoom Compliance

total: 185
passed: 119
failed: 66
skipped: 0

This command puts the above output into the file output.summary.

steampipe check --export=output.summary all

This command produces an inferred filename like all-20220119-111307.summary.

steampipe check --export=summary all

Along with asff and nunit3, the csv, html, json, and md formats now use this mechanism. So you can explore those templates in ~/.steampipe/check/templates to find more advanced examples that you can adapt for your own needs.

New plugins

Since our last release, we've added 4 new plugins:

• Confluence - query spaces and contents
• Hypothesis - query annotations, user profile
• PagerDuty - query incidents, schedules, services, users, etc
• Prometheus - query alerts, metrics, rules, targets, etc

We are always improving the suite of plugins. During this cycle we added tables to AWS, Azure, Cloudflare, Okta, and Oracle Cloud.

Let’s get building!

Steampipe delivers tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease. To support those tools, it maps a growing suite of APIs to tables that you can query, and join across, in Postgres.

Do you want to help us expand the open source documentation and control coverage for CIS, PCI, HIPAA, and NIST? Add tables to existing plugins? Create plugins to bring new APIs into the mix? The best way to get started is to join our Slack workspace and raise your hand. We would love to talk to you!

For even more good stuff in v0.12.0, check out the full release notes on GitHub.