v0.9.0: Dynamic tables & improved control outputs

select
  region,
  instance_state as state,
  instance_type as type
from
  aws_ec2_instance;

+-----------+---------+-----------+
| region    | state   | type      |
+-----------+---------+-----------+
| eu-west-1 | running | t3.medium |
| eu-west-2 | running | m5a.large |
| us-east-1 | running | t3.large  |
+-----------+---------+-----------+
        

tl;dr

→ Tables from CSV files with support for other dynamic sources.
→ Improved control output for mods with summaries and new output formats.
→ 7 new plugins and mods.
→ Even more goodies in the full release notes.

Tables from CSV files

The APIs delivered by Steampipe plugins have so far worked with fixed schemas. With this release we now enable a different kind of plugin that maps tables on the fly. The first of these is the new CSV plugin. Point it at a directory containing CSV files, and a corresponding set of tables appears in Steampipe. Now spreadsheets can join the party, along with a world of other data sources that can export to CSV.

Here's some data in a CSV file called owners.csv.

resource_type,owner
ec2_instance,dwight@dundermifflin.com
ec2_instance,jan@dundermifflin.com
security_group,george@dundermifflin.com

If that file lives in a directory where you've told the CSV plugin to look for data, its contents are magically available as a table!

select
  *
from 
  csv.owners o
where
  o.resource_type = 'ec2_instance'

+---------------+--------------------------+ 
| resource_type | owner                    | 
+---------------+--------------------------+ 
| ec2_instance  | dwight@dundermifflin.com | 
| ec2_instance  | jan@dundermifflin.com    | 
+---------------+--------------------------+ 
    

These dynamic tables will work hand-in-hand with the variables and query parameters introduced in v0.8.0. Those features enable you to customize queries and mods with user-supplied data, such as required tags. Now you can also manage tag vocabularies in spreadsheets.

For example, here's a query for the owner tag in the aws_ec2_instance table.

select 
  instance_id,
  tags ->> 'owner' as owner
from
  aws_ec2_instance

    instance_id     |          owner               
--------------------+--------------------------    
i-0e97f373db22dfa3f | dwight@dundermifflin.com     
i-0dc60dd191cb86539 | george@dundermifflin.com                         
i-06ee5c096826de741 | jan@dundermifflin.com
    

To find instances whose owner tag isn't listed in owners.csv:

select 
  instance_id,
  tags ->> 'owner' as owner
from 
  aws_ec2_instance i 
where not exists 
  (
    select 
      * 
    from 
      csv.owners o 
    where 
      o.resource_type = 'ec2_instance' 
      and o.owner = i.tags ->> 'owner'
  );

+---------------------+--------------------------+
| instance_id         | owner                    |
+---------------------+--------------------------+
| i-022a51a815773780d | jan@dundermifflin.com    |
| i-03f3b66e057009f41 | dwight@dundermifflin.com |
+---------------------+--------------------------+
    

Beyond using spreadsheets in this way, you can now begin to inventory data assets that live in those places, and use Steampipe to define and check the integrity rules you want to enforce. We love spreadsheets because they're convenient, but we hate that they're uncontrolled. Now you can write Steampipe controls for your mission-critical spreadsheets!

Note to developers: To write another plugin that creates tables from dynamically-discovered schemas, check out the new SchemaMode argument in the plugin definition.

Improved control output

When you run steampipe check all in a directory where you've installed one of the Steampipe mods, the output can be overwhelming. So we've added a nice summary to the end of the report.

We've also expanded the output formats. Along with csv and json, you can now output or export html and markdown (md). They look the same, here's an example of HTML output.

New plugins and mods

Since our last release, we've added 6 new plugins:

• Code - finds secrets in columns of other plugins' tables
• CSV - query tables defined in .CSV files
• Heroku - query account, addon, domain, dyno, pipeline, etc
• IBM Cloud - query account, access_group, iam_role, iam_user, etc
• Scaleway - query account_ssh_key, instance_server, rdb_database, etc
• Terraform Enterprise - query organization, sentinel_policy, team, etc

Major mod updates:

• Kubernetes Compliance with NSA & CISA Hardening Guidance
• Azure Compliance now includes NIST SP 800-53 Revision 5

Let’s get building!

Steampipe now delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST… and the best way to get started is to join our new Slack workspace and raise your hand; we would love to talk to you!

For even more good stuff in v0.9.0, checkout the full release notes on GitHub.