v0.11.0: Faster startup and composable mods

select
  region,
  instance_state as state,
  instance_type as type
from
  aws_ec2_instance;

+-----------+---------+-----------+
| region    | state   | type      |
+-----------+---------+-----------+
| eu-west-1 | running | t3.medium |
| eu-west-2 | running | m5a.large |
| us-east-1 | running | t3.large  |
+-----------+---------+-----------+
        

tl;dr

→ Faster startup.
→ Composable mods.
→ 4 new plugins.
→ Major mod updates.
→ Even more goodies in the full release notes.

Faster startup

This release includes a number of startup optimizations, including:

• The plugin manager instantiates plugins in parallel.

• The connection manager uses file modified time instead of filehash to detect connection changes.

• When multiple connections share a schema, we only fetch it once.

The first time you run steampipe query you'll see an immediate difference.

In v0.10 we achieved major gains by spawning database connections to run controls in parallel, and by enabling those parallel controls to share a common plugin-managed cache.

v0.11 brings two additional optimizations that benefit control runs. First, we now only create prepared statements for controls that interpolate parameters into SQL statements. When controls use non-parameterized SQL statements, we avoid that overhead. Second, we avoid fetching the database schema for non-interactive query execution.

Here's the difference for a single control run.

v10:

time steampipe check control.cis_v140_1_1
4.44s

v11:

time steampipe check control.cis_v140_1_1
3.08s

Composable mods

In v0.11 you can build mods that use queries, controls, and benchmarks defined in other mods. The new mod install command enables that scenario.

For example, let's build a custom mod that derives benchmarks from AWS Compliance and Github Sherlock, and also extends GitHub Sherlock with a new control.

We'll use the the directory mod_acme for this example. First, install two mods like so.

steampipe mod install github.com/turbot/steampipe-mod-aws-compliance

steampipe mod install github.com/turbot/steampipe-mod-github-sherlock

This creates mod.sp.

mod "local" {
  title = "mod_acme"
  require {
    mod "github.com/turbot/steampipe-mod-aws-compliance" {
      version = "latest"
    }
    mod "github.com/turbot/steampipe-mod-github-sherlock" {
      version = "latest"
    }
  }
}

steampipe mod list

local
├── github.com/turbot/steampipe-mod-aws-compliance@v0.22
└── github.com/turbot/steampipe-mod-github-sherlock@v0.5

Now we'll do three things in a separate new file, acme.sp.

1. Create a version of aws_compliance that cherrypicks just sections 1, 2, and 3 from cis_v140.

2. Create a version of github_sherlock that cherrypicks two of its four benchmarks.

3. Add a semantic versioning check to our own version of github_sherlock.

benchmark "acme_cis_v140" {
  title         = "ACME CIS v1.4.0"
  description   = "Only sections 1, 2, and 3."
  children = [
    aws_compliance.benchmark.cis_v140_1,
    aws_compliance.benchmark.cis_v140_2,
    aws_compliance.benchmark.cis_v140_3
  ]
}

benchmark "acme_github_sherlock" {
  title         = "Only issue_best_practices and private_repo, plus a new semver control."
  children = [
    github_sherlock.benchmark.issue_best_practices,
    github_sherlock.benchmark.private_repo_best_practices,
    control.repo_uses_semantic_versioning
  ]
}

control "repo_uses_semantic_versioning" {
  title = "Repo uses semantic versioning"
  sql = <<EOT
    select
      r.html_url || '@' || t.name as resource,
      case
        when t.name ~ '^v[0-9]+\.[0-9]+\.[0-9]+$' then 'ok'
        when t.name ~ '^v[0-9]+\.[0-9]+\.[0-9]+' then 'info'
        else 'alarm'
      end as status,
      r.full_name || '@' || t.name as reason,
      r.full_name
    from
      github_my_repository as r,
      github_tag as t
    where
      r.full_name = t.repository_full_name
      and r.full_name ~ 'turbotio/steampipe'
    order by
      r.full_name
  EOT
}

With this code in place, here are some checks you can run.

steampipe check all # run only these benchmarks/controls

steampipe check benchmark.acme_cis_v140 # just cis_v140 sections 1, 2, 3

steampipe check benchmark.acme_github_sherlock # 2 of 4 from github_sherlock plus semver

steampipe check control.repo_uses_semantic_versioning # just the semver check

To recap: the new mod install command serves the special case where you want to create your own mod to define benchmarks that remix, and perhaps extend, benchmarks and controls provided by existing mods.

New plugins

Since our last release, we've added 4 new plugins:

• Keycloak - query clients, identity providers, users, groups
• Splunk - query apps, indexes, users, searches
• Steampipe Cloud - query logs, users, workspaces, etc.
• Terraform - query resources in your Terraform files

We are always improving the suite of plugins. Highlights during this cycle: 8 new tables added to GitHub, 7 new tables added to Cloudflare, OIDC authentication support for Kubernetes.

Major mod updates

During this cycle we added 15 new benchmarks, comprising 192 controls, to the NIST CSF benchmarks.

Let’s get building!

Steampipe delivers tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease. To support those tools, it maps a growing suite of APIs to tables that you can query, and join across, in Postgres.

Do you want to help us expand the open source documentation and control coverage for CIS, PCI, HIPAA, and NIST? Add tables to existing plugins? Create plugins to bring new APIs into the mix? The best way to get started is to join our Slack workspace and raise your hand. We would love to talk to you!

For even more good stuff in v0.11.0, check out the full release notes on GitHub.