v0.20.0: Faster startup & revamped mod dependencies

Faster startup

v0.20.0 supercharges Steampipe's startup and connection sync time. No more waiting minutes for hundreds of connections to sync up, you're now set to query in seconds!

Connection setup is done in priority order (by search_path) and is now fully asynchronous. You can start querying your most important connections and aggregators instantly, with full autocomplete, even for your largest aggregators.

Schema optional

Until v0.20.0, Steampipe required each connection to be set up as a separate schema in Postgres. So, if you have 1000 AWS accounts in an aggregator, Steampipe would create 1000 schemas and ~400,000 tables in Postgres! That's a lot of overhead, and it's now optional.

Using import_schema = "disabled" we can tell Steampipe to include connections in the aggregator (aws_all) but not create separate schemas for them. For the example above, that means only 1 schema and 400 tables are created in Postgres!

connection "aws_all" {
  type          = "aggregator"
  plugin        = "aws"
  connections   = ["aws_*"]
}

connection "aws_1" {
  plugin        = "aws"
  import_schema = "disabled"
}

connection "aws_2" {
  plugin        = "aws"
  import_schema = "disabled"
}

Improved mod dependencies

Mod dependencies enable you to reuse and remix existing mods. In v0.20.0 we've refined this mechanism to make it more potent for mod authors.

Steampipe's new AWS Well-Architected mod, our first major mod to depend on another mod (AWS Compliance), inspired us to enhance this feature.

A mod may specify dependencies on other mods. While you can manually edit the mod dependencies in the mod.sp, they are more commonly managed by Steampipe when you install, update, or uninstall mods via the Steampipe mod commands. The version can be an exact version or a semver string:

require {
  mod "github.com/turbot/steampipe-mod-aws-compliance" {
    version = "^0.10"
  }
  mod "github.com/turbot/steampipe-mod-aws-insights" {
    version = "2.0"
  }
  mod "github.com/turbot/steampipe-mod-gcp-compliance" {
    version = "*"
  }
}

You may pass args to set variables defined in the dependency mods. You can pass hard-coded values (literals), however it is more common to define variables in your mod that encapsulate the variables and optionality of your dependencies:

variable "common_dimensions" {
  type        = list(string)
  description = "A list of common dimensions to add to each control."
  default     = [ "account_id", "region" ]
}

variable "tag_dimensions" {
  type        = list(string)
  description = "A list of tags to add as dimensions to each control."
  default     = []
}

mod "aws_well_architected" {
  require {
    mod "github.com/turbot/steampipe-mod-aws-compliance" {
      version = "^0.63.0"
      args = {
        common_dimensions = var.common_dimensions,
        tag_dimensions    = var.tag_dimensions
      }
    }
  }
}

We've also tightened up some constraints that weren't previously enforced. When a mod refers to a resource in another mod that it depends on, it now must fully qualify the reference: <modname>.<resourcetype>.<resourcename>. And a mod can no longer refer to a local variable defined in another mod. Note that if your mods were taking advantage of the prior non-enforcement of these rules, you'll need to adjust to them in v0.20.0.

Time-axis charts

Prior to v0.20.0 it was hard to chart time-series data when there were were gaps in the data: missing hours, days, or years. You could massage the data with Postgres' generate_series function but that was an awkward and verbose solution.

Enter José Reyes, a prolific explorer of Steampipe and author of a remarkable series of deep dives into various aspects of the system. In part 3 he noticed the issue, dug into the server- and client-side machinery, and worked out how to specify when the x axis is of type time. In that case, the chart library can render the series with appropriate gaps. And as a bonus, the labels become cleaner and more adaptive. He writes:

All that code has been contributed back to Steampipe, in PR 3214. Hopefully, it will get merged soon!

Indeed it has been. On several internal dashboards we've already taken advantage of the new time axis to simplify queries. One 30-line query that used the generate_series method, for example, now clocks in at just 15 lines.

Here's a chart of incidents by day, with appropriate gaps for days with no incidents.

Thanks José!

… and more

• Cache optimizations & options
• Expanded workspace configuration options
• Other bug fixes & improvements

Start your engines!

We're on a journey to make Steampipe ever more performant. The startup optimization in this release is a major milestone, please let us know how it goes for you!