Steampipe unbundled

Unix philosophy holds that a software tool should do one thing well, and combine readily with other tools that abide by the same principle. Steampipe began that way. At launch in early 2021, Steampipe was a CLI that launched Postgres along with a foreign-data-wrapper extension that managed a suite of plugins — initially, 13 of them — that translate APIs to SQL-queryable tables.

Then the first mods appeared, initially for compliance and cost control. Mods layered on top of plugins; they used HCL to define benchmarks and controls; they relied on plugin-enabled SQL queries to flow data into controls. The CLI added commands to run benchmarks and save results.

Then Steampipe added a dashboard server that connected SQL queries to a new suite of HCL widgets for all-purpose data visualization. The CLI added more commands to run dashboards and save results.

Flowpipe would have made Steampipe even more complicated, so we chose to make it a new, standalone workflow tool that can use Steampipe (among other databases) as one of its methods for acquiring data from cloud services. And that's been great! But it became clear that Steampipe itself needed to be refactored in order to align with the Unix prime directive.

The first step was to decouple Steampipe from its batteries-included Postgres database, so you could use plugins with your own database. That effort resulted in three new plugin flavors: for Postgres, for SQLite, and for standalone export. And it infused Steampipe's signature talent — zero-ETL live query of APIs — into a suite of single-purpose tools that you can pick up and use in any context.

Now, with Powerpipe, we've decoupled the core Steampipe engine from the Steampipe CLI and dashboard server. And as with Flowpipe, the queries that drive Powerpipe dashboards can connect to Steampipe, Postgres, SQLite, or others.

Here's a summary of the refactoring.

If you're using Steampipe today to run dashboards and/or benchmarks, that all still works, but over time we'll deprecate these uses of Steampipe in favor of Powerpipe. In Migrating from Steampipe to Powerpipe we show what that transition looks like. Initially we expect most people will use Powerpipe's dashboard and benchmark features with the ecosystem of plugins and mods that Steampipe birthed, but the HCL+SQL layer is now decoupled from that ecosystem. You can build your own dashboards-as-code mods to visualize your data, from any source. And you can create your own benchmarks and controls — again, over any source of data — to define and enforce your own rules for security, compliance, and data integrity.

Likewise, if you're using Flowpipe today to run workflows, the new query trigger underscores the value of refactoring Steampipe. The Flowpipe query trigger can watch any database for changes. If the changes you're looking for are best captured by way of Steampipe-managed plugins, that database will be Steampipe. But you can build workflows that rely on other databases too, and even compose solutions that mix and match Steampipe and other databases — which, again, can optionally tap into the Steampipe plugin ecosystem.

Each of these tools — Steampipe, Flowpipe, and Powerpipe — is now well-defined and composable. Use any BI tool with Steampipe, use any database with Flowpipe or Powerpipe, run any of these single binaries separately or together in CI/CD pipelines. This is the kind of arrangement we know DevOps engineers prefer. And now, with a clearer set of components and interfaces, we're set to continue expanding and innovating the Pipes family. As always, we're eager to hear from you, so please do let us know your thoughts on this strategy and direction.