v0.19.0: Dynamic table aggregation, Kubernetes CRDs

Dynamic table aggregation

In v0.7 we introduced connection aggregators that parallelize queries across Steampipe connections. Then, in v0.9, came dynamic tables, first implemented by the AirTable and CSV plugins. These are two great features but, until now, they didn't work together. The inablity to aggregate across connections prevented us from adding dynamic tables to plugins like Kubernetes.

Now, v0.19, it's possible to aggregate dynamic tables across connections. Here's a simple example: aggregation of CSV files across parallel directories.

connection "csv_all" {
  plugin = "csv"
  type = "aggregator"
  connections = [ "*" ]
}

connection "csv_dynamic1" {
  plugin = "csv"
  paths = [ "~/csv/dynamic1/*.csv" ]
}

connection "csv_dynamic2" {
  plugin = "csv"
  paths = [ "~/csv/dynamic2/*.csv" ]
}

These are the files.

~/csv$ more dynamic1/test.csv
account,name,birthplace
dwight@dundermifflin.com,dwight,scranton

~/csv$ more dynamic2/test.csv
account,name,age
jim@dundermifflin.com,jim,58

This query aggregates across them.

> select * from csv_all.test
+--------------------------+--------+--------+------------------------------------+------------+
| account                  | name   | age    | _ctx                               | birthplace |
+--------------------------+--------+--------+------------------------------------+------------+
| dwight@dundermifflin.com | dwight | <null> | {"connection_name":"csv_dynamic1"} | scranton   |
| jim@dundermifflin.com    | jim    | 58     | {"connection_name":"csv_dynamic2"} | <null>     |
+--------------------------+--------+--------+------------------------------------+------------+

Kubernetes CRDs

We've been chomping at the bit to enable query of the Kubernetes resources defined by CRDs (Custom Resource Definitions). We held back until it was possible to aggregate across Kubernetes clusters, but now that we can, the new version of the plugin unlocks dynamic tables that dramatically expand the universe of resources you can query.

In one of our Kubernetes environments, the query select name from kubernetes_custom_resource_definition returns the names of 57 definitions, including alertmanagers, certificates, and sealedsecrets, that correspond to newly-available dynamic tables.

Here, for example, is a query of resources defined by cert-manager.

select
  name,
  api_version,
  common_name
from
  kubernetes_certificate

+------------------+--------------------+------------------+
| name             | api_version        | common_name      |
+------------------+--------------------+------------------+
| my-selfsigned-ca | cert-manager.io/v1 | my-selfsigned-ca |
+------------------+--------------------+------------------+

That query runs against a single Kubernetes connection, but if we wanted to query across several we would aggregate across them.

Quality of life improvements

We're always finding ways to make Steampipe faster, smarter, and easier to use. In v0.19 the improvements include:

Live refresh of CSV schemas. If you alter a CSV file by adding or removing a column, Steampipe formerly required a restart to reflect the change. Now that happens immediately.

Asynchronous loading of workspaces. The steampipe check command could stall when loading a complex workspace, switching to asynchronous loading ensures that won't happen.

Configurable database timeout. A new environment variable, STEAMPIPE_DATABASE_START_TIMEOUT, enables you to vary how long Steampipe waits for Postgres to start.

Improved error reporting. A new error-handling package provides more granular and informative error messages. Initially this will help us diagnose issues in Steampipe itself, and we aim to extend that capability to plugin authors as well.

Let's get building!

Have fun exploring all the new Kubernetes tables! We hope you'll share your experiences querying them, and adding those queries to dashboards, in our Slack community.