Steampipe FAQ

Topics

Basics and Functionality

What kinds of data sources can Steampipe query?

Steampipe's extensible plugin model allows it so support a wide range of source data, including:

• Cloud providers like AWS, Azure, GCP, Cloudflare, Alibaba Cloud, IBM Cloud, and Oracle Cloud.
• Cloud-based services likeGitHub, Zoom, Okta, Slack, Salesforce, and ServiceNow.
• Structured files like CSV, YML and Terraform.
• Ad hoc investigation of network services like DNS & HTTP. You can even run arbitrary commands on local or remote systems and query the output.

Find published plugins in the Steampipe Hub!

Does Steampipe store query results locally?

No. Plugins make API calls, results flow into Postgres as ephemeral tables that are only cached for 5 minutes (by default). Steampipe optimizes for live data, and stores nothing by default.

Can I use psql, pgadmin, or another client with Steampipe?

Yes. Steampipe exposes a Postgres endpoint that any Postgres client can connect to. When you start the Steampipe service, Steampipe will print the connection string to the console. You can also run steampipe service status to see the connection string.

Can I export query results as CSV, JSON, etc?

Yes. You can run steampipe query with the --output argument to capture results in CSV or JSON format:

steampipe query --output json "select * from aws_account"

Does Steampipe work with WSL (Windows Subsystem for Linux)?

Yes, with WSL 2.0.

Does Steampipe support SQL write operations?

No. Steampipe is optimized for read-only query. However, it works closely with Flowpipe which can run Steampipe queries and act on the results.

How do I know what tables and columns are available to query?

In the Steampipe CLI you can use .inspect to list tables by plugin name, e.g. .inspect aws to produce a selectable list of tables. When you select one, e.g. .inspect aws_s3_bucket you'll see the schema for the table. You can see the same information on the Steampipe Hub.

Can I query more than one AWS account / Azure subscription / GCP project?

Yes. You can create an aggregator. This works for multiple connections of the same type, including AWS/Azure/GCP as well as all other plugins.

It's common practice to use a script to generate the .spc files for an organization. See, for example, https://github.com/happy240/steampipe-conn-generator-for-aws-organization.

Turbot Pipes provides integrations to simplify the setup and keep configurations up to date by automatically maintaining connections for AWS organizations, Azure tenants, GCP organizations and GitHub organizations

Performance and Scalability

How well does Steampipe perform when querying multiple connections?

The large variance in customer environments and configurations make it impossible to provide specific estimates, but many users have scaled Steampipe to hundreds of connections. Multiple connections are queried in parallel, subject to plugin-specific rate-limiting mechanisms. Recent data is served from cache. Connection-level qualifiers, like AWS account_id, can reduce the number of connections queried.

Writing good queries makes a significant difference in performance:

• Select only the columns that you need, to avoid making API calls to hydrate data that you don't require.
• Limit results with a where clause on key columns when possible to allow Steampipe to do server-side row-level filtering.

How does Steampipe handle rate-limiting?

Generally, Plugins are responsible for handling rate limiting because the details are service specific. Plugins should typically recognize when they are being rate limited and backoff and retry either using their native Go SDK, the basic rate-limiting provided by the plugin SDK, or adding limiters compiled in the plugin . You can also define your own custom limiters in configuration files.

How can I control the amount of memory used by Steampipe and plugins?

To set a set soft memory limit for the Steampipe process, use the STEAMPIPE_MEMORY_MAX_MB environment variable. For example, to set a 2GB limit: export STEAMPIPE_MEMORY_MAX_MB=2048.

Each plugin runs as its own process, and can have its own memory limit set in its configuration file using the memory_max_mb attribute. For example:

plugin "aws" {
  memory_max_mb = 2048
}

Alternatively, you can set a default memory limit for all plugin processes using the STEAMPIPE_PLUGIN_MEMORY_MAX_MB environment variable. For example, to set a 2GB limit:

export STEAMPIPE_PLUGIN_MEMORY_MAX_MB=2048

Plugins and Customization

Can plugin X have a table for Y?

If the plugin lacks a table you need, file a feature request (GitHub issue) for a new table in the applicable plugin repo, e.g. github.com/turbot/steampipe-plugin-{pluginName}/issues. Of course we welcome contributions! The following guide shows you how to write your first table.

Does Steampipe have a plugin for X?

If you have an idea for a new plugin, file a feature request (GitHub issue) with the label 'plugin suggestions'. We welcome code contributions as well. If you want to write a plugin, our guide will help you get started.

How can I dynamically create Steampipe connections?

All connections are specified in ~/.steampipe/config/*.spc files. Steampipe watches those files and reacts to changes, so if you build those files dynamically you can create connections dynamically.

Can I create and use regular Postgres tables?

Yes. Each Steampipe plugin defines its own foreign-table schema, but you can create native Postgres tables and views in the public schema.

Can I use Steampipe plugins with my own database?

Yes. Most plugins support native Postgres FDWs and SQLite Extensions. Find the details for a plugin in its Steampipe Hub documentation, e.g. the AWS plugin for Postgres and for SQLite.

Deployment

Can I run Steampipe in a CI/CD pipeline or cloud shell?

Yes, it's easy to install and use Steampipe in any CI/CD pipeline or cloud shell.

Where is the Dockerfile or container example?

Steampipe can be run in a containerized setup. We run it ourselves that way as part of Turbot Pipes. However, we don't publish or support a container definition because:

• The CLI is optimized for developer use on the command line.
• Everyone has specific goals and requirements for their containers.
• Container setup requires various mounts and access to configuration files.
• It's hard to support containers across many different environments.

We welcome users to create and share open-source container definitions for Steampipe.

Troubleshooting and Debugging

My query resulted in an error stating that is missing 1 required qual. What does that mean?

The error indicates that you must add a where = (or join...on) clause for the specified column to your query. The Steampipe database doesn't store data, it makes API calls to get data dynamically. There are times when listing ALL the elements represented by a table is impossible or prohibitively slow. In such cases, a table may require you to specify an equals qualifier in a where or join clause.

How can I know what API calls a plugin makes?

Steampipe plugins are open source, and you can inspect the code to see what calls it is making.

Some plugins (like the AWS plugin) provide information about the APIs being called using function tags that can be inspected by running Steampipe in diagnostic mode.

Can I disable the Steampipe cache?

Yes. Caching significantly improves query performance and reduces API calls to external systems. It is enabled by default, but you can disable it either for the server or for a given client session.

To disable caching at the server level, you can set the cache option to false in ~/.steampipe/config/default.spc:

options "database" {
  cache = false
}

Alternatively, set the STEAMPIPE_CACHE environment variable to false before starting Steampipe.

Within an interactive query session, you can disable caching for the client session with the .cache off meta-command.

A plugin isn't doing what I expect, how can I debug?

Steampipe writes plugin logs to ~/steampipe/logs/plugin-YYYY-MM-DD.log. By default, these logs are written at warn level. You can change the log level with the STEAMPIPE_LOG_LEVEL environment variable:

export STEAMPIPE_LOG_LEVEL=TRACE

If Steampipe is running, the plugins must be restarted for it to take effect: steampipe service stop --force && steampipe service start.