Do you run Steampipe at scale, and/or in constrained environments? Now you can control the amount of memory Steampipe and its plugins can use. Do your queries sometimes trigger excessive throttling? Now you can define plugin-level limiters for API-call concurrency and request rates. Do you wonder which permissions are needed for a query? The new diagnostic mode will show you, and it'll help you fine-tune your limiters too. Let's see how it all works.
Out of memory? Dial it down.
Steampipe can run anywhere: your laptop, an EC2 instance, a GitHub runner. In some environments, memory is a hard constraint. In Kubernetes, for example, the OOM killer can suddenly and silently shut things down. Good news! Steampipe and its plugins now default to a 1024MB limit. You override any plugin's max memory in its config file.
You can also use the new environment variables STEAMPIPE_MEMORY_MAX_MB to set memory limits for the Steampipe process and STEAMPIPE_PLUGIN_MEMORY_MAX_MB to do the same for plugins.
Overwhelming your network? Dial it down.
Steampipe is massively parallel and we've seen cases where it can take down a home network or reverse proxy. You can now tame that behavior with rate limiters. Here's a simple example that constrains the number of List, Get, and Hydrate functions that the AWS plugin can run in parallel.
See many more examples in the documentation.
Getting throttled? Dial it down.
Steampipe plugins typically use exponential backoff/retry when API calls trigger throttling. Some amount of throttling is normal, but too much can cause problems. Now you can use limiters to keep things calm. Pen testers can fly under the radar, and everyone can avoid being a noisy cloud neighbor.
Those settings will regulate the entire AWS plugin. bucket_size defines how many requests can be made per second, and fill_rate defines how many are added back per-second to refill the bucket. The two settings work together to implement a token-bucket rate limiting algorithm.
What are the optimal values for these settings? It's hard to know in advance what's best for a given API, so you'll need to experiment. Happily, the new diagnostic mode will help you do that!
Diagnostic mode
Use the new STEAMPIPE_DIAGNOSTIC_LEVEL environment variable to turn on Steampipe's version of EXPLAIN ANALYZE. Let's dial down the above settings from 1000 to 10 and run Steampipe in diagnostic mode.
Here's a single row of output.
The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including:
| Key | Description |
|---|---|
type | The type of function (list, get, or hydrate). |
function_name | The name of the function. |
scope_values | A map of scope names to values. This includes the built-in scopes as well as any matrix qualifier scopes and function tags. |
rate_limiters | A list of the rate limiters that are scoped to the function. |
rate_limiter_delay_ms | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (limiter) rate limiting. |
The _ctx column shows that the aws_global limiter is in effect, and that it has delayed two of the sub-API calls that feed the aws_sns_topic table.
Whether or not you've imposed limits, the function_name scope value is a key piece of information. Have you ever struggled to know which permissions are needed to query a particular table? The names of the functions that populate the table correspond to API calls; they'll help you figure that out.
What if you don't want to slow everything down? You can be much more precise. For example, the limits imposed by AWS on APIs can vary by region. To rate-limit just us-east-1 traffic you can do this.
A limiter defined with no scopes applies broadly. When you list one or more values in the scope argument you create a unique limiter for that set of scopes. With the AWS plugin, for example, you could create a limiter scoped to just connection, or connection and region, or any combination of available scopes — it's wildly flexible.
And if the plugin implements function tags, as the AWS plugin does, you can even target a specific function call.
The plugin block
All these examples use the new plugin block to set plugin-level options (in .spc files) that you can bind to connections. Steampipe will create a separate plugin process for each plugin/connection pair.
In this example Steampipe creates two plugin processes. aws_high binds to aws_prod_1 and aws_prod_2, sets a 2000MB maximum, and defines no limiters. aws_low binds to aws_dev_1 and aws_dev_2, sets a 500MB limit, and defines an all_requests limiter.
See it in action
Tune your Steampipe engine
New memory limits can regulate Steampipe's memory consumption overall, or per-plugin. Concurrency and rate limiters can control how plugins use APIs. And diagnostic mode helps you understand and fine-tune the diverse and flexible limiters you can create. Steampipe is a racecar that sometimes goes too fast. With these new mechanisms of control, and unprecedented visibility into the engine, you can now precisely calibrate its speed and optimize its performance. Read all about the new plugin block and limiters, then give it all a try and let us know how it goes!