# v0.21.0: Dials for tuning Steampipe > Memory management, rate limiting, and diagnostics to keep your Steampipe engine running smoothly at scale. By Steampipe Team Published: 2023-10-02 Do you run Steampipe at scale, and/or in constrained environments? Now you can control the amount of memory Steampipe and its plugins can use. Do your queries sometimes trigger excessive throttling? Now you can define plugin-level limiters for API-call concurrency and request rates. Do you wonder which permissions are needed for a query? The new diagnostic mode will show you, and it'll help you fine-tune your limiters too. Let's see how it all works. ## Out of memory? Dial it down. Steampipe can run anywhere: your laptop, an EC2 instance, a GitHub runner. In some environments, memory is a hard constraint. In Kubernetes, for example, the [OOM killer](https://medium.com/cloud-native-daily/title-demystifying-oom-killer-in-kubernetes-tracking-down-memory-issues-b5a4973fbd56) can suddenly and silently shut things down. Good news! Steampipe and its plugins now default to a 1024MB limit. You override any plugin's max memory in its [config file](https://steampipe.io/docs/managing/connections#connection-configuration-files). ```hcl plugin "aws" { memory_max_mb = 2048 # megabytes } ``` You can also use the new environment variables [STEAMPIPE_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_memory_max_mb) to set memory limits for the Steampipe process and [STEAMPIPE_PLUGIN_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_plugin_memory_max_mb) to do the same for plugins. ## Overwhelming your network? Dial it down. Steampipe is [massively parallel](https://aws.amazon.com/blogs/opensource/querying-aws-at-scale-across-apis-regions-and-accounts/) and we've seen cases where it can take down a home network or reverse proxy. You can now tame that behavior with rate limiters. Here's a simple example that constrains the number of [List, Get, and Hydrate functions](/docs/develop/writing-plugins#hydrate-functions) that the AWS plugin can run in parallel. ```hcl plugin "aws" { limiter "max_concurrency" { max_concurrency = 200 } } ``` See many more examples in the [documentation](/docs/reference/config-files/plugin). ## Getting throttled? Dial it down. Steampipe plugins typically use exponential backoff/retry when API calls trigger throttling. Some amount of throttling is normal, but too much can cause problems. Now you can use [limiters](/docs/guides/limiter) to keep things calm. Pen testers can fly under the radar, and everyone can avoid being a noisy cloud neighbor. ```hcl plugin "aws" { limiter "aws_global" { bucket_size = 1000 fill_rate = 1000 } } ``` Those settings will regulate the entire AWS plugin. `bucket_size` defines how many requests can be made per second, and `fill_rate` defines how many are added back per-second to refill the bucket. The two settings work together to implement a [token-bucket](https://en.wikipedia.org/wiki/Token_bucket) rate limiting algorithm. What are the optimal values for these settings? It's hard to know in advance what's best for a given API, so you'll need to experiment. Happily, the new diagnostic mode will help you do that! ## Diagnostic mode Use the new [STEAMPIPE_DIAGNOSTIC_LEVEL](/docs/reference/env-vars/steampipe_diagnostic_level) environment variable to turn on Steampipe's version of [EXPLAIN ANALYZE](https://www.postgresql.org/docs/current/sql-explain.html). Let's dial down the above settings from 1000 to 10 and run Steampipe in diagnostic mode. ``` $ STEAMPIPE_DIAGNOSTIC_LEVEL=ALL steampipe query > select jsonb_pretty(_ctx) as _ctx, display_names, tags from aws_sns_topic ``` Here's a single row of output. ``` +------------------------------------------------------------+--------------+--------+ | _ctx | display_name | tags | +------------------------------------------------------------+--------------+--------+ | { | | | | "diagnostics": { | | | | "calls": [ | | | | { | | | | "type": "list", | | | | "scope_values": { | | | | "table": "aws_sns_topic", | | | | "action": "ListTopics", | | | | "region": "us-east-1", | | | | "service": "sns", | | | | "connection": "sil", | | | | "function_name": "listAwsSnsTopics" | | | | }, | | | | "function_name": "listAwsSnsTopics", | | | | "rate_limiters": [ | | | | "aws_global" | | | | ], | | | | "rate_limiter_delay_ms": 0 | | | | }, | | | | { | | | | "type": "hydrate", | | | | "scope_values": { | | | | "table": "aws_sns_topic", | | | | "action": "GetTopicAttributes", | | | | "region": "us-east-1", | | | | "service": "sns", | | | | "connection": "sil", | | | | "function_name": "" | | | | }, | | | | "function_name": "getTopicAttributes", | | | | "rate_limiters": [ | | | | "aws_global" | | | | ], | | | | "rate_limiter_delay_ms": 5295 | | | | }, | | | | { | | | | "type": "hydrate", | | | | "scope_values": { | | | | "table": "aws_sns_topic", | | | | "action": "ListTagsForResource", | | | | "region": "us-east-1", | | | | "service": "sns", | | | | "connection": "sil", | | | | "function_name": "listTagsForSnsTopic" | | | | }, | | | | "function_name": "listTagsForSnsTopic", | | | | "rate_limiters": [ | | | | "aws_global" | | | | ], | | | | "rate_limiter_delay_ms": 5401 | | | | } | | | | ] | | | | }, | | | | "connection_name": "sil" | | | | } | | | ``` The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including: | Key | Description |-------------------------|---------------------- | `type` | The type of function (`list`, `get`, or `hydrate`). | `function_name` | The name of the function. | `scope_values` | A map of scope names to values. This includes the built-in scopes as well as any matrix qualifier scopes and function tags. | `rate_limiters` | A list of the rate limiters that are scoped to the function. | `rate_limiter_delay_ms` | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (`limiter`) rate limiting. The `_ctx` column shows that the `aws_global` limiter is in effect, and that it has delayed two of the sub-API calls that feed the `aws_sns_topic` table. Whether or not you've imposed limits, the `function_name` scope value is a key piece of information. Have you ever struggled to know which permissions are needed to query a particular table? The names of the functions that populate the table correspond to API calls; they'll help you figure that out. What if you don't want to slow everything down? You can be much more precise. For example, the limits imposed by AWS on APIs can vary by region. To rate-limit just `us-east-1` traffic you can do this. ```hcl plugin "aws" { limiter "us-east-1" { bucket_size = 10 fill_rate = 10 scope = [ "region" ] where = "'region = 'us-east-1'" } } ``` A limiter defined with no scopes applies broadly. When you list one or more values in the `scope` argument you create a [unique limiter](/docs/guides/limiter#defining-limiters) for that set of scopes. With the AWS plugin, for example, you could create a limiter scoped to just `connection`, or `connection` and `region`, or any combination of available scopes — it's wildly flexible. And if the plugin implements [function tags](/docs/develop/writing-plugins#function-tags), as the AWS plugin does, you can even target a specific function call. ```hcl plugin "aws" { limiter "aws-sns-topic-us-east-1-" { bucket_size = 10 fill_rate = 10 scope = [ "region" ] where = "'region = 'us-east-1' and action = 'ListTagsForResource'" } } ``` ## The `plugin` block All these examples use the new [plugin block](/docs/reference/config-files/plugin#plugins-and-connections) to set plugin-level options (in `.spc` files) that you can bind to connections. Steampipe will create a separate plugin process for each `plugin`/`connection` pair. In this example Steampipe creates two plugin processes. `aws_high` binds to `aws_prod_1` and `aws_prod_2`, sets a 2000MB maximum, and defines no limiters. `aws_low` binds to `aws_dev_1` and `aws_dev_2`, sets a 500MB limit, and defines an `all_requests` limiter. ```hcl plugin "aws_high" { memory_max_mb = 2000 source = "aws" } plugin "aws_low" { memory_max_mb = 500 source = "aws" limiter "all_requests" { bucket_size = 100 fill_rate = 100 max_concurrency = 50 } } connection "aws_prod_1" { plugin = plugin.aws_high profile = "prod1" regions = ["*"] } connection "aws_prod_2" { plugin = plugin.aws_high profile = "prod2" regions = ["*"] } connection "aws_dev_1" { plugin = plugin.aws_low profile = "dev1" regions = ["*"] } connection "aws_dev_2" { plugin = plugin.aws_low profile = "dev2" regions = ["*"] } ``` ## See it in action
## Tune your Steampipe engine New memory limits can regulate Steampipe's memory consumption overall, or per-plugin. Concurrency and rate limiters can control how plugins use APIs. And diagnostic mode helps you understand and fine-tune the diverse and flexible limiters you can create. Steampipe is a racecar that sometimes goes too fast. With these new mechanisms of control, and unprecedented visibility into the engine, you can now precisely calibrate its speed and optimize its performance. Read all about the new [plugin block](/docs/reference/config-files/plugin) and [limiters](/docs/guides/limiter), then give it all a try and [let us know](https://steampipe.io/community/join) how it goes!