Concurrency & Rate Limiting

Steampipe is designed to be fast. It provides parallel execution at multiple layers:

• It runs controls in parallel.
• It runs queries in parallel.
• For a given query it runs List, Get, and Hydrate functions in parallel.

This high degree of concurrency results in low latency and high throughput, but may at times overwhelm the underlying service or API. Features like exponential back-off & retry and caching markedly improve the situation, but at large scale you may still run out of local or remote resources. The steampipe limiter was created to help solve these types of problems. Limiters provide a simple, flexible interface to implement client-site rate limiting and concurrency thresholds at compile time or run time. You can use limiters to:

• Smooth the request rate from steampipe to reduce load on the remote API or service
• Limit the number of parallel requests to reduce contention for client and network resources
• Avoid hitting server limits and throttling

Defining limiters

Limiters may be defined in Go code and compiled into a plugin, or they may be defined in HCL in .spc configuration files. In either case, the possible settings are the same.

Each limiter must have a name. In the case of an HCL definition, the label on the limiter block is used as the rate limiter name. For a limiter defined in Go, you must include a Name.

A limiter may specify a max_concurrency which sets a ceiling on the number of List, Get, and Hydrate functions that can run in parallel.

# run up to 250 hydrate/list/get functions concurrently
plugin "aws" {
  limiter "aws_global_concurrency" {
    max_concurrency = 250 
  }
}

A limiter may also specify a bucket_size and fill_rate to limit the rate at which List, Get, and Hydrate functions may run. The rate limiter uses a token-bucket algorithm, where the bucket_size specifies the maximum number of tokens that may accrue (the burst size) and the fill_rate specifies how many tokens are refilled each second.

plugin "aws" {
  # run up to 1000 hydrate/list/get functions per second
  limiter "aws_global_rate_limit" {
    bucket_size = 1000
    fill_rate   = 1000
  }
}

Every limiter has a scope. The scope defines the context for the limit: which resources are subject to / counted against the limit. There are built-in scopes for connection, table, function_name, and any matrix qualifiers that the plugin may include. A plugin author may also add function tags that can also be used as scopes.

If no scope is specified, then the limiter applies to all functions in the plugin. For instance, this limiter will allow 1000 hydrate/list/get functions per second across all connections:

plugin "aws" {
  # run up to 1000 hydrate/list/get functions per second across all aws connections
  limiter "aws_regional_rate_limit" {
    bucket_size = 1000
    fill_rate   = 1000
  }
}

If you specify a list of scopes, then a limiter instance is created for each unique combination of scope values. It acts much like group by in a SQL statement.

For example, to limit to 1000 hydrate/list/get functions per second in each region of each connection:

plugin "aws" {

  # run up to 1000 hydrate/list/get functions per second in each region of each connection
  limiter "aws_regional_rate_limit" {
    bucket_size = 1000
    fill_rate   = 1000
    scope       = ["connection", "region"]
  }
}

You can use a where clause to further filter the scopes to specific values. For example, we can restrict the limiter so that it only applies to a specific region:

plugin "aws" {
  # run up to 1000 hydrate/list/get functions per second in us-east-1 for each connection
  limiter "aws_rate_limit_us_east_1" {
    bucket_size = 1000
    fill_rate   = 1000
    scope       = ["connection", "region"]
    where       = "region = 'us-east-1'"
  }
}

You can define multiple limiters. If a function is included in the scope of multiple rate limiters, they will all apply. The function will wait until every rate limiter that applies to it has available bucket tokens and is below its max concurrency.

plugin "aws" {

  # run up to 250 functions concurrently across all connections
  limiter "aws_global_concurrency" {
    max_concurrency = 250
  }

  # run up to 1000 functions per second in us-east-1 for each connection
  limiter "aws_rate_limit_us_east_1" {
    bucket_size = 1000
    fill_rate   = 1000
    scope       = ["connection", "region"]
    where       = "region = 'us-east-1'"
  }

  # run up to 200 functions per second in regions OTHER than us-east-1
  # for each connection
  limiter "aws_rate_limit_non_us_east_1" {
    bucket_size = 200
    fill_rate   = 200
    scope       = ["connection", "region"]
    where       = "region <> 'us-east-1'"
  }
}

Function Tags

Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation.

Function tags must be added in the plugin code by the plugin author. Once the tags are added to the plugin, you can use them in the scope and where arguments for your rate limiter.

plugin "aws" {
  limiter "sns_get_topic_attributes_us_east_1" {
    bucket_size  = 3000 
    fill_rate    = 3000

    scope  = ["connection", "region", "service", "action"]
    where  = "action = 'GetTopicAttributes' and service = 'sns' and region = 'us-east-1' "
  }
}

You can view the available tags in the scope_values when in diagnostic mode. For example, to see the tags in the aws_sns_topic table:

with one_row as materialized (
  select * from aws_sns_topic limit 1
)
select 
  c ->> 'function_name' as function_name,
  jsonb_pretty(c -> 'scope_values') as scope_values
from 
  one_row,
  jsonb_array_elements(_ctx -> 'diagnostics' -> 'calls') as c

+-------------------------------+--------------------------------------------+
| function_name                 | scope_values                               |
+-------------------------------+--------------------------------------------+
| listAwsSnsTopics              | {                                          |
|                               |     "table": "aws_sns_topic",              |
|                               |     "action": "ListTopics",                |
|                               |     "region": "us-east-1",                 |
|                               |     "service": "sns",                      |
|                               |     "connection": "aws_dmi",               |
|                               |     "function_name": "listAwsSnsTopics"    |
|                               | }                                          |
| listTagsForSnsTopic           | {                                          |
|                               |     "table": "aws_sns_topic",              |
|                               |     "action": "ListTagsForResource",       |
|                               |     "region": "us-east-1",                 |
|                               |     "service": "sns",                      |
|                               |     "connection": "aws_dmi",               |
|                               |     "function_name": "listTagsForSnsTopic" |
|                               | }                                          |
| listRegionsForServiceUncached | {                                          |
|                               |     "table": "aws_sns_topic",              |
|                               |     "region": "us-east-1",                 |
|                               |     "connection": "aws_dmi"                |
|                               | }                                          |
| getTopicAttributes            | {                                          |
|                               |     "table": "aws_sns_topic",              |
|                               |     "action": "GetTopicAttributes",        |
|                               |     "region": "us-east-1",                 |
|                               |     "service": "sns",                      |
|                               |     "connection": "aws_dmi",               |
|                               |     "function_name": ""                    |
|                               | }                                          |
+-------------------------------+--------------------------------------------+

Exploring & Troubleshooting with Diagnostic Mode

To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostic Mode. To enable Diagnostic Mode, set the STEAMPIPE_DIAGNOSTIC_LEVEL environment variable to ALL when you start the Steampipe DB:

STEAMPIPE_DIAGNOSTIC_LEVEL=all  steampipe service start

With diagnostics enabled, the _ctx column will contain information about what functions were called to fetch the row, the scope values (including any tags) for the function, the limiters that were in effect, and the amount of time the request was delayed by the limiters. This diagnostic information can help you discover what scopes are available to use in limiters as well as to see the effect and impact of limiters that you have defined.

select jsonb_pretty(_ctx) as _ctx ,display_name from aws_sns_topic limit 2

+-----------------------------------------------------------+--------------+
| _ctx                                                      | display_name |
+-----------------------------------------------------------+--------------+
| {                                                         |              |
|     "diagnostics": {                                      |              |
|         "calls": [                                        |              |
|             {                                             |              |
|                 "type": "list",                           |              |
|                 "scope_values": {                         |              |
|                     "table": "aws_sns_topic",             |              |
|                     "action": "ListTopics",               |              |
|                     "region": "us-east-2",                |              |
|                     "service": "sns",                     |              |
|                     "connection": "aws_dmi",              |              |
|                     "function_name": "listAwsSnsTopics"   |              |
|                 },                                        |              |
|                 "function_name": "listAwsSnsTopics",      |              |
|                 "rate_limiters": [                        |              |
|                     "aws_global",                         |              |
|                     "sns_list_topics"                     |              |
|                 ],                                        |              |
|                 "rate_limiter_delay_ms": 0                |              |
|             },                                            |              |
|             {                                             |              |
|                 "type": "hydrate",                        |              |
|                 "scope_values": {                         |              |
|                     "table": "aws_sns_topic",             |              |
|                     "action": "GetTopicAttributes",       |              |
|                     "region": "us-east-2",                |              |
|                     "service": "sns",                     |              |
|                     "connection": "aws_dmi",              |              |
|                     "function_name": ""                   |              |
|                 },                                        |              |
|                 "function_name": "getTopicAttributes",    |              |
|                 "rate_limiters": [                        |              |
|                     "sns_get_topic_attributes_150",       |              |
|                     "aws_global"                          |              |
|                 ],                                        |              |
|                 "rate_limiter_delay_ms": 808              |              |
|             }                                             |              |
|         ]                                                 |              |
|     },                                                    |              |
|     "connection_name": "aws_dmi"                          |              |
| }                                                         |              |
| {                                                         |              |
|     "diagnostics": {                                      |              |
|         "calls": [                                        |              |
|             {                                             |              |
|                 "type": "list",                           |              |
|                 "scope_values": {                         |              |
|                     "table": "aws_sns_topic",             |              |
|                     "action": "ListTopics",               |              |
|                     "region": "us-east-1",                |              |
|                     "service": "sns",                     |              |
|                     "connection": "aws_dmi",              |              |
|                     "function_name": "listAwsSnsTopics"   |              |
|                 },                                        |              |
|                 "function_name": "listAwsSnsTopics",      |              |
|                 "rate_limiters": [                        |              |
|                     "aws_global",                         |              |
|                     "sns_list_topics"                     |              |
|                 ],                                        |              |
|                 "rate_limiter_delay_ms": 597              |              |
|             },                                            |              |
|             {                                             |              |
|                 "type": "hydrate",                        |              |
|                 "scope_values": {                         |              |
|                     "table": "aws_sns_topic",             |              |
|                     "action": "GetTopicAttributes",       |              |
|                     "region": "us-east-1",                |              |
|                     "service": "sns",                     |              |
|                     "connection": "aws_dmi",              |              |
|                     "function_name": ""                   |              |
|                 },                                        |              |
|                 "function_name": "getTopicAttributes",    |              |
|                 "rate_limiters": [                        |              |
|                     "sns_get_topic_attributes_us_east_1", |              |
|                     "aws_global"                          |              |
|                 ],                                        |              |
|                 "rate_limiter_delay_ms": 0                |              |
|             }                                             |              |
|         ]                                                 |              |
|     },                                                    |              |
|     "connection_name": "aws_dmi"                          |              |
| }                                                         |              |
+-----------------------------------------------------------+--------------+

The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including:

Viewing and Overriding Limiters

Steampipe includes the steampipe_plugin_limiter table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL.

select name,plugin,source_type,status,bucket_size,fill_rate,max_concurrency from steampipe_plugin_limiter

+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+
| name                               | plugin                                      | source_type | status | bucket_size | fill_rate | max_concurrency |
+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+
| exec_max_concurrency_limiter       | hub.steampipe.io/plugins/turbot/exec@latest | plugin      | active | <null>      | <null>    | 15              |
| sns_get_topic_attributes_150       | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 150         | 150       | <null>          |
| sns_get_topic_attributes_30        | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 30          | 30        | <null>          |
| aws_global                         | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 10          | 10        | <null>          |
| sns_list_topics                    | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 30          | 30        | <null>          |
| sns_list_tags_for_resource         | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 10          | 10        | <null>          |
| sns_get_topic_attributes_us_east_1 | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 3000        | 3000      | <null>          |
| sns_get_topic_attributes_900       | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 900         | 900       | <null>          |
+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+

You can override a limiter that is compiled into a plugin by creating an HCL limiter with the same name. In the previous example, we can see that the exec plugin includes a default limiter named exec_max_concurrency_limiter that sets the max_concurrency to 15. We can override this value at run time by creating an HCL limiter for this plugin with the same name. The limiter block must be contained in a plugin block. Like connection, Steampipe will load all plugin blocks that it finds in any .spc file in the ~/.steampipe/config directory. For example, we can add the following snippet to the ~/.steampipe/config/exec.spc file:

plugin "exec" {
   limiter "exec_max_concurrency_limiter" {
      max_concurrency = 20
   }
}

Querying the steampipe_plugin_limiter table again, we can see that there are now 2 rate limiters for the exec plugin named exec_max_concurrency_limiter, but the one from the plugin is overridden by the one in the config file.

select name,plugin,source_type,status,bucket_size,fill_rate,max_concurrency from steampipe_plugin_limiter

+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+
| name                               | plugin                                      | source_type | status | bucket_size | fill_rate | max_concurrency |
+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+
| exec_max_concurrency_limiter       | hub.steampipe.io/plugins/turbot/exec@latest | plugin      | active | <null>      | <null>    | 15              |
| exec_max_concurrency_limiter       | hub.steampipe.io/plugins/turbot/exec@latest | config      | active | <null>      | <null>    | 20              |
| aws_global                         | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 10          | 10        | <null>          |
| sns_list_topics                    | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 30          | 30        | <null>          |
| sns_list_tags_for_resource         | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 10          | 10        | <null>          |
| sns_get_topic_attributes_us_east_1 | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 3000        | 3000      | <null>          |
| sns_get_topic_attributes_900       | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 900         | 900       | <null>          |
| sns_get_topic_attributes_150       | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 150         | 150       | <null>          |
| sns_get_topic_attributes_30        | hub.steampipe.io/plugins/turbot/aws@latest  | config      | active | 30          | 30        | <null>          |
+------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+

Hints, Tips, & Best practices

• You can use ANY scope in the where, even if it does not appear in the scope for the limiter. Remember that the scope defines the grouping; it acts similar to group by in SQL. Consider the following rate limiter:

  plugin "aws" {
    limiter "aws_sns_read_rate_limit" {
      bucket_size = 2500
      fill_rate   = 2500
      scope       = ["connection", "region", "service", "action"]
      where       = "service = 'sns' and (action like 'Get%' or action like 'List%') "
    }
  }

  This will create a separate rate limiter instance for every action in the sns service in every region of every account - You can do 2500 GetTopicAttributes requests/sec in each account/region, and also 2500 ListTagsForResource requests/sec in each account/region, and also 2500 ListTopics requests/sec in each account/region.

  If we remove action from the scope, there will be one rate limiter instance for all actions in the sns service in each region/account - You can do 2500 total GetTopicAttributes or ListTagsForResource or ListTopics requests per second in each account/region.

  plugin "aws" {
    limiter "aws_sns_read_rate_limit" {
      bucket_size = 2500
      fill_rate   = 2500
      scope       = ["connection", "region", "service"]
      where       = "service = 'sns' and (action like 'Get%' or action like 'List%') "
    }
  }

• Setting max_concurrency at the plugin level can help prevent running out of local resources like network bandwidth, ports, file handles, etc.

  plugin "aws" {
    max_concurrency = 250
  }

• Optimizing rate limiters requires knowledge of how the API is implemented. If the API publishes information about what the rate limits are, and how they are applied, that's a good starting point for setting your bucket_size and fill_rate values. Getting the limiter values right usually involves some trial and error though, and simply setting max_concurrency is often good enough to get past a problem.

• Use the plugin logs (~/.steampipe/logs/plugin*.log) to verify that the rate limiters are reducing the throttling and other errors from the API as you would expect.

• Use the steampipe_plugin_limiter table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use STEAMPIPE_DIAGNOSTIC_LEVEL=ALL to enable extra diagnostic info in the _ctx to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the STEAMPIPE_DIAGNOSTIC_LEVEL variable must be set in the database service process - if you run steampipe as a service, it must be set when you run steampipe service start

• Throttling errors from the server, such as 429 Too Many Requests, are not inherently bad. Most cloud SDKs actually account for retrying such errors and expect that it will sometimes occur. Steampipe plugins generally implement an exponential back-off & retry to account for such cases. You can use client side limiters to help avoid resource contention and to reduce throttling from the server, but completely avoiding server-side throttling is probably not necessary in most cases.