Steampipe unbundled →
Steampipe logo

Plugin Release Checklist

As of October 2023, we've absorbed 135+ plugins into the Hub. If you want to contribute one -- and we hope you do! -- here are the most common things we ask contributors to check to prepare for the plugin's release. Feel free to tick the boxes as you go through the list!

Basic Configuration

Repository name

The repository name should use the format steampipe-plugin-<pluginName>, e.g., steampipe-plugin-aws, steampipe-plugin-googledirectory, steampipe-plugin-microsoft365. The plugin name should be one word, so there are always 3 parts in the repository name.

Repository topics

To help with discoverability in GitHub, the repository topics should include:

  • postgresql
  • postgresql-fdw
  • sql
  • steampipe
  • steampipe-plugin
Repository website

The repository website/homepage should link to the Hub site. The URL is composed of the GitHub organization and plugin name, for instance:

Go version

The Go version in go.mod and any workflows is 1.21.

.goreleaser.yml

The .goreleaser.yml file uses the standard format, e.g., AWS plugin .goreleaser.yml.

CHANGELOG

A CHANGELOG.md is included and contains release notes for the upcoming version (typically v0.0.1).

License

The plugin uses the Apache License 2.0.

Makefile

The Makefile file is present and builds to the correct plugin path.

Configuration File

.spc examples

The config/PLUGIN.spc file is neatly formatted, and explains each argument with links as appropriate, using realistic values, e.g., "xoxp-abcads…" instead of "TOKEN_HERE".

Environment variables

Arguments that can also be set via environment variable include the environment variable name(s) in their descriptions.

Credentials

Terraform compatibility

If there's a Terraform provider for your API, the plugin supports the same credential methods as the provider.

Existing CLI credentials

When there are commonly used CLI credentials, like .aws/credentials, the plugin works with them.

Expiry

When credentials expire, and the API's SDK does not automatically refresh them, the plugin alerts the user and tells them how to refresh.

Environment variables

It's possible to set credentials using an environment variable if the API's SDK also supports using environment variables.

Table and Column Names

Standard names

All table and column names follow our Table & Column Naming Standards.

Table and Column Descriptions

Descriptions

Every table and column has a description. These are consistent across tables.

Other standards

All descriptions adhere to the Table and Column Descriptions Standards.

Table and Column Design

Global and per-authenticated-user data

Many plugins can return both global data, e.g., all GitHub repos or Google Drive files, and data only for the authenticated user (my repos, my files). If that's the case, there are separate tables, e.g. github_repository and github_my_repository.

Common columns

If tables share columns, these are abstracted as shown in the AWS plugin's common_columns.go.

Required configuration arguments

The plugin checks required configuration arguments are set once at load time.

Logging

Error info

When the plugin returns an error, it includes the location and any related args, along with the error itself. See example.

Data Ingestion

Default transform

The plugin sets a preferred transform as the default. For example, the GitLab plugin uses DefaultTransform: transform.FromGo().NullIfZero(). Please see Transform Functions for a full list of transform functions.

Pagination

The plugin implements pagination in each table's List function supported by the API's SDK. If pagination is implemented, the plugin sets the page size per request to the maximum allowed; however, if QueryContext.Limit is smaller than that page size, the page size should be set to the limit. See example.

Hydrate function pagination

If a non-List hydrate function requires paging, consider separating that data into a separate table. Columns that require separate hydrate data that uses paging can lead to throttling and rate limiting errors unexpectedly.

Backoff and retry

If the API SDK doesn't automate backoff and retry, the plugin leverages capabilities of the Steampipe plugin SDK's RetryHydrate function. For instance, the github_issue table uses this function when listing issues due to the strict throttling of the GitHub API.

Maximum concurrency

If the API has strict rate limiting, the table sets HydrateConfig.MaxConcurrency for the relevant hydrate functions. For instance, the googleworkspace_gmail_message table limits the number of getGmailMessage calls.

Context cancellation

Each table's list hydrate function checks for remaining rows from the API SDK, and aborts inside loops (e.g., while streaming items) if there are none. (See example.)

Column Types

Money

Money is represented as a string, not a double which is never exact.

Dynamic Tables

Specifying tables to generate

If the plugin can generate dynamic tables, a configuration argument should allow users to specify which tables the plugin will generate. This configuration argument typically accepts a list of strings and should support filesystem glob patterns like in the CSV plugin.

If this configuration argument is not set or is explicitly empty, e.g., paths = [], then no dynamic tables should be generated.

Default tables

The plugin should determine if it will generate dynamic tables by default after plugin installation based on if the configuration argument mentioned above is commented by default. For instance, in the Prometheus plugin, the metrics configuration argument is commented. After plugin installation, the plugin will not generate dynamic tables unless the user adds a non-commented value for metrics.

You may not want to load dynamic tables by default if it drastically increases the plugin initialization time due to the number of tables.

Table name prefixes

When naming dynamic tables, the plugin name prefix, e.g., kubernetes_, should be added if it helps avoid namespace collisions or if it helps group them with static tables that share the same prefix.

Documentation

Index Documentation

Front matter

The index document contains a front matter block, like the one below:

---
organization: Turbot
category: ["security"]
icon_url: "/images/plugins/turbot/duo.svg"
brand_color: "#6BBF4E"
display_name: Duo Security
name: duo
description: Steampipe plugin for querying Duo Security users, logs and more.
og_description: Query Duo Security with SQL! Open source CLI. No DB required.
og_image: "/images/plugins/turbot/duo-social-graphic.png"
---
Front matter: category

The category is an appropriate choice from the list at hub.steampipe.io/plugins.

Front matter: icon_url

The icon URL is a link to an .svg file hosted on hub.steampipe.io. Please request an icon through the Turbot Community Slack and a URL will be provided to use in this variable.

Front matter: brand color

The color matches the provider's brand guidelines, typically stated on a page like this one for Twilio.

Plugin description

The description in docs/index.md is appropriate for the provider. The AWS plugin, for example, uses:

AWS provides on-demand cloud computing platforms and APIs to authenticated customers on a metered pay-as-you-go basis.

The opening sentence of the Wikipedia page for the provider can be a good source of guidance here.

Credentials

Credentials are the most important piece of documentation. The plugin:

  • Explains scopes and required permissions
  • Links to provider documentation
  • Explains how to use existing CLI creds when that's possible
Aggregator examples

For plugins that benefit from using multiple connections and aggregators, like the AWS plugin, one or more H2 sections with examples should be added to the index document so users can easily reference it. For instance, the AWS plugin has examples in AWS Multi-Account Connections.

If a plugin doesn't strongly benefit from aggregator connections, an H2 section called Multiple Connections should be added that briefly talks about aggregators and has a link to Using Aggregators.

Table Documentation

Useful examples

Each table document shows 4 - 5 useful examples that reflect real-world scenarios. Please see Writing Example Queries for common patterns and samples.

Column specificity

Most examples specify columns. Using SELECT * is OK for one or two things, but generally not preferred as it can produce too much data to be helpful. See also When Not to SELECT *.

Required columns

If some columns are required, these are called out and explained.

Final Review

Testing

The plugin has been tested on a real account with substantial data. Please note that errors and API throttling issues may not appear when using a test account with little data.

Matching query examples

The example in README.md matches the one in docs/index.md.

Matching config examples

The example in config/PLUGIN.spc matches the one in docs/index.md#configuration.

Social graphic

The social graphic is included at the top of the README file and is uploaded to the Social preview feature in the GitHub repository. Please request a social graphic through the Steampipe Slack.

Ease of first use

The plugin really nails easy setup, there's a short path to a first successful query, and it runs quickly.

Pre-mortem

You've considered, and addressed, reasons why this plugin could fail to delight its community.