Find which secrets you still need to rotate in your CircleCI environment

On January 4th, 2023, CircleCI announced they had a breach of customer data. The breach began on or around December 21st. The nature of a CI tool is that it often has access to source code and secrets used to build the software or deploy cloud infrastructure. These secrets are potentially in the hands of bad actors, and CircleCI has advised their customers to rotate everything.

What CircleCI recommends you rotate:

1. OAuth tokens. CircleCI uses these to access GitHub, BitBucket, and GitLab.
2. Project API tokens. These allow systems to interact with CircleCI projects.
3. Project environment variables (circleci_project.env_vars). These can contain secrets or sensitive data related to your application & infrastructure: privileged Cloud API keys for deployment, database connection strings, etc.
4. Context variables. These are environment variables that can be shared across multiple projects.
5. User API tokens. These API tokens, tied to a users, allow scripts to interact with CircleCI
6. Project SSH keys (circleci_project.checkout_keys). These grant access to source code in GitHub.
7. Runner Tokens. The CircleCI SaaS uses theser to interact with self-hosted runners in your environment.

How can Steampipe help?

With the CircleCI plugin installed, you can use Steampipe to query Project environment variables and SSH Keys in the projects and organizations your User API token can access.

To list all the environment variables in all of the Projects the user has access to:

select
  slug,
  username,
  env_vars
from
  circleci_project;

+---------------------------------+---------------+---------------------------------------------------+
| slug                            | username      | env_vars                                          |
+---------------------------------+---------------+---------------------------------------------------+
| gh/fluent-cattle/sp-plugin-test | fluent-cattle | [{"name":"luis","value":"xxxxrked"}]              |
| gh/fluent-cattle/prime-osprey   | fluent-cattle | [{"name":"AWS_ACCESS_KEY_ID","value":"xxxxREAL"}] |
+---------------------------------+---------------+---------------------------------------------------+

To list the SSH Keys used by the Projects the user has access to:

select
  slug as github_repo,
  jsonb_array_elements(checkout_keys) ->> 'preferred' as preferred_key,
  jsonb_array_elements(checkout_keys) ->> 'time' as key_creation_date,
  jsonb_array_elements(checkout_keys) ->> 'type' as type_of_key,
  jsonb_array_elements(checkout_keys) ->> 'login' as key_username
from
  circleci_project;

+---------------------------------+---------------+--------------------------+-----------------+--------------+
| github_repo                     | preferred_key | key_creation_date        | type_of_key     | key_username |
+---------------------------------+---------------+--------------------------+-----------------+--------------+
| gh/fluent-cattle/sp-plugin-test | true          | 2022-12-09T14:57:43.378Z | github-user-key | luisffc      |
| gh/fluent-cattle/sp-plugin-test | false         | 2022-11-21T22:24:17.105Z | deploy-key      | <null>       |
| gh/fluent-cattle/prime-osprey   | true          | 2023-01-04T19:39:50.24Z  | deploy-key      | <null>       |
+---------------------------------+---------------+--------------------------+-----------------+--------------+

Secrets can live in an organization’s Context too. To view all of the environment variables in all of the Contexts you have access to:

select
  c.name as context,
  v.variable,
  v.created_at,
  v.updated_at
from
  circleci_context c
join
  circleci_context_environment_variable v
on
  v.context_id = c.id
where v.created_at < '2022-12-22T00:00:00-00:00'
  and v.updated_at < '2023-01-04T00:00:00-00:00';

+-------------+-------------------+---------------------------+---------------------------+
| context     | variable          | created_at                | updated_at                |
+-------------+-------------------+---------------------------+---------------------------+
| mint-yak    | fair_crab         | 2022-11-17T10:25:28-05:00 | 2023-01-02T10:25:28-05:00 |
| JCF-Context | AWS_ACCESS_KEY_ID | 2022-09-11T15:27:35-05:00 | 2022-11-11T15:27:35-05:00 |
+-------------+-------------------+---------------------------+---------------------------+

With this list, you now know which secrets in your CircleCI environment you still need to rotate.

Other actions you can take

In addition to using Steampipe to identify environment variables in the projects and contexts, you can also identify secrets recommended by CircleCI with their provided tool, or by going into their console:

• OAuth tokens.
  • All GitHub and BitBucket OAuth tokens that existed prior to the incident have been forcefully rotated.
• Project API tokens.
  • CircleCI has already rotated all Project API tokens created prior to 5 January 2023.
  • To rotate them, go to Project Settings > API Permissions > Add API Token.
• User API tokens can be found here.
  • CircleCI has already rotated all User API tokens created prior to 5 January 2023.
• Runner Tokens can be rotated via the CircleCI CLI. Directions can be found in the advisory.

Lessons to learn

As noted in the CircleCI Incident Report, much of the impact of this incident was due to CircleCI’s customers using long-term credentials. Leveraging OIDC and short-term credentials is a security best practice. While it’s not as easy as simply clicking in a cloud-provider console and doing a cut-n-paste, teams who took that effort weren’t forced to scramble to rotate keys late on a Wednesday night.

For the cases where long-term credentials are required, not just used out of convenience, Steampipe can help audit and track them in over 95 cloud services from other SaaS providers like Vercel, Snowflake, and Twilio in addition to the cloud providers like AWS, GCP, and Azure.