A deep dive into AWS Resource Explorer

The promise of the service

For the longest time, the search bar at the top of the AWS console has been tauntingly incomplete. In GCP and Azure, you can search for services and resources, but in AWS, that search bar mainly brings up the services. To find a resource, you had to go to the service page in the region where the resource was created. This created a massive needle-in-the-haystack problem for cloud practitioners who needed to find out where a resource existed.

The cloud community was excited last week when the new Resource Explorer was announced. According to the accompanying blog post, Resource Explorer lets you “search through the AWS resources in your account across Regions using metadata such as names, tags, and IDs.”

But how useful is it?

Let's configure it

First things first, you must configure Resource Explorer in your account. This may seem silly, but one of the first principles of AWS is that AWS will never move data, including metadata, from one AWS region to another without explicit direction from the customer. Enabling Resource Explorer and aggregating all the data into a single region is pretty straightforward. When you first try to search, you’ll be prompted to “Go to Resource Explorer”.

You can then select "Quick Setup" and your "Aggregator Index Region". Click “Turn On Resource Explorer,” and you’re all set.

Once configured, indexing all the resources can take up to 36 hours.

Query Syntax

Resource Explorer has its own Query Syntax. Like Google searches, free text terms are OR’d together unless they appear in double-quotes. You can use a handful of filters like service:, region:, and tag: to limit results.

How does it work in the AWS console? I can enter a basic term and get back a list of all the resources related to that resource.

Some CLI commands

AWS provided some sample queries. What do they look like in the AWS CLI?

Here is the same search for resources related to Splunk that are part of the EC2 service:

aws resource-explorer-2 search --query-string 'Splunk* service:ec2' \
    --query 'Resources[].{name: Properties[].Data[0] | [0].Value, Arn: Arn, Region: Region, ResourceType: ResourceType}'

Returns:

[
    {
        "name": "SplunkServer",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:instance/i-0f005545bd0d430d7",
        "Region": "us-east-1",
        "ResourceType": "ec2:instance"
    },
    {
        "name": "SplunkServer-SecurityGroup",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:security-group/sg-04d2cfe5c65c8993e",
        "Region": "us-east-1",
        "ResourceType": "ec2:security-group"
    },
    {
        "name": "SplunkServer",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:volume/vol-0bd5c063a65627c24",
        "Region": "us-east-1",
        "ResourceType": "ec2:volume"
    }
]

This is a search of all the EC2 Instances in the account:

aws resource-explorer-2 search --query-string 'resourcetype:ec2:instance' \
    --query 'Resources[].{name: Properties[].Data[0] | [0].Value, Arn: Arn, Region: Region}'
[
    {
        "name": "SIFT",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:instance/i-06944d48a6262f7f9",
        "Region": "us-east-1"
    },
    {
        "name": "Turbot",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:instance/i-0315ab2d028051f37",
        "Region": "us-east-1"
    },
    {
        "name": "SplunkServer",
        "Arn": "arn:aws:ec2:us-east-1:354534928996:instance/i-0f005545bd0d430d7",
        "Region": "us-east-1"
    }
]

Unfortunately, the data returned from Resource Explorer doesn’t have critical information, like the instance type, running state, or private IP address.

Limitations

AWS has a practice of putting something out there to get customer feedback. This is clearly the case with Resource Explorer. Several limitations make Resource Explorer only an OK service but not a panacea.

As a part-time incident responder, I often need to know, “What is this private IP address, and where can I find the resources for it?" Sadly, entering a private IP address doesn’t search across the attributes of a resource.

I think the biggest missing feature is search across attributes of a resource, such as IP Address, AMI ID, resource visibility, etc.

It only works on an AWS account scope, not across multiple accounts in an organization. After this was released, a former coworker reached out to ask about the service. I told him you’ve now narrowed the number of consoles from 17,000 to 1,000. Better, but not useful.

Based on AWS’s history of leveraging service-linked roles to support services across an organization, I think we can expect multi-account support as a reasonably fast follow-on.

Only about 18 of the over 200 services AWS offers are currently supported. Like CloudFormation, I suspect that support for Resource Explorer depends on the two-pizza teams that run each service to add their resources to Resource Explorer. If so, we should see more resource types added over the coming months.

Using Resource Explorer in Steampipe

Broad resource coverage is critical for our users, who can use the Steampipe AWS plugin to query 350 AWS resource types.

Unfortunately, given the limitations mentioned above we cannot (yet) fully leverage Resource Explorer as a single source to query and report on AWS resource inventory and configurations.

That said, version 0.83.0 of the Steampipe AWS plugin added support for the aws_resource_explorer_search table. If you’re running an older version of the plugin, upgrade with this command:

steampipe plugin update aws

To search for a resource in Steampipe, you can use the same query syntax AWS provides:

select
  arn,
  region,
  resource_type,
  service,
  owning_account_id
from
  aws_fooli_security.aws_resource_explorer_search
where
  query = 'Splunk* service:ec2'

The details returned are similar to the data from the CLI:

+------------------------------------------------------------------------+-----------+--------------------+---------+-------------------+
| arn                                                                    | region    | resource_type      | service | owning_account_id |
+------------------------------------------------------------------------+-----------+--------------------+---------+-------------------+
| arn:aws:ec2:us-east-1:354534928996:volume/vol-0bd5c063a65627c24        | us-east-1 | ec2:volume         | ec2     | 354534928996      |
| arn:aws:ec2:us-east-1:354534928996:instance/i-0f005545bd0d430d7        | us-east-1 | ec2:instance       | ec2     | 354534928996      |
| arn:aws:ec2:us-east-1:354534928996:security-group/sg-04d2cfe5c65c8993e | us-east-1 | ec2:security-group | ec2     | 354534928996      |
+------------------------------------------------------------------------+-----------+--------------------+---------+-------------------+

This table works with aggregators, solving for the issue of multi-account support. Note: you must manually configure Resource Explorer indexes in each account before this will work. If there are accounts without a Resource Explorer aggregator index, you will see the Resource Explorer 2: ListIndexes as shown in this next example.

> select
  arn,
  region,
  resource_type,
  service,
  owning_account_id
from
  aws_resource_explorer_search
where
  query = '"fooli-audit"*'

Error: operation error Resource Explorer 2: ListIndexes, https response error StatusCode: 403, RequestID: cbc86699-0ea2-4cab-8bbb-a7dc8852e342, AccessDeniedException:
(SQLSTATE HV000)

+--------------------------------------------+--------+---------------+---------+-------------------+
| arn                                        | region | resource_type | service | owning_account_id |
+--------------------------------------------+--------+---------------+---------+-------------------+
| arn:aws:iam::755489496295:role/fooli-audit | global | iam:role      | iam     | 755489496295      |
| arn:aws:iam::354534928996:role/fooli-audit | global | iam:role      | iam     | 354534928996      |
| arn:aws:iam::549934280147:role/fooli-audit | global | iam:role      | iam     | 549934280147      |
+--------------------------------------------+--------+---------------+---------+-------------------+

Time: 0.7s. Rows fetched: 3. Hydrate calls: 0.

If there are more than 1000 results, Resource Explorer will truncate the results and you will need to refine your query.

More examples are available in the table documentation.

While Resource Explorer can't always find the needle in the AWS haystack, it might get there someday, and meanwhile will be welcomed as a better way for users of the AWS console to search across AWS accounts.

Please give the new Steampipe AWS Resource Explorer table a try and let us know what you think! We'd welcome any suggestions or contributions to better leverage Resource Explorer and expand our coverage across AWS.