Introducing relationship graphs

Steampipe's ability to query cloud assets -- rapidly, at scale, across accounts and regions -- can deliver a flood of information. To help you make sense of it we added dashboards like AWS Insights to summarize the data using charts and tables, navigate from overviews like the AWS EC2 Instance Dashboard to detail views like AWS EC2 Instance Detail, and see relationships as flow diagrams.

The new relationship graphs in v0.18 turbocharge Steampipe's ability to help you see and navigate relationships among your cloud assets. Consider this EC2 instance:

You can immediately visualize the instance with its full context including volumes, key pairs, network topology and IAM permissions.

Hovering resources in the graph allows you to peek at their details, or click the link (e.g. the VPC) to see a graph of that resource and its own context.

Context is king

Relationship graphs make your cloud simpler and immediately highlight important information.

• Understand relationships and dependencies at a glance: What services log to this S3 bucket? What instances are targeted by this load balancer? What IAM role is used by this Lambda function? Does this VPC use peering?

• Review architecture and topology: What is the network structure of this VPC? Is this application balanced across availability zones? Is my RDS instance in the database subnet?

• Identity and permissions: What users are in this group? What resources use this IAM role? What permissions are granted by this principal?

For example, here we visualize the details of an IAM policy and can immediately see if least privilege has been followed or not:

Massive mod updates

As always in Steampipe, the batteries are included. Today we're also releasing updated Insights mods for AWS, Azure & GCP that include relationship graphs for all the resources you care about in those clouds.

Snapshot your graphs

Graphs are just panels in existing Steampipe dashboards. They load from live data and can be saved as snapshots. This means you can create a graph of your AWS EC2 instances, save it, share it with your team, then see it change over time.

Dashboards as code

The work we've done to enhance these dashboards is open source so you can inspect, modify, and remix. To simplify the creation of relationship graphs -- for ourselves and for you -- we've added two new HCL blocks to represent the nodes and edges of these graphs. A node represents one or more cloud resources: compute instances, virtual networks, roles, volumes. Here's a node for an EC2 instance.

node "ec2_instance" {
  category = category.ec2_instance
  sql = <<EOQ
    select
      arn as id,
      title,
      jsonb_build_object(
        'Instance ID', instance_id,
        'Name', tags ->> 'Name',
        'ARN', arn,
        'Account ID', account_id,
        'Region', region
      ) as properties
    from
     aws_ec2_instance
  EOQ
}

It's a query that returns, for a set of instances, the columns id, title, and properties. The title column makes the node legible on a graph, and properties supplies more contextual info, but only id is required. That's the value that connects this node to other kinds of nodes by way of an edge. The category argument points to a definition that controls the node's icon and color and defines its link, in this case to the instance's detail view.

category "ec2_instance" {
  title = "EC2 Instance"
  color = local.compute_color
  href  = "/aws_insights.dashboard.ec2_instance_detail?input.instance_arn={{.properties.'ARN' | @uri}}"
  icon  = "memory"
}

Here's an edge that connects an EC2 instance to a volume that it mounts. Again it's a query, in this case parameterized by ARN, that returns the required columns from_id (the instance that mounts the volume) and to_id (the volume that it mounts).

edge "ec2_instance_to_ebs_volume" {
  title = "mounts"

  sql = <<EOQ
    select
      i.arn as from_id,
      v.arn as to_id
    from
      aws_ec2_instance as i,
      jsonb_array_elements(block_device_mappings) as bd,
      aws_ebs_volume as v
    where
      v.volume_id = bd -> 'Ebs' ->> 'VolumeId'
      and i.arn = any($1);
  EOQ

  param "ec2_instance_arns" {}
}

Nodes and edges come together in graph blocks to produce the combined result set that the graph visualizer renders. It's not strictly necessary to define nodes and edges, you can use the original monolithic format, but we recommend using node and edge blocks which are named resources that you can reuse across graphs.

Let's get building!

Have fun working with these new graphs and components! We hope you'll share screenshots and snapshots to show us what you've done.

The Steampipe ecosystem now offers more than 90 plugins; each provides several, or dozens, or even hundreds of tables. You've always been able to use SQL to join those tables not only within an API family like AWS, but across diverse APIs. Now you can also use SQL to build relationship graphs within and across APIs. If you can imagine a useful way to visualize your team's activity in GitHub as it relates to conversations in Slack, you can now build that graph. The sky's the limit, and we look forward to hearing about your adventures in our Slack community.