An upgraded GitHub plugin uses GraphQL APIs to overcome throttling woes

These changes also create welcome headroom for the Compliance and Sherlock mods.

Graza Andersson
6 min. read - Jun 21, 2023
These changes also create welcome headroom for the Compliance and Sherlock mods.

The GitHub plugin ranks second among all Steampipe plugins with respect to stargazers. Clearly it has served the community well. But there were concerns about its rapid consumption of your rate limit allowance. When running GitHub Compliance against our own organization, for example, we weren't able to complete the benchmark without exceeding the rate limit. That's now resolved by the v0.28 release we're announcing today. Thanks to GitHub's GraphQL-based v4 API, we've enabled the plugin to fetch a lot more data before hitting the limit.

This is a major overhaul that entails both breaking changes to existing tables, as well as some new tables. So if you have existing queries you'll want to review the changes in order to align with v0.28. To help you orient to the new version, and for the benefit of anyone working on a v3-to-v4 transition, here's an overview of our initial research and migration strategy.

Unblocking the GitHub Compliance and GitHub Sherlock mods

For an organization like ours, with many repos, it was challenging to run the Compliance and Sherlock mods in a timely fashion — or even just to complete them. We're delighted to report that, with v0.28, we were finally able to complete the running of the Compliance mod, and use only about 1/5th of our rate limits in doing so.

compliance mod run complete

The sentiment expressed by our CEO, Nathan Wallace, nicely sums up the progress made with these changes to the plugin.

Awesome we can finally run that compliance mod!!!

How these changes affect you

This is a major overhaul, with breaking changes across most tables, so you'll want to ensure that you have capacity to upgrade any queries and/or mods you depend on before upgrading the plugin.

Column changes are documented comprehensively in the following tracking issues:

We've also included some new tables:

Note: that when paging the repository resources, responses are a bit slower than they were with the previous REST-only tables. We hope GitHub will address this minor regression by doing work to improve performance in that case.

Next steps

We're not done trying to make this plugin better! As the GraphQL API becomes more mature, we may revisit tables that still use REST primarily, or in a secondary way to hydrate extra data, looking for any performance improvements to be gained by swapping these tables to GraphQL. Meanwhile, once you determine there's a clear path to upgrading to v0.28, please give it a try and let us know how it goes!