The GitHub plugin ranks second among all Steampipe plugins with respect to stargazers. Clearly it has served the community well. But there were concerns about its rapid consumption of your rate limit allowance. When running GitHub Compliance against our own organization, for example, we weren't able to complete the benchmark without exceeding the rate limit. That's now resolved by the v0.28 release we're announcing today. Thanks to GitHub's GraphQL-based v4 API, we've enabled the plugin to fetch a lot more data before hitting the limit.
This is a major overhaul that entails both breaking changes to existing tables, as well as some new tables. So if you have existing queries you'll want to review the changes in order to align with v0.28. To help you orient to the new version, and for the benefit of anyone working on a v3-to-v4 transition, here's an overview of our initial research and migration strategy.
Unblocking the GitHub Compliance and GitHub Sherlock mods
For an organization like ours, with many repos, it was challenging to run the Compliance and Sherlock mods in a timely fashion — or even just to complete them. We're delighted to report that, with v0.28, we were finally able to complete the running of the Compliance mod, and use only about 1/5th of our rate limits in doing so.
The sentiment expressed by our CEO, Nathan Wallace, nicely sums up the progress made with these changes to the plugin.
Awesome we can finally run that compliance mod!!!
How these changes affect you
This is a major overhaul, with breaking changes across most tables, so you'll want to ensure that you have capacity to upgrade any queries and/or mods you depend on before upgrading the plugin.
Column changes are documented comprehensively in the following tracking issues:
- Update github_tag table to use GraphQL API #238
- Update github_team & github_my_team tables to use GraphQL API #241
- Update github_team_member table to use GraphQL API #243
- Update github_branch table to use GraphQL API #244
- Update github_user table to use GraphQL API #248
- Update github_branch_protection table to use GraphQL API #251
- Update github_team_repository table to use GraphQL API #254
- Update github_repository table to use GraphQL API #256
- Update github_my_repository table to use GraphQL API #257
- Update github_organization table to use GraphQL API #258
- Update github_my_organization table to use GraphQL API #259
- Update github_issue & github_my_issue tables to use GraphQL API #260
- Update github_commit table to use GraphQL API #262
- Update github_community_profile table to use GraphQL API #266
- Update github_search_issue, github_search_pull_request, github_search_repository, github_search_user tables to use GraphQL API #268
- Update github_stargazer & github_my_star tables to use GraphQL API #270
- Update github_pull_request table to use GraphQL API #276
- Update github_organization_member table to include all user fields #279
We've also included some new tables:
- github_issue_comment (#281)
- github_pull_request_comment (#281)
- github_rate_limit_graphql (#280)
- github_repository_collaborator (#280)
- github_repository_deployment (#282)
- github_repository_environment (#282)
Note: that when paging the repository resources, responses are a bit slower than they were with the previous REST-only tables. We hope GitHub will address this minor regression by doing work to improve performance in that case.
Next steps
We're not done trying to make this plugin better! As the GraphQL API becomes more mature, we may revisit tables that still use REST primarily, or in a secondary way to hydrate extra data, looking for any performance improvements to be gained by swapping these tables to GraphQL. Meanwhile, once you determine there's a clear path to upgrading to v0.28, please give it a try and let us know how it goes!