Skip to content

Add benchmark automation for PRs #1823

@jaydeluca

Description

@jaydeluca

Now that we have a benchmarks branch that will run nightly, we can build upon this. It is not uncommon for someone to introduce a change, and for us to want to validate the benchmarks before merging. We should create automation where you can add a tag to a PR, and benchmarks will run, followed by a bot that will publish a comment on the PR with the results and much they deviate from the baseline on the benchmarks branch.

We will have to see how the final results look in terms of how accurate they will be, since we don't have much control over the hosts running the tests.

Requirements:

  • When a "Run benchmarks" label is added to a PR, a github action job is triggered
  • The action will run the "ci" benchmark job for the particular branch
  • The action will diff the results with the latest info on the benchmarks branch
  • The action will leave a comment on the PR with a summary of the results and their delta compared to the baseline

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions