e2e tests on Cirrus Labs runners #5485

alwx · 2025-12-18T10:38:02Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Enabled Cirrus Labs runners for e2e tests. Fixes #5418

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
I updated the docs if needed.
I updated the wizard if needed.
All tests passing
No breaking changes

🔮 Next steps

.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


github-actions · 2026-01-26T14:18:26Z

Semver Impact of This PR

⚪ None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

e2e tests on Cirrus Labs runners by alwx in #5485

chore(deps): bump getsentry/craft from 2.20.0 to 2.20.1 by dependabot in #5603
chore(deps): bump getsentry/craft/.github/workflows/changelog-preview.yml from 2.20.0 to 2.20.1 by dependabot in #5604
chore(deps): bump github/codeql-action from 4.31.11 to 4.32.0 by dependabot in #5605
Ref: Add Claude task by lucas-zimerman in #5602
chore(deps): update JavaScript SDK to v10.38.0 by github-actions in #5596

_{🤖 This preview updates automatically when you update the PR.}

github-actions · 2026-01-28T11:02:23Z

Android (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	423.06 ms	418.38 ms	-4.68 ms
Size	43.75 MiB	48.41 MiB	4.65 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	501.18 ms	536.70 ms	35.52 ms
`bfe454a`+dirty	573.44 ms	579.46 ms	6.02 ms
`294387d`+dirty	424.30 ms	465.40 ms	41.10 ms
`955f2eb`+dirty	422.74 ms	410.19 ms	-12.55 ms
`3bd3f0d`+dirty	447.21 ms	472.31 ms	25.10 ms
`d081295`+dirty	408.08 ms	453.62 ms	45.54 ms
`ad27f6e`+dirty	471.44 ms	516.23 ms	44.79 ms
`07808fb`+dirty	419.10 ms	419.08 ms	-0.02 ms
`170d5ea`+dirty	407.92 ms	422.49 ms	14.57 ms
`8e653ac`+dirty	360.28 ms	372.04 ms	11.76 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`bfe454a`+dirty	17.75 MiB	19.69 MiB	1.94 MiB
`294387d`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`955f2eb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`3bd3f0d`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`d081295`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`ad27f6e`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`07808fb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`170d5ea`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`8e653ac`+dirty	17.75 MiB	19.75 MiB	2.00 MiB

github-actions · 2026-01-28T11:02:34Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	381.59 ms	421.54 ms	39.95 ms
Size	43.94 MiB	49.26 MiB	5.33 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	422.90 ms	468.30 ms	45.40 ms
`294387d`+dirty	359.44 ms	393.40 ms	33.97 ms
`1d62dde`+dirty	366.59 ms	408.80 ms	42.21 ms
`d081295`+dirty	416.95 ms	461.24 ms	44.29 ms
`ad27f6e`+dirty	484.67 ms	532.79 ms	48.12 ms
`1226664`+dirty	377.65 ms	453.94 ms	76.29 ms
`083f560`+dirty	383.96 ms	417.76 ms	33.80 ms
`ec14be7`+dirty	401.58 ms	475.26 ms	73.68 ms
`eb07ba3`+dirty	419.49 ms	482.12 ms	62.63 ms
`d1fd647`+dirty	374.46 ms	409.51 ms	35.05 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`294387d`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`1d62dde`+dirty	7.15 MiB	8.46 MiB	1.31 MiB
`d081295`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`ad27f6e`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`1226664`+dirty	7.15 MiB	8.46 MiB	1.30 MiB
`083f560`+dirty	7.15 MiB	8.43 MiB	1.28 MiB
`ec14be7`+dirty	7.15 MiB	8.42 MiB	1.26 MiB
`eb07ba3`+dirty	7.15 MiB	8.42 MiB	1.27 MiB
`d1fd647`+dirty	7.15 MiB	8.43 MiB	1.28 MiB

.github/workflows/e2e-v2.yml

github-actions · 2026-01-29T09:24:54Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1207.06 ms	1214.31 ms	7.25 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1207.02 ms	1199.27 ms	-7.76 ms
`d73150f`+dirty	1224.52 ms	1227.17 ms	2.65 ms
`d081295`+dirty	1205.24 ms	1207.52 ms	2.28 ms
`d1fd647`+dirty	1218.16 ms	1225.82 ms	7.65 ms
`ea3e26e`+dirty	1216.61 ms	1214.15 ms	-2.47 ms
`80e4616`+dirty	1206.90 ms	1205.94 ms	-0.96 ms
`955f2eb`+dirty	1225.78 ms	1239.27 ms	13.49 ms
`5ee3314`+dirty	1234.25 ms	1235.44 ms	1.19 ms
`70250df`+dirty	1211.96 ms	1222.31 ms	10.35 ms
`664c66f`+dirty	1195.94 ms	1194.80 ms	-1.14 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	3.19 MiB	4.56 MiB	1.37 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	3.19 MiB	4.55 MiB	1.36 MiB
`5ee3314`+dirty	3.19 MiB	4.55 MiB	1.37 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

github-actions · 2026-01-29T09:24:54Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1216.98 ms	1204.23 ms	-12.74 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1212.20 ms	1217.89 ms	5.70 ms
`d73150f`+dirty	1198.44 ms	1210.06 ms	11.62 ms
`d081295`+dirty	1214.40 ms	1211.27 ms	-3.13 ms
`d1fd647`+dirty	1219.35 ms	1233.18 ms	13.83 ms
`ea3e26e`+dirty	1229.13 ms	1228.46 ms	-0.67 ms
`80e4616`+dirty	1221.32 ms	1225.64 ms	4.32 ms
`955f2eb`+dirty	1235.06 ms	1253.88 ms	18.81 ms
`5ee3314`+dirty	1215.18 ms	1207.64 ms	-7.54 ms
`70250df`+dirty	1214.51 ms	1215.04 ms	0.53 ms
`664c66f`+dirty	1215.37 ms	1221.30 ms	5.92 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	2.63 MiB	3.99 MiB	1.36 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	2.63 MiB	3.98 MiB	1.35 MiB
`5ee3314`+dirty	2.63 MiB	3.99 MiB	1.35 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

lucas-zimerman · 2026-01-29T14:54:58Z

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

is it ready for review?

alwx · 2026-01-30T08:31:03Z

@lucas-zimerman there is still a (hopefully) minor issue with some tests, figuring it out now

antonis · 2026-01-30T08:35:44Z

there is still a (hopefully) minor issue with some tests, figuring it out now

@alwx heads up that the captureReplay e2e tests on iOS seem to be failing on main too since Jan 28th. I think this is flakiness since I don't see any change that may have caused this but I'll check and iterate back

.github/workflows/e2e-v2.yml

alwx · 2026-01-30T09:43:31Z

@antonis true that, I see that on main branch as well

.github/workflows/e2e-v2.yml

antonis

I told Claude to summarize the performance difference between main and this PR and the improvement is massive 🚀

Job	Old/Main	New/PR	Difference
auth_token_check / auth_token_check	2s	3s	+1s
Ready-to-merge gate / Missing "ready-to-merge" label	2s	4s	+2s
diff_check / diff_check	4s	10s	+6s
metrics (legacy, android)	13m 26s	13m 15s	-11s
metrics (legacy, ios)	23m 33s	14m 36s	-8m 57s
metrics (new, android)	18m 25s	14m 38s	-3m 47s
metrics (new, ios)	24m 0s	20m 9s	-3m 51s
Build RN 0.71.19 legacy hermes android production no	9m 57s	5m 15s	-4m 42s
Build RN 0.71.19 legacy hermes ios production no	16m 50s	6m 44s	-10m 6s
Build RN 0.71.19 legacy jsc android production no	8m 30s	4m 29s	-4m 1s
Build RN 0.71.19 legacy jsc ios production no	20m 12s	6m 47s	-13m 25s
Build RN 0.83.0 legacy hermes android production no	9m 2s	5m 39s	-3m 23s
Build RN 0.83.0 legacy hermes ios production no	35m 58s	8m 57s	-27m 1s
Build RN 0.83.0 legacy hermes ios production dynamic	28m 56s	8m 13s	-20m 43s
Build RN 0.83.0 legacy hermes ios production static	32m 36s	8m 27s	-24m 9s
Build RN 0.83.0 new hermes android production no	9m 24s	4m 59s	-4m 25s
Build RN 0.83.0 new hermes ios production no	26m 41s	7m 55s	-18m 46s
Build RN 0.83.0 new hermes ios production static	23m 34s	8m 11s	-15m 23s
Test RN 0.83.0 legacy hermes android production no	6m 31s	6m 6s	-25s
Test RN 0.83.0 legacy hermes ios production no	18m 32s	4m 12s	-14m 20s
Test RN 0.83.0 new hermes android production no	7m 14s	6m 39s	-35s
Test RN 0.83.0 new hermes ios production no	16m 8s	4m 37s	-11m 31s

However the iOS E2E tests seem to be fragile and the app is crashing (tried two reruns) with:

App crashed or stopped while executing flow, please check diagnostic logs: ~/Library/Logs/DiagnosticReports directory

lucas-zimerman · 2026-01-30T15:37:21Z

That's a great improvement!
We should add the path ~/Library/Logs/DiagnosticReports to the logs when e2e tests fails too.

dev-packages/e2e-tests/cli.mjs

alwx · 2026-02-02T11:07:28Z

@antonis I reverted back changes made to react-native-test — it now uses macos-26 GH runners like before because those jobs don't seem to work with Cirrus Labs runners on iOS :(
I tried reproducing the issue, tried checking the logs, changing versions of libs, played with different runners, and played with Claude to help with debugging — nothing helped find the solution yet 😞 But considering that e2e tests need more work in general and I planned to do this work this week anyway, I think it makes sense to move on with that and handle the issue with e2e tests either separately or as part of this: #5381

But even if not considering those jobs, we see a pretty good speed increase in test execution time (thanks for the stats, @claude and @antonis :) ), not to mention that the CI does seem more predictable now. That all will definitely need more testing and more adjustments but I'm pretty happy with the results so far.

antonis

LGTM 🎸
Thank you for working on this @alwx 🙇

e2e tests on Cirrus Labs runners

e2397d0

alwx self-assigned this Dec 18, 2025

alwx marked this pull request as ready for review December 18, 2025 10:44

alwx requested review from antonis and lucas-zimerman as code owners December 18, 2025 10:44

alwx added the ready-to-merge Triggers the full CI test suite label Dec 18, 2025

alwx changed the title ~~e2e tests on Cirrus Labs runners~~ DO NOT MERGE: e2e tests on Cirrus Labs runners Dec 18, 2025

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 18, 2025 14:08

Checking another group

a04f351

rolling back last change

83a3d6a

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 23, 2025 09:36

Merge branch 'main' into alwx/experiment/cirrus-labs

1cb4e72

Merge branch 'main' into alwx/experiment/cirrus-labs

bd991eb

alwx added 3 commits January 28, 2026 09:08

Corepack enable

a933ded

Corepack enable

9899e3d

Ninja

61211d5

Ruby install

b5c48c1

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

Ruby install

9142e2f

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

alwx added 2 commits January 29, 2026 10:06

Updated env params

da9811a

Trying to fix metrics job

3034d58

alwx changed the title ~~DO NOT MERGE: e2e tests on Cirrus Labs runners~~ e2e tests on Cirrus Labs runners Jan 29, 2026

alwx added 2 commits January 29, 2026 11:13

Merge branch 'main' into alwx/experiment/cirrus-labs

8c3a337

Trying it out with Macos 26.0.1

550e671

alwx added 4 commits January 29, 2026 14:06

Rolling back some of experiments

0c1d899

Solving concurrency issues

3ca98e6

More GH runner issues

a685372

Merge branch 'main' into alwx/experiment/cirrus-labs

6a25aaa

Merge branch 'main' into alwx/experiment/cirrus-labs

cbd93fc

sentry bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

alwx added 2 commits January 30, 2026 10:46

Small fix

ebb8321

Fix-2

bf21537

sentry bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

Fix-3

a0aadff

antonis added the skip-changelog label Jan 30, 2026

antonis reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

Hopefully final fix

94cbad2

alwx requested a review from antonis January 30, 2026 12:27

sentry bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

antonis reviewed Jan 30, 2026

View reviewed changes

Updated e2e tests, experiment

3ab0d17

sentry bot reviewed Feb 2, 2026

View reviewed changes

dev-packages/e2e-tests/cli.mjs Outdated Show resolved Hide resolved

Fixes

0bffd95

Merge branch 'main' into alwx/experiment/cirrus-labs

cf75a1c

alwx requested a review from antonis February 2, 2026 11:07

antonis approved these changes Feb 2, 2026

View reviewed changes

alwx merged commit 49d1744 into main Feb 2, 2026
94 of 99 checks passed

alwx deleted the alwx/experiment/cirrus-labs branch February 2, 2026 13:02

Uh oh!

e2e tests on Cirrus Labs runners #5485

e2e tests on Cirrus Labs runners #5485

Conversation

alwx commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Semver Impact of This PR

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

lucas-zimerman commented Jan 29, 2026

Uh oh!

alwx commented Jan 30, 2026

Uh oh!

antonis commented Jan 30, 2026

Uh oh!

Uh oh!

alwx commented Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antonis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucas-zimerman commented Jan 30, 2026

Uh oh!

Uh oh!

alwx commented Feb 2, 2026

Uh oh!

antonis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

alwx commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Jan 26, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

antonis left a comment •

edited

Loading