Have stuck detection account for worker-level timeouts as well as client-level #1126
+115
−70
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brandur
force-pushed
the
brandur-stuck-worker-timeout
branch
from
a76ba97 to
11838bf
Compare
brandur
force-pushed
the
brandur-stuck-worker-timeout
branch
3 times, most recently
from
9134eaf to
a0cd70e
Compare
…ent-level As reported by #1125, the stuck job detection mechanism currently only considers client-level timeouts and without properly accounting for the possibility of worker-level overrides. Here, make sure to account for worker-level timeouts as well by hosting the job timeout calculation further up in the executor logic. Fixes #1125.
brandur
force-pushed
the
brandur-stuck-worker-timeout
branch
from
a0cd70e to
7eebbb3
Compare
brandur
commented
Jan 18, 2026
| } | ||
|
|
||
| return context.WithCancel(ctx) | ||
| } |
Contributor
Author
There was a problem hiding this comment.
This was doing very little (it's the same as a if jobTimeout > 0 { check) and I needed the conditional for something else, so I just pulled it into execute.
bgentry
approved these changes
Jan 19, 2026
Contributor
bgentry
left a comment
bgentry
left a comment
There was a problem hiding this comment.
LGTM, great quick fix. Probably worthy of a quick .1 release?
Comment on lines
+234
to
+235
| watchStuckCancel := e.watchStuck(ctx) | ||
| defer watchStuckCancel() |
Contributor
There was a problem hiding this comment.
IMO this is so much more readable than the large inline block before. I don't feel like I lose the context and flow of the surrounding code, while the fairly complex stuck job watching logic is nicely contained in a helper. 👏
Contributor
Author
There was a problem hiding this comment.
Yeah +1. Should have done this from the beginning.
Contributor
Author
Thx. And yeah, let's get a patch release in given there's no upcoming plans for another release anytime soon. |
brandur
added a commit
that referenced
this pull request
Jan 19, 2026
Prepare patch release largely containing #1126. [skip ci]
brandur
added a commit
that referenced
this pull request
Jan 19, 2026
Prepare patch release largely containing #1126. [skip ci]
brandur
added a commit
that referenced
this pull request
Jan 26, 2026
Fix a bug that came in with #1126 in which we we were correctly calculating timeout, but then not passing it down to the stuck job function when starting the stuck detection goroutine. There is a test that was checking this worked, but due to the nature of the bug, it was in effect detecting a stuck job after 0s and therefore passing by accident. I looked into ways to add additional testing here, but elected not to add more because they'd involve the sort of test I really hate, which has to wait arbitrarily wait to try and check that something did not happen, introducing both slowness and intermittency. After the fix here lands, this is the sort of thing that's not too likely to regress, and should be noticed quickly in case it does.
brandur
added a commit
that referenced
this pull request
Jan 26, 2026
Fix a bug that came in with #1126 in which we we were correctly calculating timeout, but then not passing it down to the stuck job function when starting the stuck detection goroutine. There is a test that was checking this worked, but due to the nature of the bug, it was in effect detecting a stuck job after 0s and therefore passing by accident. I looked into ways to add additional testing here, but elected not to add more because they'd involve the sort of test I really hate, which has to wait arbitrarily wait to try and check that something did not happen, introducing both slowness and intermittency. After the fix here lands, this is the sort of thing that's not too likely to regress, and should be noticed quickly in case it does. Fixes #1125.
brandur
added a commit
that referenced
this pull request
Jan 26, 2026
Fix a bug that came in with #1126 in which we we were correctly calculating timeout, but then not passing it down to the stuck job function when starting the stuck detection goroutine. There is a test that was checking this worked, but due to the nature of the bug, it was in effect detecting a stuck job after 0s and therefore passing by accident. I looked into ways to add additional testing here, but elected not to add more because they'd involve the sort of test I really hate, which has to wait arbitrarily wait to try and check that something did not happen, introducing both slowness and intermittency. After the fix here lands, this is the sort of thing that's not too likely to regress, and should be noticed quickly in case it does. Fixes #1125.
brandur
added a commit
that referenced
this pull request
Jan 27, 2026
Fix a bug that came in with #1126 in which we we were correctly calculating timeout, but then not passing it down to the stuck job function when starting the stuck detection goroutine. There is a test that was checking this worked, but due to the nature of the bug, it was in effect detecting a stuck job after 0s and therefore passing by accident. I looked into ways to add additional testing here, but elected not to add more because they'd involve the sort of test I really hate, which has to wait arbitrarily wait to try and check that something did not happen, introducing both slowness and intermittency. After the fix here lands, this is the sort of thing that's not too likely to regress, and should be noticed quickly in case it does. Fixes #1125.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As reported by #1125, the stuck job detection mechanism currently only
considers client-level timeouts and without properly accounting for the
possibility of worker-level overrides.
Here, make sure to account for worker-level timeouts as well by hosting
the job timeout calculation further up in the executor logic.
Fixes #1125.