You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "amaechler (via GitHub)" <gi...@apache.org> on 2026/10/20 19:14:52 UTC
[PR] fix: Do not release a stopped streaming task to publish after a prior task group times out (druid)
amaechler opened a new pull request, #20651:
URL: https://github.com/apache/druid/pull/20651
Fixes #20604.
### Description
Since #20505, a streaming task's segment publish action waits on the Overlord until earlier task groups on the same partitions finish. When an earlier pending-completion group hits its completion timeout, `checkPendingCompletionTasks` kills it and stops the later groups on the same partitions, then calls `TaskGroup.onCompleted()` on all of them. That fires the later task's publish wait with `true`, so the Overlord commits its segments and offsets while the task is being stopped. The task's publish thread is interrupted, it never sees the publish succeed, and it does not emit `ingest/rows/published`.
This is what makes `KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart` flaky. The Coordinator restart stalls handoff past the test's 5s completion timeout. From the server logs of a failing run:
1. Task A (partition 1, offsets 500->712) has published and is waiting for handoff. Task B (712->1213) is blocked: "needs to wait before publishing as [1] other taskGroups are currently publishing".
2. `17:46:36,956` A times out and is killed; B is stopped ("one of earlier groups ... timed out before publishing segments").
3. `17:46:36,963` Overlord: "Updated metadata for supervisor ... from {p1=712} to {p1=1213}".
4. `17:46:36,966` B: "Failed publish (java.lang.RuntimeException: java.lang.InterruptedException)".
5. The supervisor resumes from 1213. All rows are ingested, but `ingest/rows/published` stays 500 short and `verifyAndTearDown` times out after 240s.
Passing runs hit the same completion timeout; they just have no publish in flight when it fires.
#### Fix
`killTask` and `stopTask(id, false)` now complete the task's pending publish wait with `false` before shutting it down. The later `set(true)` from `onCompleted()` is a no-op. `false` maps to the existing `SegmentPublishResult.retryableFailure("Task is not ready to publish yet")`; the stop request reaches the task well within the first retry backoff (~1s).
An alternative is a release-time check in `TaskWaitingToPublish` that the waiting task is still tracked by a task group. I went with the kill/stop hook because the supervisor already routes every shutdown through these two methods, and it leaves the existing readiness tests unchanged.
<hr>
##### Key changed/added classes in this PR
* `SeekableStreamSupervisor`
* `SeekableStreamSupervisorStateTest`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Do not release a stopped streaming task to publish after a prior task group times out (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #20651:
URL: https://github.com/apache/druid/pull/20651#issuecomment-6312095779
merge it to see if we still suffer from the flakiness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Do not release a stopped streaming task to publish after a prior task group times out (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #20651:
URL: https://github.com/apache/druid/pull/20651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org