You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "FrankChen021 (via GitHub)" <gi...@apache.org> on 2026/09/26 11:15:21 UTC
[I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
FrankChen021 opened a new issue, #20433:
URL: https://github.com/apache/druid/issues/20433
_This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them._
**Status:** Open: no fix PR
**Subject:** `KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards` (embedded-tests, `unit tests (25, K*,U*,Z*,Y*,X*)`)
**Failures:** 1 · **First seen:** 2026-09-23 · **Last seen:** 2026-09-23
### Root cause
`StreamIndexTestBase.verifyRowCount` gates on `waitForAllSegmentsToBeAvailable`, which counts Broker schema-refresh metrics. That count can be reached before the handed-off segments are queryable, and the suspend has already dropped the realtime segments, so `COUNT(*)` returned 2001 instead of 3000.
### Suggested fix
Switch `verifyRowCount` to `EmbeddedClusterApis.waitForAllSegmentsToBeQueryable` (added in #20319), or poll `COUNT(*)` until it matches.
### Occurrences
Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.
| Date | Commit | Job | Failure log | Detail | Reported in |
|---|---|---|---|---|---|
| 2026-09-23 | a379ea849f (#20397) | `unit tests (25, K*,U*,Z*,Y*,X*)` | [job 107010233701](https://github.com/apache/druid/actions/runs/35807088521/job/107010233701) | `COUNT(*)` 2001 vs expected 3000; 1/1 attempt | #20414 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on issue #20433:
URL: https://github.com/apache/druid/issues/20433#issuecomment-6073140031
Failed again on 2026-10-08: 51b66c7819 [job 113291222476](https://github.com/apache/druid/actions/runs/37771286884/job/113291222476), `COUNT(*)` 2001 vs expected 3000 (1/1 attempt).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "amaechler (via GitHub) via commits" <co...@druid.apache.org>.
amaechler closed issue #20433: Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards
URL: https://github.com/apache/druid/issues/20433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "OjashKush (via GitHub) via commits" <co...@druid.apache.org>.
OjashKush commented on issue #20433:
URL: https://github.com/apache/druid/issues/20433#issuecomment-6711737883
Hi @FrankChen021, I would like to take this. `StreamIndexTestBase.verifyRowCount()` waits for a Broker schema refresh before running `COUNT(*)`, which fits the race described here. I’ll try `waitForAllSegmentsToBeQueryable()` instead and run the affected Kinesis test to confirm. Any concerns with changing this shared helper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on issue #20433:
URL: https://github.com/apache/druid/issues/20433#issuecomment-6183159588
Failed again on 2026-10-14: afe71f5381 [job 115925729106](https://github.com/apache/druid/actions/runs/38585343856/job/115925729106), `COUNT(*)` returned 2001 instead of 3000 in `verifyAndTearDown` (1/1 attempt).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "FrankChen021 (via GitHub) via commits" <co...@druid.apache.org>.
FrankChen021 commented on issue #20433:
URL: https://github.com/apache/druid/issues/20433#issuecomment-6757277055
@OjashKush thanks for taking this, please go ahead. Changing the shared helper is fine, but switching to `waitForAllSegmentsToBeQueryable()` alone will not fully close the race, so please add one of the two changes below.
A few observations from the code:
- By the time `verifyAndTearDown()` runs, `ingest/rows/published` has already reached 3000, and that metric is emitted after the metadata-store commit. So all rows are published before the suspend; this is not an unpublished-data problem.
- The suspend is graceful (`gracefulShutdownInternal()` only finalizes the reading task, nothing is killed). The publishing tasks are mid-handoff: each still serves its segment until a Historical loads it, then unannounces it.
- `verifyRowCount()` runs `COUNT(*)` exactly once with no retry. The schema-refresh metric gate says nothing about what the Broker timeline holds at query time, which is why `waitForAllSegmentsToBeQueryable()` is an improvement: `is_available = 1` is derived from the same Broker server view the query uses.
- However `is_available` also counts the copy served by the Indexer. The poll can pass while a task still serves its segment, the task then unannounces it before the Broker has processed the Historical's announcement, and the single-shot `COUNT(*)` lands in that gap. Same failure, narrower window.
Two ways to make the assertion deterministic; either is fine:
1. **Make the state stable before asserting.** The supervisor is already suspended in `verifyAndTearDown()`, so wait for all tasks of the datasource to complete (so every segment is served only by Historicals), then call `waitForAllSegmentsToBeQueryable()`, then run the `COUNT(*)` once.
2. **Poll the query instead of asserting once.** Replace the single `verifySqlQuery()` in `verifyRowCount()` with a `waitForResult(...)` loop that polls `SELECT COUNT(*)` until it equals the expected count, with a timeout. This tolerates every transient inventory state, which is appropriate for a test whose contract is eventual consistency.
One minor caveat with `waitForAllSegmentsToBeQueryable()`: it counts used segments including overshadowed ones, while Historicals drop overshadowed copies. Harmless here since streaming appends never overshadow, but worth keeping in mind if `verifyRowCount()` is ever used after compaction. The other callers (`KafkaBoundedSupervisorTest`, `FaultyClusterTest`, `KafkaMultiSupervisorTest`) do not compact, so they are safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards (druid)
Posted by "OjashKush (via GitHub) via commits" <co...@druid.apache.org>.
OjashKush commented on issue #20433:
URL: https://github.com/apache/druid/issues/20433#issuecomment-6772476072
Thanks, Frank. I checked the Surefire reports for the September 23, October 8, and October 14 failures. In each, two tasks start at the same offsets but have different exclusiveStartPartitions. They get the same sequence name and segment ID. The first publishes a 1-row segment; the second builds a 999-row segment, but the metadata store reports that its ID already exists and skips it. The published-row metric reaches 3000 even though that segment is absent. I also tried waitForAllSegmentsToBeQueryable() followed by polling COUNT(*) for 240 seconds locally; it stayed at 2001. Could you take a look at this October 14 test report (https://github.com/apache/druid/actions/runs/38585343856/artifacts/11912997595)? I have raised a PR https://github.com/apache/druid/pull/20877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org