You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "FrankChen021 (via GitHub)" <gi...@apache.org> on 2026/09/26 11:15:25 UTC
[I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) (druid)
FrankChen021 opened a new issue, #20435:
URL: https://github.com/apache/druid/issues/20435
_This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them._
**Status:** Open: no fix PR
**Subject:** `IngestionSmokeTest.test_streamLogs_ofCancelledTask`, seen via `ConsulDiscoveryPlainDockerTest` (embedded-tests, `docker-tests`)
**Failures:** 1 · **First seen:** 2026-09-25 · **Last seen:** 2026-09-25
### Root cause
Race: the test polls `TaskLogStreamer.streamTaskLog(taskId, 0)` only until the `Optional` is present. In the Docker setup, the peon's log is pushed after the process exits, so the stream can be present but empty, which fails `assertFalse(logs.isEmpty())` at `IngestionSmokeTest.java:398`. The TLS and MTLS variants passed in the same job.
### Suggested fix
Move the read into the `waitForResult` supplier, and wait until the log contains `"Running task[%s] for [%d] millis"` instead of just `Optional::isPresent`.
### Occurrences
Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.
| Date | Commit | Job | Failure log | Detail | Reported in |
|---|---|---|---|---|---|
| 2026-09-25 | 53e814e87c (#20425) | `docker-tests` | [job 108039867472](https://github.com/apache/druid/actions/runs/36125282875/job/108039867472) | streamed task log was empty; single attempt (no surefire retries) | #20428 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 closed issue #20435: Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest)
URL: https://github.com/apache/druid/issues/20435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (docker tests) (druid)
Posted by "FrankChen021 (via GitHub) via commits" <co...@druid.apache.org>.
FrankChen021 commented on issue #20435:
URL: https://github.com/apache/druid/issues/20435#issuecomment-6621025148
Failed again on 2026-11-03: 0bce210c62 (#20826) [job 126014185494](https://github.com/apache/druid/actions/runs/41663963621/job/126014185494), this time in `KubernetesClusterDockerTest`: the log stream stayed empty for the full 10 s poll. The fix in #20549 did not hold for this subclass, where the embedded Overlord is not the leader; reopening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) (druid)
Posted by "ykisana (via GitHub)" <gi...@apache.org>.
ykisana commented on issue #20435:
URL: https://github.com/apache/druid/issues/20435#issuecomment-6092209760
@rishi-rana I think this fixes the test. But I believe there may be an underlying bug here.
The Overlord's HttpRemoteTaskRunner.streamTaskLog passes the worker's response through without checking the HTTP status. A 404, which happens once the worker has dropped a finished task, becomes a present-but-empty log stream.
SwitchingTaskLogStreamer then never falls back to deep storage, where the log actually is.
https://github.com/apache/druid/blob/98bbcb4c7f78/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L936-L940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) (druid)
Posted by "ykisana (via GitHub)" <gi...@apache.org>.
ykisana commented on issue #20435:
URL: https://github.com/apache/druid/issues/20435#issuecomment-6092403280
> 6\. [@rishi-rana](https://github.com/rishi-rana) I think this fixes the test. But I believe there may be an underlying bug here.
> The Overlord's HttpRemoteTaskRunner.streamTaskLog passes the worker's response through without checking the HTTP status. A 404, which happens once the worker has dropped a finished task, becomes a present-but-empty log stream.
> SwitchingTaskLogStreamer then never falls back to deep storage, where the log actually is.
> https://github.com/apache/druid/blob/98bbcb4c7f78/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L936-L940
Nvm ignore me, looking at logs, the test flake happened before this bug would've opened up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) (druid)
Posted by "rishi-rana (via GitHub)" <gi...@apache.org>.
rishi-rana commented on issue #20435:
URL: https://github.com/apache/druid/issues/20435#issuecomment-6092056506
I'd like to pick this up — will move the log read into the `waitForResult` poll loop so it waits for the expected log line (`Running task[%s] for [%d] millis`) rather than just stream presence, per the suggested fix above. Will open a PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org