You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "rishi-rana (via GitHub)" <gi...@apache.org> on 2026/10/02 22:19:42 UTC
[PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
rishi-rana opened a new pull request, #23679:
URL: https://github.com/apache/kafka/pull/23679
## Summary
`testFenceMultipleBrokers` used a 1000ms session timeout while relying on the
test harness's 100ms heartbeat resend interval (`TestUtils.DEFAULT_POLL_INTERVAL_MS`)
to keep `brokersToKeepUnfenced` alive while waiting for `brokersToFence` to time out.
Under normal conditions this gives a ~10x margin, but under CI load (GC pauses,
scheduler contention) that margin can be exhausted between heartbeat resends,
causing a broker that should stay unfenced to be spuriously fenced instead —
matching the intermittent failure reported in this ticket.
- Raise `sessionTimeoutMillis` from 1000ms to 2000ms for this test, matching
the value already used by the structurally identical `testBalancePartitionLeaders`
in the same file, which has not been reported flaky.
- No production code changes; this only widens the real-time margin in the test.
Note: this is a reasoned fix based on tracing the timing mechanism in code
(confirmed `QuorumControllerTestEnv` uses `Time.SYSTEM` with no mock-time
injection point, and confirmed the 100ms-poll-vs-1000ms-timeout margin), not
a locally reproduced failure — the race is inherently load-dependent and did
not reproduce in repeated clean local runs on an idle machine.
Developed with AI assistance.
## Test plan
- [x] `./gradlew metadata:test --tests "org.apache.kafka.controller.QuorumControllerTest"` — full suite passes
- [x] `./gradlew :metadata:spotlessCheck :metadata:checkstyleTest` — clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23679:
URL: https://github.com/apache/kafka/pull/23679#issuecomment-6093268310
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
Posted by "rishi-rana (via GitHub)" <gi...@apache.org>.
rishi-rana commented on PR #23679:
URL: https://github.com/apache/kafka/pull/23679#issuecomment-5975567115
> Could you share the original failure output and any logs indicating that a broker intended to remain unfenced expired because of a delayed heartbeat?
Its in https://issues.apache.org/jira/browse/KAFKA-16634 , also it's more of guesstimated fix as I couldn't reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
Posted by "github-actions[bot] (via GitHub) via jira" <ji...@kafka.apache.org>.
github-actions[bot] commented on PR #23679:
URL: https://github.com/apache/kafka/pull/23679#issuecomment-6701494225
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
Posted by "andyhuangdev (via GitHub)" <gi...@apache.org>.
andyhuangdev commented on PR #23679:
URL: https://github.com/apache/kafka/pull/23679#issuecomment-5963959701
Could you share the original failure output and any logs indicating that a broker intended to remain unfenced expired because of a delayed heartbeat?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16634: Fix flaky testFenceMultipleBrokers by increasing session timeout [kafka]
Posted by "rishi-rana (via GitHub) via jira" <ji...@kafka.apache.org>.
rishi-rana commented on PR #23679:
URL: https://github.com/apache/kafka/pull/23679#issuecomment-6565575600
can I please get this reviewed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org