You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ChoMinGi (via GitHub)" <gi...@apache.org> on 2026/04/16 01:25:14 UTC
[PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
ChoMinGi opened a new pull request, #22073:
URL: https://github.com/apache/kafka/pull/22073
[KAFKA-20253](https://issues.apache.org/jira/browse/KAFKA-20253)
Reproduced the busy loop reported by @mstruk using the
[reproducer repo](https://github.com/mstruk/kafka-consumer-reproducer).
Root cause sits in `AbstractCoordinator.HeartbeatThread`: when an
`AuthenticationException` reaches the outer catch (e.g. after a transient
authorization server outage), it calls `setFailureCause(e)` and exits.
Nothing requests a rejoin, so `state` stays `STABLE`, `rejoinNeeded`
stays `false`, and `heartbeatThread` is cleared to `null`. `timeToNextHeartbeat()`
then returns the stale `heartbeatTimer.remainingMs() == 0`, and
`NetworkClient.poll(0)` spins via `selectNow()` indefinitely.
Fixed by extending the existing "error → `requestRejoin()`" pattern
already used for `REBALANCE_IN_PROGRESS` / `UNKNOWN_MEMBER_ID` /
`FENCED_INSTANCE_ID` in `HeartbeatResponseHandler`. The next `poll()`
then runs `ensureActiveGroup()` → `startHeartbeatThreadIfNeeded()` →
`heartbeat.resetTimeouts()`, restoring the consumer to a consistent
state. `GroupAuthorizationException` is intentionally left unchanged —
rejoining on an ACL policy denial would loop.
Added unit test `testAuthenticationErrorInHeartbeatThreadTriggersRejoin`
and verified end-to-end against the reproducer:
| | CPU (stuck consumer) | Generation after recovery |
|---|---|---|
| Before fix | ~101 % sustained | stuck at 5 |
| After fix | ~0.3 % | 5 → 15 (fresh rejoin) |
Covers the "other non-re-auth cases" angle raised by @kirktrue — the
unit test exercises the general mechanism of an `AuthenticationException`
reaching the heartbeat thread, which is the single common denominator
for every variant.
Thanks to @mstruk for the thorough bug report and the reproducer — made
this straight-to-root-cause.
All existing `AbstractCoordinatorTest` / `ConsumerCoordinatorTest` tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "mstruk (via GitHub)" <gi...@apache.org>.
mstruk commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-4443184092
I tested this PR with the reproducer, and I was unable to trigger the issue. It looks like this fix works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield merged PR #22073:
URL: https://github.com/apache/kafka/pull/22073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "ChoMinGi (via GitHub)" <gi...@apache.org>.
ChoMinGi commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5092402747
@mimaison Yes — resolved the JIRA as Fixed (4.4.0)
- Classic consumer: fixed by #22073
- Async/share consumer: fixed by #22836
The two don't overlap (`AbstractCoordinator.java` vs the `RequestManager` classes), so both paths are covered now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5093256418
@ChoMinGi Personally, I find backport PRs easiest so please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-4301664334
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "ChoMinGi (via GitHub)" <gi...@apache.org>.
ChoMinGi commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5002000212
Thanks for the reviews :)
Pushed bdc9755 with the extended test — details in the inline reply.
On requestRejoin not being quite intended for this — agreed.
I did consider restarting only the heartbeat thread. But the client can't know upfront whether the membership is still valid, so that path would need its own fallback to a rejoin when the session has expired, effectively duplicating recovery logic. Reusing the proven rejoin path felt like the right scope for this fix, even though it can cost an unnecessary rebalance when the outage is shorter than the session timeout.
If there's interest, I'd be happy to explore a dedicated heartbeat-recovery path in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "ChoMinGi (via GitHub)" <gi...@apache.org>.
ChoMinGi commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5093858822
@AndrewJSchofield Opened the backport PR #22964 (4.3), #22965 (4.2) :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #22073:
URL: https://github.com/apache/kafka/pull/22073#discussion_r3595030348
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinatorTest.java:
##########
@@ -1288,6 +1288,43 @@ public void testUncaughtExceptionInHeartbeatThread() throws Exception {
}
}
+ @Test
+ public void testAuthenticationErrorInHeartbeatThreadTriggersRejoin() throws Exception {
Review Comment:
This test in a way tests that the chosen fix was implemented, namely that rejoin was requested. What I'd really like to see is a test that does an authentication failure on the first heartbeat, then the rejoin is requested, and a subsequent successful heartbeat works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "ChoMinGi (via GitHub)" <gi...@apache.org>.
ChoMinGi commented on code in PR #22073:
URL: https://github.com/apache/kafka/pull/22073#discussion_r3602104578
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinatorTest.java:
##########
@@ -1288,6 +1288,43 @@ public void testUncaughtExceptionInHeartbeatThread() throws Exception {
}
}
+ @Test
+ public void testAuthenticationErrorInHeartbeatThreadTriggersRejoin() throws Exception {
Review Comment:
@AndrewJSchofield
Done in https://github.com/apache/kafka/pull/22073/commits/bdc9755f7d768667620b975740d966d015244d07 — The test now covers the full cycle.
auth failure on the first heartbeat, rejoin requested, successful JoinGroup/SyncGroup once auth recovers, then a subsequent heartbeat completing without another rejoin.
To keep the last assertion from passing vacuously, the test requires the prepared heartbeat response to be consumed with no heartbeat left in flight — the matcher only accepts a HeartbeatRequest, so the post-recovery heartbeat must actually have been sent and processed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5091991016
Can we now close KAFKA-20253?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "ChoMinGi (via GitHub)" <gi...@apache.org>.
ChoMinGi commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5092591523
+1 from me
it's a production-impacting busy-spin and the JIRA lists 4.0.0/4.2.0 as affected, so both branches seem worth it.
FWIW I tried the cherry-picks locally
- #22073 applies cleanly to both 4.3 and 4.2.
- #22836 has a test conflict on 4.3, `AbstractHeartbeatRequestManager` on 4.2.
Happy to open the backport PRs if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-4456902762
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-4403261128
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-4443357229
@kirktrue @FrankYang0529 Can you take a look? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20253: Trigger rejoin on heartbeat thread AuthenticationException [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #22073:
URL: https://github.com/apache/kafka/pull/22073#issuecomment-5092417322
Should we backport these to 4.3 and 4.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org