You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2026/06/01 18:58:06 UTC
[PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
dajac opened a new pull request, #22444:
URL: https://github.com/apache/kafka/pull/22444
When a `__consumer_offsets` partition transitions to follower, its
local log is truncated and re-replicated from the new leader. The
group coordinator hosting the partition remains active until it is
unloaded asynchronously. During that window, the partition's high
watermark advances again over records that this coordinator did not
write, while the coordinator still holds in-memory state (and pending
deferred operations) for its own records that were truncated and never
durably committed.
Applying such a high watermark has two consequences. It can violate the
invariants of the snapshot registry and fail the `HighWatermarkUpdate`
event, logging a spurious error such as "Execution of
HighWatermarkUpdate failed due to New committed offset X of
__consumer_offsets-N must be less than or equal to Y". More
importantly, when it does not fail, it advances the committed offset
over the coordinator's uncommitted state and completes the
corresponding deferred writes with a success response, even though
those records were lost. A client can therefore receive a successful
offset-commit acknowledgment for a commit that is silently dropped once
the new coordinator takes over.
This patch gates high watermark propagation in
`CoordinatorPartitionWriter.ListenerAdapter` on the partition's
leadership. The adapter stops forwarding high watermark updates once
the partition transitions to follower, is deleted, or fails. The
partition signals these transitions (via `PartitionListener`) before
its fetcher is restarted (see `ReplicaManager#applyDelta`), i.e. before
any such high watermark can be produced, so the coordinator never
observes a high watermark that it should not apply. The pending
deferred operations then remain in place and are failed with
`NOT_COORDINATOR` when the coordinator is unloaded, so clients
correctly retry against the new coordinator.
Gating on leadership rather than inspecting the offset is deliberate:
after truncation an offset can still have a snapshot in the registry
while holding the new leader's data, so no offset-based check can tell
whether a high watermark is safe to apply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #22444:
URL: https://github.com/apache/kafka/pull/22444#issuecomment-4610219425
> I think the new API contract for `registerListener` is unusual. However I don't have a better suggestion right now.
Could you please develop this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #22444:
URL: https://github.com/apache/kafka/pull/22444#issuecomment-4614425119
Merged to trunk, 4.3, 4.2, 4.1 and 4.0 as this is a small correctness issue too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #22444:
URL: https://github.com/apache/kafka/pull/22444#issuecomment-4610221522
@squah-confluent I have addressed your comments. Please take another look when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #22444:
URL: https://github.com/apache/kafka/pull/22444#discussion_r3342721603
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/PartitionWriter.java:
##########
@@ -33,8 +33,24 @@ public interface PartitionWriter {
/**
* Listener allowing to listen to high watermark changes. This is meant
* to be used in conjunction with {@link PartitionWriter#append(TopicPartition, VerificationGuard, MemoryRecords, short)}.
+ * <p>
+ * High watermark updates are delivered only while this broker is the leader of the
+ * partition. Once the partition is no longer led by this broker (it transitions to
+ * follower, is deleted or fails), no further updates are delivered. This guarantee
Review Comment:
On the first pass, I read this as "updates are only delivered while the broker is the leader" instead of "updates stop forever once the broker is no longer the leader", probably because of the first sentence. Maybe we can make this clearer?
`no further updates are delivered, even if the broker regains leadership.`?
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/PartitionWriter.java:
##########
Review Comment:
I think we should also call out the unexpected behavior on the `registerListener` javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22444:
URL: https://github.com/apache/kafka/pull/22444#discussion_r3346775461
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/PartitionWriter.java:
##########
@@ -33,8 +33,24 @@ public interface PartitionWriter {
/**
* Listener allowing to listen to high watermark changes. This is meant
* to be used in conjunction with {@link PartitionWriter#append(TopicPartition, VerificationGuard, MemoryRecords, short)}.
+ * <p>
+ * High watermark updates are delivered only while this broker is the leader of the
+ * partition. Once the partition is no longer led by this broker (it transitions to
+ * follower, is deleted or fails), no further updates are delivered. This guarantee
Review Comment:
Updated the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22444:
URL: https://github.com/apache/kafka/pull/22444#discussion_r3346777015
##########
coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/PartitionWriter.java:
##########
Review Comment:
Extended the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #22444:
URL: https://github.com/apache/kafka/pull/22444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20634: Spurious HighWatermarkUpdate failed errors in the group coordinator after partition leadership change [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on PR #22444:
URL: https://github.com/apache/kafka/pull/22444#issuecomment-4610699148
> > I think the new API contract for registerListener is unusual. However I don't have a better suggestion right now.
>
> Could you please develop this?
The part where the listener stops working after the first leadership loss is just unexpected. As long as we document it clearly I think it's okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org