You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2026/08/20 07:15:02 UTC
[PR] KAFKA-20845: Don't flush empty batch when appending large records in group coordinator (4.0) [kafka]
dajac opened a new pull request, #23209:
URL: https://github.com/apache/kafka/pull/23209
When appending records, we create a new batch if no batch exists.
KAFKA-19760 introduced a flush of any existing batch when appending
large records, to maximize our chances of compressing the records under
the max.message.bytes. When these two things happen in the same append
operation, we flush an empty batch.
Flushing an empty batch is written to fail and revert the coordinator
state. This would be harmless, except some group coordinator operations
update the coordinator state directly without replay. Upon appending
their records and triggering the empty batch flush, their state changes
are then reverted while their records are written. The group
coordinator's in-memory state diverges from the on-disk state and
subsequent writes for the group can be invalid for the on-disk state.
eg. we may have a consumer group downgraded to a classic group, followed
by consumer group records which is invalid.
Do not flush empty batches when appending large records.
Backport of #22969 to 4.0. `CoordinatorContext.batchEpoch` does not
exist on this branch, so the test cannot assert that a single batch was
created. Instead, it uses a non-replaying write operation which applies
its in-memory changes directly and verifies that they are not reverted
by the flush of an empty batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20845: Don't flush empty batch when appending large records in group coordinator (4.0) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #23209:
URL: https://github.com/apache/kafka/pull/23209#issuecomment-5364976193
`UserQuotaTest` passes locally, and I have backported #20121 to 4.0 and 4.1 to stabilize `testHWCheckpointWithFailuresMultipleLogSegments`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20845: Don't flush empty batch when appending large records in group coordinator (4.0) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #23209:
URL: https://github.com/apache/kafka/pull/23209#issuecomment-5360400952
```
FAILED ❌ UserQuotaTest > testThrottledProducerConsumer(String, String).quorum=kraft.groupProtocol=classic
FAILED ❌ UserQuotaTest > testThrottledProducerConsumer(String, String).quorum=kraft.groupProtocol=consumer
FAILED ❌ LogRecoveryTest > "testHWCheckpointWithFailuresMultipleLogSegments(String).quorum=kraft"
```
I have re-run the failed jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20845: Don't flush empty batch when appending large records in group coordinator (4.0) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #23209:
URL: https://github.com/apache/kafka/pull/23209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org