You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "TaiJuWu (via GitHub)" <gi...@apache.org> on 2026/02/08 00:52:01 UTC
[PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
TaiJuWu opened a new pull request, #21424:
URL: https://github.com/apache/kafka/pull/21424
Document that unsubscribe() doesn't commit offsets even with auto-commit enabled, and add test to verify this behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-3910256309
Interesting… Is is also true with the async consumer? cc @lianetm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3015013791
##########
clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:
##########
@@ -850,8 +854,9 @@ public void unsubscribe() {
* metadata change. Note that it is not possible to use both manual partition assignment with {@link #assign(Collection)}
* and group assignment with {@link #subscribe(Collection, ConsumerRebalanceListener)}.
* <p>
- * If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new
- * assignment replaces the old one.
Review Comment:
shouldn't we leave this as it is?
`assign` does have the logic to autocommit before changing the assignment (just that it relies on the timer atm, but that's being fixed with https://github.com/apache/kafka/pull/21584)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-3911298564
Since the assign method (KAFKA-20166) has a similar issue, we can discuss the side effects there. This PR focuses on updating the documentation to reflect the 'legacy' behavior.
@dajac @lianetm WDYT?
@TaiJuWu would you mind updating the documentation of "assign" as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3039800530
##########
clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:
##########
@@ -832,6 +832,10 @@ public void subscribe(SubscriptionPattern pattern) {
/**
* Unsubscribe from topics currently subscribed with {@link #subscribe(Collection)} or {@link #subscribe(Pattern)}.
* This also clears any partitions directly assigned through {@link #assign(Collection)}.
+ * <p>
+ * <b>Note:</b> Unlike {@link #close()}, this method does not commit that pending offsets before
Review Comment:
`that pending offsets` -> `the pending offsets`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3038992321
##########
clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:
##########
@@ -832,6 +832,10 @@ public void subscribe(SubscriptionPattern pattern) {
/**
* Unsubscribe from topics currently subscribed with {@link #subscribe(Collection)} or {@link #subscribe(Pattern)}.
* This also clears any partitions directly assigned through {@link #assign(Collection)}.
+ * <p>
+ * <b>Note:</b> Unlike {@link #close()}, this method does not guarantee that pending offsets are committed before
Review Comment:
"does not guarantee" is a little weak as a warning. It should state "does not commit pending offsets before unsubscribing, even if enable.auto.commit is enabled"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-3914946801
Makes sense to me @chia7712 (update docs here with current behaviour, fix separately).
As for `assign`, agree with the gap, but the situation is slightly different (assign does have the intention/logic to commit, just that it seems to wrongly check the timer)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r2784771300
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java:
##########
@@ -1814,6 +1814,38 @@ public void testUnsubscribeWithoutGroupId() {
verify(applicationEventHandler).add(ArgumentMatchers.isA(UnsubscribeEvent.class));
}
+ /**
+ * Verify that unsubscribe() does not commit offsets even when auto-commit is enabled.
+ * This ensures users are aware that they need to explicitly call commitSync() before
+ * unsubscribing to avoid duplicate processing upon re-joining the group.
+ */
+ @Test
Review Comment:
Could you consider adding a integration tests instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3038973527
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerTest.java:
##########
@@ -1823,6 +1823,57 @@ private ConsumerRecords<byte[], byte[]> awaitNonEmptyRecords(
return result.get();
}
+ @ClusterTest
+ public void testClassicConsumerUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled() throws Exception {
+ testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map.of(
+ GROUP_PROTOCOL_CONFIG, GroupProtocol.CLASSIC.name().toLowerCase(Locale.ROOT)
+ ));
+ }
+
+ @ClusterTest
+ public void testAsyncConsumerUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled() throws Exception {
+ testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map.of(
+ GROUP_PROTOCOL_CONFIG, GroupProtocol.CONSUMER.name().toLowerCase(Locale.ROOT)
+ ));
+ }
+
+ /**
+ * Verify that {@link Consumer#unsubscribe()} does not commit offsets even when
+ * {@code enable.auto.commit} is enabled. A second consumer using the same group ID
+ * should see no committed offsets after the first consumer unsubscribes.
+ */
+ private void testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map<String, Object> consumerConfig) throws Exception {
+ var numRecords = 10;
+ var groupId = "unsubscribe-no-commit-test";
+ sendRecords(cluster, TP, numRecords);
+
+ // Consumer 1: subscribe, consume records, then unsubscribe (without explicit commit)
+ Map<String, Object> config = new HashMap<>(consumerConfig);
+ config.put(GROUP_ID_CONFIG, groupId);
+ config.put(ENABLE_AUTO_COMMIT_CONFIG, true);
+ config.put(AUTO_OFFSET_RESET_CONFIG, "earliest");
+
+ try (Consumer<byte[], byte[]> consumer1 = cluster.consumer(config)) {
+ consumer1.subscribe(List.of(TOPIC));
+ consumeRecords(consumer1, numRecords);
+
+ // Unsubscribe - this should NOT commit offsets even though auto-commit is enabled
+ consumer1.unsubscribe();
+ }
+
+ // Consumer 2: use the same group ID to check committed offsets
+ Map<String, Object> verifyConfig = new HashMap<>(consumerConfig);
+ verifyConfig.put(GROUP_ID_CONFIG, groupId);
+ verifyConfig.put(ENABLE_AUTO_COMMIT_CONFIG, false);
Review Comment:
this is unnecessary, right?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerTest.java:
##########
@@ -1823,6 +1823,57 @@ private ConsumerRecords<byte[], byte[]> awaitNonEmptyRecords(
return result.get();
}
+ @ClusterTest
+ public void testClassicConsumerUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled() throws Exception {
+ testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map.of(
+ GROUP_PROTOCOL_CONFIG, GroupProtocol.CLASSIC.name().toLowerCase(Locale.ROOT)
+ ));
+ }
+
+ @ClusterTest
+ public void testAsyncConsumerUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled() throws Exception {
+ testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map.of(
+ GROUP_PROTOCOL_CONFIG, GroupProtocol.CONSUMER.name().toLowerCase(Locale.ROOT)
+ ));
+ }
+
+ /**
+ * Verify that {@link Consumer#unsubscribe()} does not commit offsets even when
+ * {@code enable.auto.commit} is enabled. A second consumer using the same group ID
+ * should see no committed offsets after the first consumer unsubscribes.
+ */
+ private void testUnsubscribeDoesNotCommitOffsetsWithAutoCommitEnabled(Map<String, Object> consumerConfig) throws Exception {
Review Comment:
If it accepts GroupProtocol instead of Map, we could construct the map more cleanly later, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-3874285651
@TaiJuWu could you check the latest comment on the jira and make corresponding updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-4115731413
@TaiJuWu would you please fix conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3015782971
##########
clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:
##########
@@ -850,8 +854,9 @@ public void unsubscribe() {
* metadata change. Note that it is not possible to use both manual partition assignment with {@link #assign(Collection)}
* and group assignment with {@link #subscribe(Collection, ConsumerRebalanceListener)}.
* <p>
- * If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new
- * assignment replaces the old one.
Review Comment:
Yes, we should revert this change and let that PR fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #21424:
URL: https://github.com/apache/kafka/pull/21424#discussion_r3041730617
##########
clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:
##########
@@ -832,6 +832,10 @@ public void subscribe(SubscriptionPattern pattern) {
/**
* Unsubscribe from topics currently subscribed with {@link #subscribe(Collection)} or {@link #subscribe(Pattern)}.
* This also clears any partitions directly assigned through {@link #assign(Collection)}.
+ * <p>
+ * <b>Note:</b> Unlike {@link #close()}, this method does not commit that pending offsets before
Review Comment:
fixed, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #21424:
URL: https://github.com/apache/kafka/pull/21424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20119: Clarify that `Consumer#unsubscribe` does not trigger auto-commit [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21424:
URL: https://github.com/apache/kafka/pull/21424#issuecomment-3910349948
Yeap, this is the same in both consumers (consumer pro-actively triggers callbacks to release assignment first, then sends leave request), so no commit in the pic (auto-commit on revocation only happens on reconciliations, when the revocation is triggered by the GC)
That being said, seems we could improve this and provide the commit guarantee I imagine (just like we ensure commit before revocation triggered by the GC). Not to push it in this PR, but to think about it, we would have to review the contract of unsubscribe and auto-commit, the app expectations, consider error handling on unsubscribe (it doesn't throw much at the moment, letting the user unsubscribe regardless of errors like auth related, etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org