You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lianetm (via GitHub)" <gi...@apache.org> on 2023/09/13 19:52:37 UTC

[GitHub] [kafka] lianetm opened a new pull request, #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

lianetm opened a new pull request, #14385:
URL: https://github.com/apache/kafka/pull/14385

   Support for using committed offsets to update fetch positions.
   
   This PR includes:
   - moving`refreshCommittedOffsets` function out of the existing `ConsumerCoordinator` so it can be reused (no code changes)
   - using the above `refreshCommittedOffsets` for updating fetch positions if the consumer has enabled the Kafka-based offsets management by defining a groupId
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327802559


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerUtils.java:
##########
@@ -141,4 +146,53 @@ public static <K, V> List<ConsumerInterceptor<K, V>> configuredConsumerIntercept
         return (List<ConsumerInterceptor<K, V>>) ClientUtils.configuredInterceptors(config, ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, ConsumerInterceptor.class);
     }
 
+    /**
+     * Update subscription state and metadata using the provided committed offsets:
+     * <li>Update partition offsets with the committed offsets</li>
+     * <li>Update the metadata with any newer leader epoch discovered in the committed offsets
+     * metadata</li>
+     * </p>
+     * This will ignore any partition included in the <code>offsetsAndMetadata</code> parameter that
+     * may no longer be assigned.
+     *
+     * @param offsetsAndMetadata Committed offsets and metadata to be used for updating the
+     *                           subscription state and metadata object.
+     * @param metadata           Metadata object to update with a new leader epoch if discovered in the
+     *                           committed offsets' metadata.
+     * @param subscriptions      Subscription state to update, setting partitions' offsets to the
+     *                           committed offsets.
+     * @return False if null <code>offsetsAndMetadata</code> is provided. True in any other case.

Review Comment:
   Will we ever pass in null for offsetsAndMetadata?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1329179154


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumerTest.java:
##########
@@ -361,6 +364,84 @@ public void testWakeup_committed() {
         assertNoPendingWakeup(consumer.wakeupTrigger());
     }
 
+    @Test
+    public void testRefreshCommittedOffsetsSuccess() {
+        Map<TopicPartition, OffsetAndMetadata> committedOffsets =
+                Collections.singletonMap(new TopicPartition("t1", 1), new OffsetAndMetadata(10L));
+        testRefreshCommittedOffsetsSuccess(committedOffsets);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsSuccessButNoCommittedOffsetsFound() {
+        testRefreshCommittedOffsetsSuccess(Collections.emptyMap());
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsShouldNotResetIfFailedWithTimeout() {
+        // Create consumer with group id to enable committed offset usage
+        this.groupId = "consumer-group-1";
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(true);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsNotCalledIfNoGroupId() {
+        // Create consumer without group id so committed offsets are not used for updating positions
+        this.groupId = null;
+        consumer = newConsumer(time, new StringDeserializer(), new StringDeserializer());
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(false);
+    }
+
+    private void testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(boolean committedOffsetsEnabled) {

Review Comment:
   Lianet, thanks for the explanation. Make sense to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1329117484


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumerTest.java:
##########
@@ -361,6 +364,84 @@ public void testWakeup_committed() {
         assertNoPendingWakeup(consumer.wakeupTrigger());
     }
 
+    @Test
+    public void testRefreshCommittedOffsetsSuccess() {
+        Map<TopicPartition, OffsetAndMetadata> committedOffsets =
+                Collections.singletonMap(new TopicPartition("t1", 1), new OffsetAndMetadata(10L));
+        testRefreshCommittedOffsetsSuccess(committedOffsets);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsSuccessButNoCommittedOffsetsFound() {
+        testRefreshCommittedOffsetsSuccess(Collections.emptyMap());
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsShouldNotResetIfFailedWithTimeout() {
+        // Create consumer with group id to enable committed offset usage
+        this.groupId = "consumer-group-1";
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(true);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsNotCalledIfNoGroupId() {
+        // Create consumer without group id so committed offsets are not used for updating positions
+        this.groupId = null;
+        consumer = newConsumer(time, new StringDeserializer(), new StringDeserializer());
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(false);
+    }
+
+    private void testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(boolean committedOffsetsEnabled) {

Review Comment:
   Why is the method named with Timeout? There doesn't seem to be any logic related to timeout.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327784079


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {
+        final Set<TopicPartition> initializingPartitions = subscriptions.initializingPartitions();
+
+        log.debug("Refreshing committed offsets for partitions {}", initializingPartitions);
+        try {
+            final Map<TopicPartition, OffsetAndMetadata> offsets = eventHandler.addAndGet(new OffsetFetchApplicationEvent(initializingPartitions), timer);
+            return ConsumerUtils.refreshCommittedOffsets(offsets, this.metadata, this.subscriptions);
+        } catch (org.apache.kafka.common.errors.TimeoutException e) {
+            log.error("Couldn't refresh committed offsets before timeout expired");
+            return false;
+        }
+    }

Review Comment:
   Yes, it is possible that not all were retrieved and we return false, just to indicate that the operation (retrieve offsets for all partitions that needed init) failed. This is the same behaviour of the current consumer. 
   
   If current consumer times out when retrieving committed offsets [here](https://github.com/apache/kafka/blob/f46db86d34f9e5fe1b0d7604306a5108a89c113e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L1035) , it will then return false on the `refreshCommittedOffsetsIfNeeded` [here](https://github.com/apache/kafka/blob/f46db86d34f9e5fe1b0d7604306a5108a89c113e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L961). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #14385:
URL: https://github.com/apache/kafka/pull/14385#issuecomment-1721865351

   Thanks for the comments @junrao , all addressed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327848517


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerUtils.java:
##########
@@ -141,4 +146,53 @@ public static <K, V> List<ConsumerInterceptor<K, V>> configuredConsumerIntercept
         return (List<ConsumerInterceptor<K, V>>) ClientUtils.configuredInterceptors(config, ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, ConsumerInterceptor.class);
     }
 
+    /**
+     * Update subscription state and metadata using the provided committed offsets:
+     * <li>Update partition offsets with the committed offsets</li>
+     * <li>Update the metadata with any newer leader epoch discovered in the committed offsets
+     * metadata</li>
+     * </p>
+     * This will ignore any partition included in the <code>offsetsAndMetadata</code> parameter that
+     * may no longer be assigned.
+     *
+     * @param offsetsAndMetadata Committed offsets and metadata to be used for updating the
+     *                           subscription state and metadata object.
+     * @param metadata           Metadata object to update with a new leader epoch if discovered in the
+     *                           committed offsets' metadata.
+     * @param subscriptions      Subscription state to update, setting partitions' offsets to the
+     *                           committed offsets.
+     * @return False if null <code>offsetsAndMetadata</code> is provided. True in any other case.

Review Comment:
   We won't, when calling this from the new implementation. But when calling it from the old consumer yes, `offsetsAndMetadata` can be null here (passed as null from the [fetchCommittedOffsets](https://github.com/apache/kafka/blob/f46db86d34f9e5fe1b0d7604306a5108a89c113e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L996C51-L996C72) if it times out)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327614488


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {

Review Comment:
   Could we add the new param to the javadoc?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerUtils.java:
##########
@@ -141,4 +146,36 @@ public static <K, V> List<ConsumerInterceptor<K, V>> configuredConsumerIntercept
         return (List<ConsumerInterceptor<K, V>>) ClientUtils.configuredInterceptors(config, ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, ConsumerInterceptor.class);
     }
 
+    public static boolean refreshCommittedOffsets(final Map<TopicPartition, OffsetAndMetadata> offsets,

Review Comment:
   Could we add a description of the method?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {
+        final Set<TopicPartition> initializingPartitions = subscriptions.initializingPartitions();
+
+        log.debug("Refreshing committed offsets for partitions {}", initializingPartitions);
+        try {
+            final Map<TopicPartition, OffsetAndMetadata> offsets = eventHandler.addAndGet(new OffsetFetchApplicationEvent(initializingPartitions), timer);

Review Comment:
   We block the `poll` call here to initialize the offset, but when the offset is reset, we handle the offsetReset asynchronously without blocking the `poll` call. Should we do these consistently? Does the existing consumer have the same behavior?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {
+        final Set<TopicPartition> initializingPartitions = subscriptions.initializingPartitions();
+
+        log.debug("Refreshing committed offsets for partitions {}", initializingPartitions);
+        try {
+            final Map<TopicPartition, OffsetAndMetadata> offsets = eventHandler.addAndGet(new OffsetFetchApplicationEvent(initializingPartitions), timer);
+            return ConsumerUtils.refreshCommittedOffsets(offsets, this.metadata, this.subscriptions);
+        } catch (org.apache.kafka.common.errors.TimeoutException e) {
+            log.error("Couldn't refresh committed offsets before timeout expired");
+            return false;
+        }
+    }

Review Comment:
   At this point, it's still possible that not all partitions' position have been initialized. Should we return false in that case?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.

Review Comment:
   There are no provided partitions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #14385:
URL: https://github.com/apache/kafka/pull/14385#issuecomment-1724069642

   Latest comment addressed, thanks @junrao ! I merged latest changes from trunk and updated some tests accordingly. Build completed with 4 unrelated test failures:
   
   Build / JDK 17 and Scala 2.13 / testCreateTokenForOtherUserFails(String).quorum=kraft – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest
   Build / JDK 11 and Scala 2.13 / testFencingOnSend(String).quorum=zk – org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest
   Build / JDK 11 and Scala 2.13 / shouldRemoveOneNamedTopologyWhileAnotherContinuesProcessing() – org.apache.kafka.streams.integration.NamedTopologyIntegrationTest
   Build / JDK 20 and Scala 2.13 / testReplicateSourceDefault() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327788743


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {

Review Comment:
   the `timer` param is included in the java doc, do you mean something else maybe?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327771100


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {
+        final Set<TopicPartition> initializingPartitions = subscriptions.initializingPartitions();
+
+        log.debug("Refreshing committed offsets for partitions {}", initializingPartitions);
+        try {
+            final Map<TopicPartition, OffsetAndMetadata> offsets = eventHandler.addAndGet(new OffsetFetchApplicationEvent(initializingPartitions), timer);

Review Comment:
   You described it correctly, but yes, that's how the existing consumer does it so we wanted to ensure the same behaviour. When initializing positions:
   - if using committed offsets, the current consumer blocks on [fetchCommittedOffsets](https://github.com/apache/kafka/blob/f46db86d34f9e5fe1b0d7604306a5108a89c113e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L996C51-L996C72) until it gets the committed offsets from the coordinator (or the timeout expires). 
   - if not using committed offsets, it will just transition the partitions from init to awaiting_reset [here](https://github.com/apache/kafka/blob/f46db86d34f9e5fe1b0d7604306a5108a89c113e/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L2450), and then the positions will be updated following the reset logic, that does not block.
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1327801964


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java:
##########
@@ -667,6 +683,35 @@ private static <K, V> ClusterResourceListeners configureClusterResourceListeners
         return clusterResourceListeners;
     }
 
+    /**
+     *
+     * Indicates if the consumer is using the Kafka-based offset management strategy,
+     * according to config {@link CommonClientConfigs#GROUP_ID_CONFIG}
+     */
+    private boolean isCommittedOffsetsManagementEnabled() {
+        return groupId.isPresent();
+    }
+
+    /**
+     * Refresh the committed offsets for provided partitions.
+     *
+     * @param timer Timer bounding how long this method can block
+     * @return true iff the operation completed within the timeout
+     */
+    private boolean refreshCommittedOffsetsIfNeeded(Timer timer) {

Review Comment:
   You are right. It didn't see it. Sorry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao merged pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao merged PR #14385:
URL: https://github.com/apache/kafka/pull/14385


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lianetm commented on a diff in pull request #14385: KAFKA-15306 - Integrating committed offsets for updating fetch positions

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #14385:
URL: https://github.com/apache/kafka/pull/14385#discussion_r1329165007


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumerTest.java:
##########
@@ -361,6 +364,84 @@ public void testWakeup_committed() {
         assertNoPendingWakeup(consumer.wakeupTrigger());
     }
 
+    @Test
+    public void testRefreshCommittedOffsetsSuccess() {
+        Map<TopicPartition, OffsetAndMetadata> committedOffsets =
+                Collections.singletonMap(new TopicPartition("t1", 1), new OffsetAndMetadata(10L));
+        testRefreshCommittedOffsetsSuccess(committedOffsets);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsSuccessButNoCommittedOffsetsFound() {
+        testRefreshCommittedOffsetsSuccess(Collections.emptyMap());
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsShouldNotResetIfFailedWithTimeout() {
+        // Create consumer with group id to enable committed offset usage
+        this.groupId = "consumer-group-1";
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(true);
+    }
+
+    @Test
+    public void testRefreshCommittedOffsetsNotCalledIfNoGroupId() {
+        // Create consumer without group id so committed offsets are not used for updating positions
+        this.groupId = null;
+        consumer = newConsumer(time, new StringDeserializer(), new StringDeserializer());
+
+        testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(false);
+    }
+
+    private void testUpdateFetchPositionsWithFetchCommittedOffsetsTimeout(boolean committedOffsetsEnabled) {

Review Comment:
   There is no explicit timeout exception indeed, but there is a future, provided as result of the fetch committed offsets, that is never completed:
   ```
   // Uncompleted future that will time out if used
   CompletableFuture<Map<TopicPartition, OffsetAndMetadata>> committedFuture = new CompletableFuture<>();
   ```
   So this function will time out when getting the results of the fetch committed offsets inside the `offsetFetchEventMocker`, on the `ConsumerUtils.getResult(future, timer)`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org