You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kirktrue (via GitHub)" <gi...@apache.org> on 2023/09/08 16:23:29 UTC

[GitHub] [kafka] kirktrue opened a new pull request, #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

kirktrue opened a new pull request, #14359:
URL: https://github.com/apache/kafka/pull/14359

   This continues the work of providing the groundwork for the fetch refactoring work by introducing some new classes and refactoring the existing code to use the new classes where applicable.
   
   Changes:
   
   * Minor clean up of the events classes to make data immutable, private, and implement `toString()`.
   * Added `IdempotentCloser` which prevents a resource from being closed more than once. It's general enough that it could be used elsewhere in the project, but it's limited to the consumer internals for now.
   * Extract inner classes from `AbstractFetch`  into top-level classes to a) buffer raw results (`FetchBuffer`), and b) collect raw results into `ConsumerRecords` (`FetchCollector`). These can be tested and changed in isolation from the core fetcher logic.
   * Added `NodeStatusDetector` which abstracts methods from `ConsumerNetworkClient` so that it and `NetworkClientDelegate` can be used in `AbstractFetch` via the interface instead of using `ConsumerNetworkClient` directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323614899


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -205,14 +190,14 @@ protected void handleFetchResponse(final Node fetchTarget,
      * Implements the core logic for a failed fetch request/response.
      *
      * @param fetchTarget {@link Node} from which the fetch data was requested
-     * @param e {@link RuntimeException} representing the error that resulted in the failure
+     * @param t {@link RuntimeException} representing the error that resulted in the failure

Review Comment:
   Changed JavaDoc to match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325056350


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;

Review Comment:
   I hadn't addressed it because I wasn't 100% sure if that was accurate. I temporarily added some assertion checking to verify that and re-run the unit tests and they passed, so I'll take that as sufficient confirmation. I've updated the class-level documentation to include that fact. LMK if you're OK with the comments. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323659568


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/CommitApplicationEvent.java:
##########
@@ -19,46 +19,33 @@
 import org.apache.kafka.clients.consumer.OffsetAndMetadata;
 import org.apache.kafka.common.TopicPartition;
 
+import java.util.Collections;
 import java.util.Map;
-import java.util.Optional;
-import java.util.concurrent.CompletableFuture;
 
-public class CommitApplicationEvent extends ApplicationEvent {
-    final private CompletableFuture<Void> future;
-    final private Map<TopicPartition, OffsetAndMetadata> offsets;
+public class CommitApplicationEvent extends CompletableApplicationEvent<Void> {
+
+    private final Map<TopicPartition, OffsetAndMetadata> offsets;
 
     public CommitApplicationEvent(final Map<TopicPartition, OffsetAndMetadata> offsets) {
         super(Type.COMMIT);
-        this.offsets = offsets;
-        Optional<Exception> exception = isValid(offsets);
-        if (exception.isPresent()) {
-            throw new RuntimeException(exception.get());
+        this.offsets = Collections.unmodifiableMap(offsets);
+        for (OffsetAndMetadata offsetAndMetadata : offsets.values()) {
+            if (offsetAndMetadata.offset() < 0) {
+                throw new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset());
+            }
         }
-        this.future = new CompletableFuture<>();
-    }
-
-    public CompletableFuture<Void> future() {
-        return future;
     }
 
     public Map<TopicPartition, OffsetAndMetadata> offsets() {
         return offsets;
     }
 
-    private Optional<Exception> isValid(final Map<TopicPartition, OffsetAndMetadata> offsets) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            TopicPartition topicPartition = entry.getKey();
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            if (offsetAndMetadata.offset() < 0) {
-                return Optional.of(new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset()));
-            }
-        }
-        return Optional.empty();
-    }
-
     @Override
     public String toString() {
-        return "CommitApplicationEvent("
-                + "offsets=" + offsets + ")";
+        return "CommitApplicationEvent{" +
+                "offsets=" + offsets +
+                ", future=" + future +
+                ", type=" + type +

Review Comment:
   LMK if you're OK with the way I went about solving this. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323646178


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;

Review Comment:
   I went ahead and added `FetchUtils` back so that this utility method lives in a neutral third party.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -98,16 +115,59 @@ public void onFailure(RuntimeException e) {
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {
+        final List<RequestFuture<ClientResponse>> requestFutures = new ArrayList<>();
+        Map<Node, FetchSessionHandler.FetchRequestData> fetchRequestMap = prepareCloseFetchSessionRequests();
+
+        for (Map.Entry<Node, FetchSessionHandler.FetchRequestData> entry : fetchRequestMap.entrySet()) {
+            final Node fetchTarget = entry.getKey();
+            final FetchSessionHandler.FetchRequestData data = entry.getValue();
+            final FetchRequest.Builder request = createFetchRequest(fetchTarget, data);
+            final RequestFuture<ClientResponse> responseFuture = client.send(fetchTarget, request);
+
+            responseFuture.addListener(new RequestFutureListener<ClientResponse>() {
+                @Override
+                public void onSuccess(ClientResponse value) {
+                    handleCloseFetchSessionResponse(fetchTarget, data);
+                }
+
+                @Override
+                public void onFailure(RuntimeException e) {
+                    handleCloseFetchSessionResponse(fetchTarget, data, e);
+                }
+            });
+
+            requestFutures.add(responseFuture);
         }
 
+        // Poll to ensure that request has been written to the socket. Wait until either the timer has expired or until
+        // all requests have received a response.
+        while (timer.notExpired() && !requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            client.poll(timer, null, true);
+        }
+
+        if (!requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            // we ran out of time before completing all futures. It is ok since we don't want to block the shutdown
+            // here.
+            log.debug("All requests couldn't be sent in the specific timeout period {}ms. " +
+                    "This may result in unnecessary fetch sessions at the broker. Consider increasing the timeout passed for " +
+                    "KafkaConsumer.close(Duration timeout)", timer.timeoutMs());
+        }
+    }
+
+    @Override
+    protected void closeInternal(final Timer timer) {
         // Shared states (e.g. sessionHandlers) could be accessed by multiple threads (such as heartbeat thread), hence,
         // it is necessary to acquire a lock on the fetcher instance before modifying the states.
         synchronized (this) {
-            super.close(timer);
+            // we do not need to re-enable wakeups since we are closing already
+            client.disableWakeups();
+            maybeCloseFetchSessions(timer);
+            Utils.closeQuietly(decompressionBufferSupplier, "decompressionBufferSupplier");

Review Comment:
   I agree. I'll see what I can do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326602940


##########
clients/src/main/java/org/apache/kafka/common/internals/IdempotentCloser.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.internals;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Supplier;
+
+/**
+ * {@code IdempotentCloser} encapsulates some basic logic to ensure that a given resource is only closed once.
+ * It provides methods to invoke callbacks (via optional {@link Runnable}s) for either the <em>initial</em> close
+ * and/or any <em>subsequent</em> closes.
+ *
+ * <p/>
+ *
+ * Here's an example:
+ *
+ * <pre>
+ *
+ * public class MyDataFile implements Closeable {
+ *
+ *     private final IdempotentCloser closer = new IdempotentCloser();
+ *
+ *     private final File file;
+ *
+ *     . . .
+ *
+ *     public boolean write() {
+ *         closer.maybeThrowIllegalStateException(() -> String.format("Data file %s already closed!", file));
+ *         writeToFile();
+ *     }
+ *
+ *
+ *     public boolean isClosed() {
+ *         return closer.isClosed();
+ *     }
+ *
+ *     &#064;Override
+ *     public void close() {
+ *         Runnable onInitialClose = () -> {
+ *             cleanUpFile(file);
+ *             log.debug("Data file {} closed", file);
+ *         };
+ *         Runnable onSubsequentClose = () -> {
+ *             log.warn("Data file {} already closed!", file);
+ *         };
+ *         closer.close(onInitialClose, onSubsequentClose);
+ *     }
+ * }
+ * </pre>
+ *
+ * Note that the callbacks are optional and if unused operates as a simple means to ensure resources
+ * are only closed once.
+ */
+public class IdempotentCloser implements AutoCloseable {
+
+    private final AtomicBoolean isClosed;
+
+    /**
+     * Creates an {@code IdempotentCloser} that is not yet closed.
+     */
+    public IdempotentCloser() {
+        this(false);
+    }
+
+    /**
+     * Creates an {@code IdempotentCloser} with the given initial state.
+     *
+     * @param isClosed Initial value for underlying state
+     */
+    public IdempotentCloser(boolean isClosed) {
+        this.isClosed = new AtomicBoolean(isClosed);
+    }
+
+    public void maybeThrowIllegalStateException(Supplier<String> message) {

Review Comment:
   The comment makes sense. Would `assertOpen` be a clearer method name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323750878


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;
+
+/**
+ * {@code FetchCollector} operates at the {@link RecordBatch} level, as that is what is stored in the
+ * {@link FetchBuffer}. Each {@link org.apache.kafka.common.record.Record} in the {@link RecordBatch} is converted
+ * to a {@link ConsumerRecord} and added to the returned {@link Fetch}.
+ *
+ * @param <K> Record key type
+ * @param <V> Record value type
+ */
+public class FetchCollector<K, V> {
+
+    private final Logger log;
+    private final ConsumerMetadata metadata;
+    private final SubscriptionState subscriptions;
+    private final FetchConfig<K, V> fetchConfig;
+    private final FetchMetricsManager metricsManager;
+    private final Time time;
+
+    public FetchCollector(final LogContext logContext,
+                          final ConsumerMetadata metadata,
+                          final SubscriptionState subscriptions,
+                          final FetchConfig<K, V> fetchConfig,
+                          final FetchMetricsManager metricsManager,
+                          final Time time) {
+        this.log = logContext.logger(FetchCollector.class);
+        this.metadata = metadata;
+        this.subscriptions = subscriptions;
+        this.fetchConfig = fetchConfig;
+        this.metricsManager = metricsManager;
+        this.time = time;
+    }
+
+    /**
+     * Return the fetched {@link ConsumerRecord records}, empty the {@link FetchBuffer record buffer}, and
+     * update the consumed position.
+     *
+     * </p>
+     *
+     * NOTE: returning an {@link Fetch#empty() empty} fetch guarantees the consumed position is not updated.
+     *
+     * @param fetchBuffer {@link FetchBuffer} from which to retrieve the {@link ConsumerRecord records}
+     *
+     * @return A {@link Fetch} for the requested partitions
+     * @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
+     *         the defaultResetPolicy is NONE
+     * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
+     */
+    public Fetch<K, V> collectFetch(final FetchBuffer fetchBuffer) {
+        final Fetch<K, V> fetch = Fetch.empty();
+        final Queue<CompletedFetch> pausedCompletedFetches = new ArrayDeque<>();
+        int recordsRemaining = fetchConfig.maxPollRecords;
+
+        try {
+            while (recordsRemaining > 0) {
+                final CompletedFetch nextInLineFetch = fetchBuffer.nextInLineFetch();
+
+                if (nextInLineFetch == null || nextInLineFetch.isConsumed()) {
+                    final CompletedFetch records = fetchBuffer.peek();
+
+                    if (records == null)
+                        break;
+
+                    if (!records.isInitialized()) {
+                        try {
+                            fetchBuffer.setNextInLineFetch(initialize(records));

Review Comment:
   I'm gonna call a time out on this because I've spent a couple of hours looking into this and am getting nowhere.
   
   My attempts are either a) subtly wrong because all tests fail!, or b) no more clear than the code is now.
   
   It's an irritatingly impregnable bit of code, I'll admit. Do you mind if I file a Jira and come back to this a bit later?
   
   LMK. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #14359:
URL: https://github.com/apache/kafka/pull/14359#issuecomment-3372367774

   ```
           // Return all partitions that are in an otherwise fetchable state *and* for which we don't already have some
           // messages sitting in our buffer.
           return new HashSet<>(subscriptions.fetchablePartitions(isNotBuffered));
   ```
   
   Out of curiosity, why are they being copied into a new `Set` collection? The `ShareConsumeRequestManager` directly use the `List` type without an extra deep copy. Also, the elements returned by `subscriptions.fetchablePartitions` should already be deduplicated.
   
   https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java#L1076


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323771865


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {

Review Comment:
   `tryConnect` and `maybeTryConnect` seem highly coupled. Why is only the former part of the `NodeStatusDetector` interface? Also, it seems that we have used `maybeTryConnect`, but not `tryConnect` in this class.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchUtils.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+
+/**
+ * {@code FetchUtils} provides a place for disparate parts of the fetch logic to live.
+ */
+public class FetchUtils {
+
+    /**
+     * Performs two combined actions based on the state related to the {@link TopicPartition}:
+     *
+     * <ol>
+     *     <li>
+     *         Invokes {@link ConsumerMetadata#requestUpdate(boolean)} to signal that the metadata is incorrect and
+     *         needs to be updated
+     *     </li>
+     *     <li>
+     *         Invokes {@link SubscriptionState#clearPreferredReadReplica(TopicPartition)} to clear out any read replica
+     *         information that may be present.
+     *     </li>
+     * </ol>
+     *
+     * This utility method should be invoked if the client detects (or is told by a node in the broker) that an
+     * attempt was made to fetch from a node that isn't the leader or preferred replica.
+     *
+     * @param metadata {@link ConsumerMetadata} for which to request an update
+     * @param subscriptions {@link SubscriptionState} to clear any internal read replica node
+     * @param topicPartition {@link TopicPartition} for which this state change is related
+     */
+    static void requestMetadataUpdate(final ConsumerMetadata metadata,
+                                      final SubscriptionState subscriptions,
+                                      final TopicPartition topicPartition) {
+        metadata.requestUpdate(true);

Review Comment:
   This was set to `false`. Why is it changed to `true`? I think `false` is correct since we don't want to reset the metadata retry count when receiving retriable error due to stale metadata.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NodeStatusDetector.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.KafkaClient;
+import org.apache.kafka.clients.NetworkClientUtils;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.utils.Time;
+
+/**
+ * Use {@code NodeStatusDetector} to determine the status of a given broker {@link Node}. It's also
+ * possible to check for previous authentication errors if the node isn't available.
+ *
+ * @see ConsumerNetworkClient
+ * @see NetworkClientDelegate
+ */
+public interface NodeStatusDetector {
+
+    /**
+     * Check if the node is disconnected and unavailable for immediate reconnection (i.e. if it is in
+     * reconnect backoff window following the disconnect).
+     *
+     * @param node {@link Node} to check for availability
+     * @see NetworkClientUtils#isUnavailable(KafkaClient, Node, Time)
+     */
+    boolean isUnavailable(Node node);
+
+    /**
+     * Checks for an authentication error on a given node and throws the exception if it exists.
+     *
+     * @param node {@link Node} to check for a previous {@link AuthenticationException}; if found it is thrown
+     * @see NetworkClientUtils#maybeThrowAuthFailure(KafkaClient, Node)
+     */
+    void maybeThrowAuthFailure(Node node);
+
+    /**
+     * Initiate a connection if currently possible. This is only really useful for resetting

Review Comment:
   Hmm, the implementation in ConsumerNetworkClient initiates a connection, but the implementation in NetworkClientDelegate doesn't. So, is this the right abstraction to capture both?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {
+        if (completedFetch != null && !partitions.contains(completedFetch.partition)) {
+            log.debug("Removing {} from buffered fetch data as it is not in the set of partitions to retain ({})", completedFetch.partition, partitions);
+            completedFetch.drain();
+            return true;
+        } else {
+            return false;
+        }
+    }
+
+    Set<TopicPartition> partitions() {

Review Comment:
   Perhaps bufferedPartitions?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;

Review Comment:
   Has this been addressed?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -256,138 +258,23 @@ protected FetchRequest.Builder createFetchRequest(final Node fetchTarget,
     }
 
     /**
-     * Return the fetched records, empty the record buffer and update the consumed position.
-     *
-     * </p>
+     * Return the list of <em>fetchable</em> partitions, which are the set of partitions to which we are subscribed,
+     * but <em>excluding</em> any partitions for which we still have buffered data. The idea is that since the user
+     * has yet to process the data for the partition that has already been fetch, we should not go send for more data

Review Comment:
   been fetch => been fetched



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -91,23 +105,70 @@ public void onFailure(RuntimeException e) {
                 }
             };
 
-            final RequestFuture<ClientResponse> future = client.send(fetchTarget, request);
+            final RequestFuture<ClientResponse> future = nodeStatusDetector.send(fetchTarget, request);
             future.addListener(listener);
         }
 
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {

Review Comment:
   Hmm, I don't quite understand this refactoring now. This method only depends on `nodeStatusDetector`, which is in `AbstractFetch`. Why is this method moved here instead of remaining in `AbstractFetch`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323659054


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {
+        if (completedFetch != null && !partitions.contains(completedFetch.partition)) {
+            log.debug("Removing {} from buffered fetch data as it is not in the set of partitions to retain ({})", completedFetch.partition, partitions);
+            completedFetch.drain();
+            return true;
+        } else {
+            return false;
+        }
+    }
+
+    Set<TopicPartition> partitions() {

Review Comment:
   The partitions in the buffer are actually not the fetchable partitions, "by definition." When it comes time to send out fetch requests, if there is still data for a given partition in the buffer, we intentionally skip over it it as the user has not processed it yet.
   
   I've added some documentation/comments around this area to hopefully make that clear.
   
   LMK what you think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1327394904


##########
clients/src/main/java/org/apache/kafka/common/internals/IdempotentCloser.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.internals;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Supplier;
+
+/**
+ * {@code IdempotentCloser} encapsulates some basic logic to ensure that a given resource is only closed once.
+ * It provides methods to invoke callbacks (via optional {@link Runnable}s) for either the <em>initial</em> close
+ * and/or any <em>subsequent</em> closes.
+ *
+ * <p/>
+ *
+ * Here's an example:
+ *
+ * <pre>
+ *
+ * public class MyDataFile implements Closeable {
+ *
+ *     private final IdempotentCloser closer = new IdempotentCloser();
+ *
+ *     private final File file;
+ *
+ *     . . .
+ *
+ *     public boolean write() {
+ *         closer.maybeThrowIllegalStateException(() -> String.format("Data file %s already closed!", file));
+ *         writeToFile();
+ *     }
+ *
+ *
+ *     public boolean isClosed() {
+ *         return closer.isClosed();
+ *     }
+ *
+ *     &#064;Override
+ *     public void close() {
+ *         Runnable onInitialClose = () -> {
+ *             cleanUpFile(file);
+ *             log.debug("Data file {} closed", file);
+ *         };
+ *         Runnable onSubsequentClose = () -> {
+ *             log.warn("Data file {} already closed!", file);
+ *         };
+ *         closer.close(onInitialClose, onSubsequentClose);
+ *     }
+ * }
+ * </pre>
+ *
+ * Note that the callbacks are optional and if unused operates as a simple means to ensure resources
+ * are only closed once.
+ */
+public class IdempotentCloser implements AutoCloseable {
+
+    private final AtomicBoolean isClosed;
+
+    /**
+     * Creates an {@code IdempotentCloser} that is not yet closed.
+     */
+    public IdempotentCloser() {
+        this(false);
+    }
+
+    /**
+     * Creates an {@code IdempotentCloser} with the given initial state.
+     *
+     * @param isClosed Initial value for underlying state
+     */
+    public IdempotentCloser(boolean isClosed) {
+        this.isClosed = new AtomicBoolean(isClosed);
+    }
+
+    public void maybeThrowIllegalStateException(Supplier<String> message) {

Review Comment:
   Agreed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323626613


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NodeStatusDetector.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.KafkaClient;
+import org.apache.kafka.clients.NetworkClientUtils;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.utils.Time;
+
+/**
+ * Use {@code NodeStatusDetector} to determine the status of a given broker {@link Node}. It's also
+ * possible to check for previous authentication errors if the node isn't available.
+ *
+ * @see ConsumerNetworkClient
+ * @see NetworkClientDelegate
+ */
+public interface NodeStatusDetector {

Review Comment:
   One implementation is using the existing `ConsumerNetworkClient` which is closely related to the network/threading model of the existing `KafkaConsumer`. The newer implementation is `NetworkClientDelegate` which does not need the synchronization and odds-and-ends that `ConsumerNetworkClient` needs.
   
   `NodeStatusDetector` gives us a general interface that a good chunk of the refactored code in `AbstractFetch` and elsewhere can use without having to tie us to a single implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323614034


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -67,35 +54,34 @@ public abstract class AbstractFetch<K, V> implements Closeable {
 
     private final Logger log;
     protected final LogContext logContext;
-    protected final ConsumerNetworkClient client;
+    protected final NodeStatusDetector nodeStatusDetector;
     protected final ConsumerMetadata metadata;
     protected final SubscriptionState subscriptions;
     protected final FetchConfig<K, V> fetchConfig;
     protected final Time time;
     protected final FetchMetricsManager metricsManager;
+    protected final FetchBuffer fetchBuffer;
+    protected final BufferSupplier decompressionBufferSupplier;
+    protected final Set<Integer> nodesWithPendingFetchRequests;
+    protected final IdempotentCloser idempotentCloser = new IdempotentCloser();
 
-    private final BufferSupplier decompressionBufferSupplier;
-    private final ConcurrentLinkedQueue<CompletedFetch<K, V>> completedFetches;
     private final Map<Integer, FetchSessionHandler> sessionHandlers;
-    private final Set<Integer> nodesWithPendingFetchRequests;
-
-    private CompletedFetch<K, V> nextInLineFetch;
 
     public AbstractFetch(final LogContext logContext,
-                         final ConsumerNetworkClient client,
+                         NodeStatusDetector nodeStatusDetector,

Review Comment:
   Yes. Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1322084716


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {
+        if (completedFetch != null && !partitions.contains(completedFetch.partition)) {
+            log.debug("Removing {} from buffered fetch data as it is not in the set of partitions to retain ({})", completedFetch.partition, partitions);
+            completedFetch.drain();
+            return true;
+        } else {
+            return false;
+        }
+    }
+
+    Set<TopicPartition> partitions() {

Review Comment:
   partitions => fetchablePartitions?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;
+
+/**
+ * {@code FetchCollector} operates at the {@link RecordBatch} level, as that is what is stored in the
+ * {@link FetchBuffer}. Each {@link org.apache.kafka.common.record.Record} in the {@link RecordBatch} is converted
+ * to a {@link ConsumerRecord} and added to the returned {@link Fetch}.
+ *
+ * @param <K> Record key type
+ * @param <V> Record value type
+ */
+public class FetchCollector<K, V> {
+
+    private final Logger log;
+    private final ConsumerMetadata metadata;
+    private final SubscriptionState subscriptions;
+    private final FetchConfig<K, V> fetchConfig;
+    private final FetchMetricsManager metricsManager;
+    private final Time time;
+
+    public FetchCollector(final LogContext logContext,
+                          final ConsumerMetadata metadata,
+                          final SubscriptionState subscriptions,
+                          final FetchConfig<K, V> fetchConfig,
+                          final FetchMetricsManager metricsManager,
+                          final Time time) {
+        this.log = logContext.logger(FetchCollector.class);
+        this.metadata = metadata;
+        this.subscriptions = subscriptions;
+        this.fetchConfig = fetchConfig;
+        this.metricsManager = metricsManager;
+        this.time = time;
+    }
+
+    /**
+     * Return the fetched {@link ConsumerRecord records}, empty the {@link FetchBuffer record buffer}, and
+     * update the consumed position.
+     *
+     * </p>
+     *
+     * NOTE: returning an {@link Fetch#empty() empty} fetch guarantees the consumed position is not updated.
+     *
+     * @param fetchBuffer {@link FetchBuffer} from which to retrieve the {@link ConsumerRecord records}
+     *
+     * @return A {@link Fetch} for the requested partitions
+     * @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
+     *         the defaultResetPolicy is NONE
+     * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
+     */
+    public Fetch<K, V> collectFetch(final FetchBuffer fetchBuffer) {
+        final Fetch<K, V> fetch = Fetch.empty();
+        final Queue<CompletedFetch> pausedCompletedFetches = new ArrayDeque<>();
+        int recordsRemaining = fetchConfig.maxPollRecords;
+
+        try {
+            while (recordsRemaining > 0) {
+                final CompletedFetch nextInLineFetch = fetchBuffer.nextInLineFetch();
+
+                if (nextInLineFetch == null || nextInLineFetch.isConsumed()) {
+                    final CompletedFetch records = fetchBuffer.peek();

Review Comment:
   This is an existing issue. But would it be better to name `records` `completedFetch`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -98,16 +115,59 @@ public void onFailure(RuntimeException e) {
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {
+        final List<RequestFuture<ClientResponse>> requestFutures = new ArrayList<>();
+        Map<Node, FetchSessionHandler.FetchRequestData> fetchRequestMap = prepareCloseFetchSessionRequests();
+
+        for (Map.Entry<Node, FetchSessionHandler.FetchRequestData> entry : fetchRequestMap.entrySet()) {
+            final Node fetchTarget = entry.getKey();
+            final FetchSessionHandler.FetchRequestData data = entry.getValue();
+            final FetchRequest.Builder request = createFetchRequest(fetchTarget, data);
+            final RequestFuture<ClientResponse> responseFuture = client.send(fetchTarget, request);
+
+            responseFuture.addListener(new RequestFutureListener<ClientResponse>() {
+                @Override
+                public void onSuccess(ClientResponse value) {
+                    handleCloseFetchSessionResponse(fetchTarget, data);
+                }
+
+                @Override
+                public void onFailure(RuntimeException e) {
+                    handleCloseFetchSessionResponse(fetchTarget, data, e);
+                }
+            });
+
+            requestFutures.add(responseFuture);
         }
 
+        // Poll to ensure that request has been written to the socket. Wait until either the timer has expired or until
+        // all requests have received a response.
+        while (timer.notExpired() && !requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            client.poll(timer, null, true);
+        }
+
+        if (!requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            // we ran out of time before completing all futures. It is ok since we don't want to block the shutdown
+            // here.
+            log.debug("All requests couldn't be sent in the specific timeout period {}ms. " +
+                    "This may result in unnecessary fetch sessions at the broker. Consider increasing the timeout passed for " +
+                    "KafkaConsumer.close(Duration timeout)", timer.timeoutMs());
+        }
+    }
+
+    @Override
+    protected void closeInternal(final Timer timer) {
         // Shared states (e.g. sessionHandlers) could be accessed by multiple threads (such as heartbeat thread), hence,
         // it is necessary to acquire a lock on the fetcher instance before modifying the states.
         synchronized (this) {
-            super.close(timer);
+            // we do not need to re-enable wakeups since we are closing already
+            client.disableWakeups();
+            maybeCloseFetchSessions(timer);
+            Utils.closeQuietly(decompressionBufferSupplier, "decompressionBufferSupplier");

Review Comment:
   `decompressionBufferSupplier` is in AbstractFetch. So, it feels a bit weird that it's being closed in `Fetcher`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);

Review Comment:
   Is this the correct implementation? It doesn't seem tryConnectNodes is being used.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -405,7 +274,7 @@ private List<TopicPartition> fetchablePartitions() {
      * @param partition {@link TopicPartition} for which we want to fetch data
      * @param leaderReplica {@link Node} for the leader of the given partition
      * @param currentTimeMs Current time in milliseconds; used to determine if we're within the optional lease window
-     * @return Replica {@link Node node} from which to request the data
+     * @return Replic {@link Node node} from which to request the data

Review Comment:
   Replica



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;

Review Comment:
   It feels a bit weird for FetchCollector to depend on AbstractFetch.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NodeStatusDetector.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.KafkaClient;
+import org.apache.kafka.clients.NetworkClientUtils;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.utils.Time;
+
+/**
+ * Use {@code NodeStatusDetector} to determine the status of a given broker {@link Node}. It's also
+ * possible to check for previous authentication errors if the node isn't available.
+ *
+ * @see ConsumerNetworkClient
+ * @see NetworkClientDelegate
+ */
+public interface NodeStatusDetector {

Review Comment:
   Why do we need two implementations of NodeStatusDetector? They both seem to call NetworkClientUtils.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -67,35 +54,34 @@ public abstract class AbstractFetch<K, V> implements Closeable {
 
     private final Logger log;
     protected final LogContext logContext;
-    protected final ConsumerNetworkClient client;
+    protected final NodeStatusDetector nodeStatusDetector;
     protected final ConsumerMetadata metadata;
     protected final SubscriptionState subscriptions;
     protected final FetchConfig<K, V> fetchConfig;
     protected final Time time;
     protected final FetchMetricsManager metricsManager;
+    protected final FetchBuffer fetchBuffer;
+    protected final BufferSupplier decompressionBufferSupplier;
+    protected final Set<Integer> nodesWithPendingFetchRequests;
+    protected final IdempotentCloser idempotentCloser = new IdempotentCloser();
 
-    private final BufferSupplier decompressionBufferSupplier;
-    private final ConcurrentLinkedQueue<CompletedFetch<K, V>> completedFetches;
     private final Map<Integer, FetchSessionHandler> sessionHandlers;
-    private final Set<Integer> nodesWithPendingFetchRequests;
-
-    private CompletedFetch<K, V> nextInLineFetch;
 
     public AbstractFetch(final LogContext logContext,
-                         final ConsumerNetworkClient client,
+                         NodeStatusDetector nodeStatusDetector,

Review Comment:
   Should this be final too?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);
+    }
+
+    public void maybeTryConnect() {

Review Comment:
   This seems unused?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {

Review Comment:
   Should this be private?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -205,14 +190,14 @@ protected void handleFetchResponse(final Node fetchTarget,
      * Implements the core logic for a failed fetch request/response.
      *
      * @param fetchTarget {@link Node} from which the fetch data was requested
-     * @param e {@link RuntimeException} representing the error that resulted in the failure
+     * @param t {@link RuntimeException} representing the error that resulted in the failure

Review Comment:
   t is now a Throwable, not RuntimeException.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;
+
+/**
+ * {@code FetchCollector} operates at the {@link RecordBatch} level, as that is what is stored in the
+ * {@link FetchBuffer}. Each {@link org.apache.kafka.common.record.Record} in the {@link RecordBatch} is converted
+ * to a {@link ConsumerRecord} and added to the returned {@link Fetch}.
+ *
+ * @param <K> Record key type
+ * @param <V> Record value type
+ */
+public class FetchCollector<K, V> {
+
+    private final Logger log;
+    private final ConsumerMetadata metadata;
+    private final SubscriptionState subscriptions;
+    private final FetchConfig<K, V> fetchConfig;
+    private final FetchMetricsManager metricsManager;
+    private final Time time;
+
+    public FetchCollector(final LogContext logContext,
+                          final ConsumerMetadata metadata,
+                          final SubscriptionState subscriptions,
+                          final FetchConfig<K, V> fetchConfig,
+                          final FetchMetricsManager metricsManager,
+                          final Time time) {
+        this.log = logContext.logger(FetchCollector.class);
+        this.metadata = metadata;
+        this.subscriptions = subscriptions;
+        this.fetchConfig = fetchConfig;
+        this.metricsManager = metricsManager;
+        this.time = time;
+    }
+
+    /**
+     * Return the fetched {@link ConsumerRecord records}, empty the {@link FetchBuffer record buffer}, and
+     * update the consumed position.
+     *
+     * </p>
+     *
+     * NOTE: returning an {@link Fetch#empty() empty} fetch guarantees the consumed position is not updated.
+     *
+     * @param fetchBuffer {@link FetchBuffer} from which to retrieve the {@link ConsumerRecord records}
+     *
+     * @return A {@link Fetch} for the requested partitions
+     * @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
+     *         the defaultResetPolicy is NONE
+     * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
+     */
+    public Fetch<K, V> collectFetch(final FetchBuffer fetchBuffer) {
+        final Fetch<K, V> fetch = Fetch.empty();
+        final Queue<CompletedFetch> pausedCompletedFetches = new ArrayDeque<>();
+        int recordsRemaining = fetchConfig.maxPollRecords;
+
+        try {
+            while (recordsRemaining > 0) {
+                final CompletedFetch nextInLineFetch = fetchBuffer.nextInLineFetch();
+
+                if (nextInLineFetch == null || nextInLineFetch.isConsumed()) {
+                    final CompletedFetch records = fetchBuffer.peek();
+
+                    if (records == null)
+                        break;
+
+                    if (!records.isInitialized()) {
+                        try {
+                            fetchBuffer.setNextInLineFetch(initialize(records));

Review Comment:
   This is ok. However, it feels a bit unnatural since we take `records` from `fetchBuffer` only to call `fetchBuffer` to set the next in line. Could `fetchBuffer.peek()`, `initialize` and `setNextInLineFetch` be done in one method inside fetchBuffer?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CompletedFetch.java:
##########
@@ -253,7 +248,7 @@ private Record nextFetchedRecord() {
      * @param maxRecords The number of records to return; the number returned may be {@code 0 <= maxRecords}
      * @return {@link ConsumerRecord Consumer records}
      */
-    List<ConsumerRecord<K, V>> fetchRecords(int maxRecords) {
+    <K, V> List<ConsumerRecord<K, V>> fetchRecords(FetchConfig<K, V> fetchConfig, int maxRecords) {

Review Comment:
   Missing javadoc for `fetchConfig`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/CommitApplicationEvent.java:
##########
@@ -19,46 +19,33 @@
 import org.apache.kafka.clients.consumer.OffsetAndMetadata;
 import org.apache.kafka.common.TopicPartition;
 
+import java.util.Collections;
 import java.util.Map;
-import java.util.Optional;
-import java.util.concurrent.CompletableFuture;
 
-public class CommitApplicationEvent extends ApplicationEvent {
-    final private CompletableFuture<Void> future;
-    final private Map<TopicPartition, OffsetAndMetadata> offsets;
+public class CommitApplicationEvent extends CompletableApplicationEvent<Void> {
+
+    private final Map<TopicPartition, OffsetAndMetadata> offsets;
 
     public CommitApplicationEvent(final Map<TopicPartition, OffsetAndMetadata> offsets) {
         super(Type.COMMIT);
-        this.offsets = offsets;
-        Optional<Exception> exception = isValid(offsets);
-        if (exception.isPresent()) {
-            throw new RuntimeException(exception.get());
+        this.offsets = Collections.unmodifiableMap(offsets);
+        for (OffsetAndMetadata offsetAndMetadata : offsets.values()) {
+            if (offsetAndMetadata.offset() < 0) {
+                throw new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset());
+            }
         }
-        this.future = new CompletableFuture<>();
-    }
-
-    public CompletableFuture<Void> future() {
-        return future;
     }
 
     public Map<TopicPartition, OffsetAndMetadata> offsets() {
         return offsets;
     }
 
-    private Optional<Exception> isValid(final Map<TopicPartition, OffsetAndMetadata> offsets) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            TopicPartition topicPartition = entry.getKey();
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            if (offsetAndMetadata.offset() < 0) {
-                return Optional.of(new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset()));
-            }
-        }
-        return Optional.empty();
-    }
-
     @Override
     public String toString() {
-        return "CommitApplicationEvent("
-                + "offsets=" + offsets + ")";
+        return "CommitApplicationEvent{" +
+                "offsets=" + offsets +
+                ", future=" + future +
+                ", type=" + type +

Review Comment:
   Could we restructure the `toString` code to avoid each event duplicating the shared fields in the parent class?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;

Review Comment:
   Could we add a comment that there is only up to one CompletedFetch per partition in `completedFetches`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -50,7 +55,8 @@
 public class Fetcher<K, V> extends AbstractFetch<K, V> {
 
     private final Logger log;
-    private final AtomicBoolean isClosed = new AtomicBoolean(false);
+    private final FetchCollector<K, V> fetchCollector;
+    private final ConsumerNetworkClient client;

Review Comment:
   Hmm, why do we need to duplicate `client` in `Fetcher` and `AbstractFetch`? `client` is only used in `maybeCloseFetchSessions`. Could we move maybeCloseFetchSessions to AbstractFetch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager
URL: https://github.com/apache/kafka/pull/14359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #14359:
URL: https://github.com/apache/kafka/pull/14359#issuecomment-1721974877

   @junrao yeah, lots of errors in the socket/selector level for some reason:
   
   ```
   kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.testProduceConsumeWithWildcardAcls()
   kafka.api.TransactionsTest.testBumpTransactionalEpoch()
   kafka.network.SocketServerTest.testControlPlaneAsPrivilegedListener()
   kafka.network.SocketServerTest.testControlPlaneTakePrecedenceOverInterBrokerListenerAsPrivilegedListener()
   kafka.network.SocketServerTest.testInterBrokerListenerAsPrivilegedListener()
   kafka.network.SocketServerTest.testStagedListenerShutdownWhenConnectionQueueIsFull()
   kafka.network.SocketServerTest.testStagedListenerStartup()
   kafka.network.SocketServerTest.testZeroMaxConnectionsPerIp()
   kafka.server.DescribeClusterRequestTest.testDescribeClusterRequestExcludingClusterAuthorizedOperations()
   kafka.server.DescribeClusterRequestTest.testDescribeClusterRequestIncludingClusterAuthorizedOperations()
   o.a.k.common.network.SelectorTest.testConnectionsByClientMetric()
   o.a.k.common.network.SelectorTest.testInboundConnectionsCountInConnectionCreationMetric()
   o.a.k.common.network.SelectorTest.testMuteOnOOM()
   o.a.k.common.network.Tls12SelectorTest.testConnectionsByClientMetric()
   o.a.k.common.network.Tls12SelectorTest.testInboundConnectionsCountInConnectionCreationMetric()
   o.a.k.common.network.Tls12SelectorTest.testMuteOnOOM()
   o.a.k.common.network.Tls13SelectorTest.testConnectionsByClientMetric()
   o.a.k.common.network.Tls13SelectorTest.testInboundConnectionsCountInConnectionCreationMetric()
   o.a.k.common.network.Tls13SelectorTest.testMuteOnOOM()
   o.a.k.connect.integration.ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector()
   o.a.k.controller.QuorumControllerTest.testBalancePartitionLeaders()
   o.a.k.image.loader.MetadataLoaderTest.testNoPublishEmptyImage()
   o.a.k.streams.integration.NamedTopologyIntegrationTest.shouldRemoveOneNamedTopologyWhileAnotherContinuesProcessing()
   o.a.k.tiered.storage.integration.TransactionsWithTieredStoreTest.testAbortTransactionTimeout()
   o.a.k.tiered.storage.integration.TransactionsWithTieredStoreTest.testSendOffsetsToTransactionTimeout()
   o.a.k.tools.JmxToolTest.initializationError()
   ```
   
   I don't think any of these are related, but I'm going to force a rebuild, just to check.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326507379


##########
clients/src/main/java/org/apache/kafka/common/internals/IdempotentCloser.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.internals;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Supplier;
+
+/**
+ * {@code IdempotentCloser} encapsulates some basic logic to ensure that a given resource is only closed once.
+ * It provides methods to invoke callbacks (via optional {@link Runnable}s) for either the <em>initial</em> close
+ * and/or any <em>subsequent</em> closes.
+ *
+ * <p/>
+ *
+ * Here's an example:
+ *
+ * <pre>
+ *
+ * public class MyDataFile implements Closeable {
+ *
+ *     private final IdempotentCloser closer = new IdempotentCloser();
+ *
+ *     private final File file;
+ *
+ *     . . .
+ *
+ *     public boolean write() {
+ *         closer.maybeThrowIllegalStateException(() -> String.format("Data file %s already closed!", file));
+ *         writeToFile();
+ *     }
+ *
+ *
+ *     public boolean isClosed() {
+ *         return closer.isClosed();
+ *     }
+ *
+ *     &#064;Override
+ *     public void close() {
+ *         Runnable onInitialClose = () -> {
+ *             cleanUpFile(file);
+ *             log.debug("Data file {} closed", file);
+ *         };
+ *         Runnable onSubsequentClose = () -> {
+ *             log.warn("Data file {} already closed!", file);
+ *         };
+ *         closer.close(onInitialClose, onSubsequentClose);
+ *     }
+ * }
+ * </pre>
+ *
+ * Note that the callbacks are optional and if unused operates as a simple means to ensure resources
+ * are only closed once.
+ */
+public class IdempotentCloser implements AutoCloseable {
+
+    private final AtomicBoolean isClosed;
+
+    /**
+     * Creates an {@code IdempotentCloser} that is not yet closed.
+     */
+    public IdempotentCloser() {
+        this(false);
+    }
+
+    /**
+     * Creates an {@code IdempotentCloser} with the given initial state.
+     *
+     * @param isClosed Initial value for underlying state
+     */
+    public IdempotentCloser(boolean isClosed) {
+        this.isClosed = new AtomicBoolean(isClosed);
+    }
+
+    public void maybeThrowIllegalStateException(Supplier<String> message) {

Review Comment:
   Yes. I added comments to the methods in this class. LMK if they there's additional detail needed. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323660196


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NodeStatusDetector.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.KafkaClient;
+import org.apache.kafka.clients.NetworkClientUtils;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.utils.Time;
+
+/**
+ * Use {@code NodeStatusDetector} to determine the status of a given broker {@link Node}. It's also
+ * possible to check for previous authentication errors if the node isn't available.
+ *
+ * @see ConsumerNetworkClient
+ * @see NetworkClientDelegate
+ */
+public interface NodeStatusDetector {

Review Comment:
   LMK if you still have questions/concerns here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323758178


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;

Review Comment:
   LMK if you're OK with this. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager
URL: https://github.com/apache/kafka/pull/14359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323494877


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;
+
+/**
+ * {@code FetchCollector} operates at the {@link RecordBatch} level, as that is what is stored in the
+ * {@link FetchBuffer}. Each {@link org.apache.kafka.common.record.Record} in the {@link RecordBatch} is converted
+ * to a {@link ConsumerRecord} and added to the returned {@link Fetch}.
+ *
+ * @param <K> Record key type
+ * @param <V> Record value type
+ */
+public class FetchCollector<K, V> {
+
+    private final Logger log;
+    private final ConsumerMetadata metadata;
+    private final SubscriptionState subscriptions;
+    private final FetchConfig<K, V> fetchConfig;
+    private final FetchMetricsManager metricsManager;
+    private final Time time;
+
+    public FetchCollector(final LogContext logContext,
+                          final ConsumerMetadata metadata,
+                          final SubscriptionState subscriptions,
+                          final FetchConfig<K, V> fetchConfig,
+                          final FetchMetricsManager metricsManager,
+                          final Time time) {
+        this.log = logContext.logger(FetchCollector.class);
+        this.metadata = metadata;
+        this.subscriptions = subscriptions;
+        this.fetchConfig = fetchConfig;
+        this.metricsManager = metricsManager;
+        this.time = time;
+    }
+
+    /**
+     * Return the fetched {@link ConsumerRecord records}, empty the {@link FetchBuffer record buffer}, and
+     * update the consumed position.
+     *
+     * </p>
+     *
+     * NOTE: returning an {@link Fetch#empty() empty} fetch guarantees the consumed position is not updated.
+     *
+     * @param fetchBuffer {@link FetchBuffer} from which to retrieve the {@link ConsumerRecord records}
+     *
+     * @return A {@link Fetch} for the requested partitions
+     * @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
+     *         the defaultResetPolicy is NONE
+     * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
+     */
+    public Fetch<K, V> collectFetch(final FetchBuffer fetchBuffer) {
+        final Fetch<K, V> fetch = Fetch.empty();
+        final Queue<CompletedFetch> pausedCompletedFetches = new ArrayDeque<>();
+        int recordsRemaining = fetchConfig.maxPollRecords;
+
+        try {
+            while (recordsRemaining > 0) {
+                final CompletedFetch nextInLineFetch = fetchBuffer.nextInLineFetch();
+
+                if (nextInLineFetch == null || nextInLineFetch.isConsumed()) {
+                    final CompletedFetch records = fetchBuffer.peek();
+
+                    if (records == null)
+                        break;
+
+                    if (!records.isInitialized()) {
+                        try {
+                            fetchBuffer.setNextInLineFetch(initialize(records));

Review Comment:
   I absolutely agree that it's awkward. This is essentially copy-and-paste from the existing implementation and I wanted to change things as little as possible. I'll take another look and see if I can make it more intuitive and easy to follow.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.OffsetOutOfRangeException;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.RecordTooLargeException;
+import org.apache.kafka.common.errors.TopicAuthorizationException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.requests.FetchResponse;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+
+import static org.apache.kafka.clients.consumer.internals.AbstractFetch.requestMetadataUpdate;
+
+/**
+ * {@code FetchCollector} operates at the {@link RecordBatch} level, as that is what is stored in the
+ * {@link FetchBuffer}. Each {@link org.apache.kafka.common.record.Record} in the {@link RecordBatch} is converted
+ * to a {@link ConsumerRecord} and added to the returned {@link Fetch}.
+ *
+ * @param <K> Record key type
+ * @param <V> Record value type
+ */
+public class FetchCollector<K, V> {
+
+    private final Logger log;
+    private final ConsumerMetadata metadata;
+    private final SubscriptionState subscriptions;
+    private final FetchConfig<K, V> fetchConfig;
+    private final FetchMetricsManager metricsManager;
+    private final Time time;
+
+    public FetchCollector(final LogContext logContext,
+                          final ConsumerMetadata metadata,
+                          final SubscriptionState subscriptions,
+                          final FetchConfig<K, V> fetchConfig,
+                          final FetchMetricsManager metricsManager,
+                          final Time time) {
+        this.log = logContext.logger(FetchCollector.class);
+        this.metadata = metadata;
+        this.subscriptions = subscriptions;
+        this.fetchConfig = fetchConfig;
+        this.metricsManager = metricsManager;
+        this.time = time;
+    }
+
+    /**
+     * Return the fetched {@link ConsumerRecord records}, empty the {@link FetchBuffer record buffer}, and
+     * update the consumed position.
+     *
+     * </p>
+     *
+     * NOTE: returning an {@link Fetch#empty() empty} fetch guarantees the consumed position is not updated.
+     *
+     * @param fetchBuffer {@link FetchBuffer} from which to retrieve the {@link ConsumerRecord records}
+     *
+     * @return A {@link Fetch} for the requested partitions
+     * @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
+     *         the defaultResetPolicy is NONE
+     * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
+     */
+    public Fetch<K, V> collectFetch(final FetchBuffer fetchBuffer) {
+        final Fetch<K, V> fetch = Fetch.empty();
+        final Queue<CompletedFetch> pausedCompletedFetches = new ArrayDeque<>();
+        int recordsRemaining = fetchConfig.maxPollRecords;
+
+        try {
+            while (recordsRemaining > 0) {
+                final CompletedFetch nextInLineFetch = fetchBuffer.nextInLineFetch();
+
+                if (nextInLineFetch == null || nextInLineFetch.isConsumed()) {
+                    final CompletedFetch records = fetchBuffer.peek();

Review Comment:
   Will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323495460


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {

Review Comment:
   Sure. Will change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323611170


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -405,7 +274,7 @@ private List<TopicPartition> fetchablePartitions() {
      * @param partition {@link TopicPartition} for which we want to fetch data
      * @param leaderReplica {@link Node} for the leader of the given partition
      * @param currentTimeMs Current time in milliseconds; used to determine if we're within the optional lease window
-     * @return Replica {@link Node node} from which to request the data
+     * @return Replic {@link Node node} from which to request the data

Review Comment:
   Not sure how this keeps coming back 😦 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -405,7 +274,7 @@ private List<TopicPartition> fetchablePartitions() {
      * @param partition {@link TopicPartition} for which we want to fetch data
      * @param leaderReplica {@link Node} for the leader of the given partition
      * @param currentTimeMs Current time in milliseconds; used to determine if we're within the optional lease window
-     * @return Replica {@link Node node} from which to request the data
+     * @return Replic {@link Node node} from which to request the data

Review Comment:
   Not sure how this keeps coming back 😦 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -405,7 +274,7 @@ private List<TopicPartition> fetchablePartitions() {
      * @param partition {@link TopicPartition} for which we want to fetch data
      * @param leaderReplica {@link Node} for the leader of the given partition
      * @param currentTimeMs Current time in milliseconds; used to determine if we're within the optional lease window
-     * @return Replica {@link Node node} from which to request the data
+     * @return Replic {@link Node node} from which to request the data

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326504231


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -421,14 +309,45 @@ Node selectReadReplica(final TopicPartition partition, final Node leaderReplica,
                         " using the leader instead.", nodeId, partition);
                 // Note that this condition may happen due to stale metadata, so we clear preferred replica and
                 // refresh metadata.
-                requestMetadataUpdate(partition);
+                requestMetadataUpdate(metadata, subscriptions, partition);
                 return leaderReplica;
             }
         } else {
             return leaderReplica;
         }
     }
 
+    protected Map<Node, FetchSessionHandler.FetchRequestData> prepareCloseFetchSessionRequests() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager
URL: https://github.com/apache/kafka/pull/14359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323645279


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);

Review Comment:
   It is now 😉
   
   Unlike the `ConsumerNetworkClient`, we don't attempt to connect to the node immediately. We queue up the set of `Node`s to which to connect and call them from the background thread shortly before calling `poll`. The existing implementation would attempt to connect immediately, and potentially more than once in some cases.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);

Review Comment:
   It is now 😉
   
   Unlike the `ConsumerNetworkClient`, we don't attempt to connect to the node immediately. We queue up the set of `Node`s to which to connect and call them from the background thread shortly before calling `poll`. The existing implementation would attempt to connect immediately, and potentially more than once in some cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323758616


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -98,16 +115,59 @@ public void onFailure(RuntimeException e) {
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {
+        final List<RequestFuture<ClientResponse>> requestFutures = new ArrayList<>();
+        Map<Node, FetchSessionHandler.FetchRequestData> fetchRequestMap = prepareCloseFetchSessionRequests();
+
+        for (Map.Entry<Node, FetchSessionHandler.FetchRequestData> entry : fetchRequestMap.entrySet()) {
+            final Node fetchTarget = entry.getKey();
+            final FetchSessionHandler.FetchRequestData data = entry.getValue();
+            final FetchRequest.Builder request = createFetchRequest(fetchTarget, data);
+            final RequestFuture<ClientResponse> responseFuture = client.send(fetchTarget, request);
+
+            responseFuture.addListener(new RequestFutureListener<ClientResponse>() {
+                @Override
+                public void onSuccess(ClientResponse value) {
+                    handleCloseFetchSessionResponse(fetchTarget, data);
+                }
+
+                @Override
+                public void onFailure(RuntimeException e) {
+                    handleCloseFetchSessionResponse(fetchTarget, data, e);
+                }
+            });
+
+            requestFutures.add(responseFuture);
         }
 
+        // Poll to ensure that request has been written to the socket. Wait until either the timer has expired or until
+        // all requests have received a response.
+        while (timer.notExpired() && !requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            client.poll(timer, null, true);
+        }
+
+        if (!requestFutures.stream().allMatch(RequestFuture::isDone)) {
+            // we ran out of time before completing all futures. It is ok since we don't want to block the shutdown
+            // here.
+            log.debug("All requests couldn't be sent in the specific timeout period {}ms. " +
+                    "This may result in unnecessary fetch sessions at the broker. Consider increasing the timeout passed for " +
+                    "KafkaConsumer.close(Duration timeout)", timer.timeoutMs());
+        }
+    }
+
+    @Override
+    protected void closeInternal(final Timer timer) {
         // Shared states (e.g. sessionHandlers) could be accessed by multiple threads (such as heartbeat thread), hence,
         // it is necessary to acquire a lock on the fetcher instance before modifying the states.
         synchronized (this) {
-            super.close(timer);
+            // we do not need to re-enable wakeups since we are closing already
+            client.disableWakeups();
+            maybeCloseFetchSessions(timer);
+            Utils.closeQuietly(decompressionBufferSupplier, "decompressionBufferSupplier");

Review Comment:
   I pushed a change to clean up how the resources in `AbstractFetch` and `Fetcher` are released in the `close()` methods. The unit and integration tests pass (locally) and it _makes_ sense to me, but please LMK if there's anything that doesn't sit right with the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325174579


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -91,23 +105,70 @@ public void onFailure(RuntimeException e) {
                 }
             };
 
-            final RequestFuture<ClientResponse> future = client.send(fetchTarget, request);
+            final RequestFuture<ClientResponse> future = nodeStatusDetector.send(fetchTarget, request);
             future.addListener(listener);
         }
 
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {

Review Comment:
   Yes, it is a bit obtuse. My apologies.
   
   There are two reasons that `maybeCloseFetchSessions` exists in `Fetcher`:
   
   1. The forthcoming `FetchRequestManager` doesn't close sessions in the same way, so it doesn't make sense to have the method in the shared `AbstractFetch` class.
   2. `FetchRequestManager` (as with the other `RequestManager`s) doesn't use the `ConsumerNetworkClient` but uses the `NetworkClientDelegate`. The code in `AbstractFetch` that is shared by both `Fetcher` and—soon—`FetchRequestManager` needed a way to call some of the methods that `ConsumerNetworkClient` and `NetworkClientDelegate` have in common. Thus the `NodeStatusDetector` interface was created.
   
   Because the `FetchRequestManager` isn't in this PR, it's understandably unclear the reason for the current structure. For now I've made the following changes:
   
   1. I moved `maybeCloseFetchSessions` to `AbstractFetch`
   2. I left `AbstractFetch` with a reference to a `ConsumerNetworkClient`
   3. I've removed the `NodeStatusDetector`
   
   As needed, the above may be changed in forthcoming PRs.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -91,23 +105,70 @@ public void onFailure(RuntimeException e) {
                 }
             };
 
-            final RequestFuture<ClientResponse> future = client.send(fetchTarget, request);
+            final RequestFuture<ClientResponse> future = nodeStatusDetector.send(fetchTarget, request);
             future.addListener(listener);
         }
 
         return fetchRequestMap.size();
     }
 
-    public void close(final Timer timer) {
-        if (!isClosed.compareAndSet(false, true)) {
-            log.info("Fetcher {} is already closed.", this);
-            return;
+    public Fetch<K, V> collectFetch() {
+        return fetchCollector.collectFetch(fetchBuffer);
+    }
+
+    protected void maybeCloseFetchSessions(final Timer timer) {

Review Comment:
   Yes, it is a bit obtuse. My apologies.
   
   There are two reasons that `maybeCloseFetchSessions` exists in `Fetcher`:
   
   1. The forthcoming `FetchRequestManager` doesn't close sessions in the same way, so it doesn't make sense to have the method in the shared `AbstractFetch` class.
   2. `FetchRequestManager` (as with the other `RequestManager`s) doesn't use the `ConsumerNetworkClient` but uses the `NetworkClientDelegate`. The code in `AbstractFetch` that is shared by both `Fetcher` and—soon—`FetchRequestManager` needed a way to call some of the methods that `ConsumerNetworkClient` and `NetworkClientDelegate` have in common. Thus the `NodeStatusDetector` interface was created.
   
   Because the `FetchRequestManager` isn't in this PR, it's understandably unclear the reason for the current structure. For now I've made the following changes:
   
   1. I moved `maybeCloseFetchSessions` to `AbstractFetch`
   2. I left `AbstractFetch` with a reference to a `ConsumerNetworkClient`
   3. I've removed the `NodeStatusDetector`
   
   As needed, the above may be changed in forthcoming PRs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326504874


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CompletedFetchTest.java:
##########
@@ -115,13 +122,17 @@ public void testCommittedTransactionRecordsIncluded() {
         Records rawRecords = newTranscactionalRecords(ControlRecordType.COMMIT, numRecords);
         FetchResponseData.PartitionData partitionData = new FetchResponseData.PartitionData()
                 .setRecords(rawRecords);
-        CompletedFetch<String, String> completedFetch = newCompletedFetch(IsolationLevel.READ_COMMITTED,
-                OffsetResetStrategy.NONE,
-                true,
-                0,
+        CompletedFetch completedFetch = newCompletedFetch(0,

Review Comment:
   Done. I double-checked that I caught the places where the method args were unnecessarily broken over two lines, but let me know if I missed any. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323610487


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CompletedFetch.java:
##########
@@ -253,7 +248,7 @@ private Record nextFetchedRecord() {
      * @param maxRecords The number of records to return; the number returned may be {@code 0 <= maxRecords}
      * @return {@link ConsumerRecord Consumer records}
      */
-    List<ConsumerRecord<K, V>> fetchRecords(int maxRecords) {
+    <K, V> List<ConsumerRecord<K, V>> fetchRecords(FetchConfig<K, V> fetchConfig, int maxRecords) {

Review Comment:
   Done.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/CommitApplicationEvent.java:
##########
@@ -19,46 +19,33 @@
 import org.apache.kafka.clients.consumer.OffsetAndMetadata;
 import org.apache.kafka.common.TopicPartition;
 
+import java.util.Collections;
 import java.util.Map;
-import java.util.Optional;
-import java.util.concurrent.CompletableFuture;
 
-public class CommitApplicationEvent extends ApplicationEvent {
-    final private CompletableFuture<Void> future;
-    final private Map<TopicPartition, OffsetAndMetadata> offsets;
+public class CommitApplicationEvent extends CompletableApplicationEvent<Void> {
+
+    private final Map<TopicPartition, OffsetAndMetadata> offsets;
 
     public CommitApplicationEvent(final Map<TopicPartition, OffsetAndMetadata> offsets) {
         super(Type.COMMIT);
-        this.offsets = offsets;
-        Optional<Exception> exception = isValid(offsets);
-        if (exception.isPresent()) {
-            throw new RuntimeException(exception.get());
+        this.offsets = Collections.unmodifiableMap(offsets);
+        for (OffsetAndMetadata offsetAndMetadata : offsets.values()) {
+            if (offsetAndMetadata.offset() < 0) {
+                throw new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset());
+            }
         }
-        this.future = new CompletableFuture<>();
-    }
-
-    public CompletableFuture<Void> future() {
-        return future;
     }
 
     public Map<TopicPartition, OffsetAndMetadata> offsets() {
         return offsets;
     }
 
-    private Optional<Exception> isValid(final Map<TopicPartition, OffsetAndMetadata> offsets) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            TopicPartition topicPartition = entry.getKey();
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            if (offsetAndMetadata.offset() < 0) {
-                return Optional.of(new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset()));
-            }
-        }
-        return Optional.empty();
-    }
-
     @Override
     public String toString() {
-        return "CommitApplicationEvent("
-                + "offsets=" + offsets + ")";
+        return "CommitApplicationEvent{" +
+                "offsets=" + offsets +
+                ", future=" + future +
+                ", type=" + type +

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323634434


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);
+    }
+
+    public void maybeTryConnect() {

Review Comment:
   Yes, it was. Thanks for catching this. It's now executed in the `DefaultBackgroundThread` right before we send out requests.
   
   This is an artifact of manually merging different commits from the staging branch into a single PR for `trunk` 😞 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);
+    }
+
+    public void maybeTryConnect() {

Review Comment:
   Yes, it was. Thanks for catching this. It's now executed in the `DefaultBackgroundThread` right before we send out requests.
   
   This is an artifact of manually merging different commits from the staging branch into a single PR for `trunk` 😞 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325061757


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -256,138 +258,23 @@ protected FetchRequest.Builder createFetchRequest(final Node fetchTarget,
     }
 
     /**
-     * Return the fetched records, empty the record buffer and update the consumed position.
-     *
-     * </p>
+     * Return the list of <em>fetchable</em> partitions, which are the set of partitions to which we are subscribed,
+     * but <em>excluding</em> any partitions for which we still have buffered data. The idea is that since the user
+     * has yet to process the data for the partition that has already been fetch, we should not go send for more data

Review Comment:
   Fixed. Also made this method return a `Set` which is more appropriate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323660004


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {
+        tryConnectNodes.add(node);

Review Comment:
   LMK if that makes sense to you (or not). Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323783693


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -50,7 +55,8 @@
 public class Fetcher<K, V> extends AbstractFetch<K, V> {
 
     private final Logger log;
-    private final AtomicBoolean isClosed = new AtomicBoolean(false);
+    private final FetchCollector<K, V> fetchCollector;
+    private final ConsumerNetworkClient client;

Review Comment:
   I made a change here that I'm still not crazy about.
   
   Instead of having a `NodeStatusDetector` instance variable in `AbstractFetch`, I introduced a type parameter in `AbstractFetch` that allows a subclass to provide a more specific type. Now `Fetcher` can provide the type as `ConsumerNetworkClient` and thus it can call the specialized methods on the instance variable from `AbstractFetch`.
   
   The part I'm not crazy about is that the instance variable is still called `nodeStatusDetector`. I think I would be a bit confused when I see calls like `nodeStatusDetector.disableWakeups()`. But maybe that's just me. I thought of changing the name of the instance variable to `client` or `networkInfo` or other equally vague names.
   
   If you have any input/suggestions/preferences, do LMK. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326276789


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CompletedFetchTest.java:
##########
@@ -115,13 +122,17 @@ public void testCommittedTransactionRecordsIncluded() {
         Records rawRecords = newTranscactionalRecords(ControlRecordType.COMMIT, numRecords);
         FetchResponseData.PartitionData partitionData = new FetchResponseData.PartitionData()
                 .setRecords(rawRecords);
-        CompletedFetch<String, String> completedFetch = newCompletedFetch(IsolationLevel.READ_COMMITTED,
-                OffsetResetStrategy.NONE,
-                true,
-                0,
+        CompletedFetch completedFetch = newCompletedFetch(0,

Review Comment:
   Could we merge the next line in? Ditto for similar places below.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchBufferTest.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.Time;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createFetchMetricsManager;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createMetrics;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createSubscriptionState;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * This tests the {@link FetchBuffer} functionality in addition to what {@link FetcherTest} tests during the course
+ * of its tests. One of the main concerns of these tests are that we correctly handle both places that data is held
+ * internally:
+ *
+ * <ol>
+ *     <li>A special "next in line" buffer</li>
+ *     <li>The remainder of the buffers in a queue</li>
+ * </ol>
+ */
+public class FetchBufferTest {
+
+    private final Time time = new MockTime(0, 0, 0);
+    private final TopicPartition topicAPartition0 = new TopicPartition("topic-a", 0);
+    private final TopicPartition topicAPartition1 = new TopicPartition("topic-a", 1);
+    private final TopicPartition topicAPartition2 = new TopicPartition("topic-a", 2);
+    private final Set<TopicPartition> allPartitions = partitions(topicAPartition0, topicAPartition1, topicAPartition2);
+    private LogContext logContext;
+
+    private SubscriptionState subscriptions;
+
+    private FetchMetricsManager metricsManager;
+
+    @BeforeEach
+    public void setup() {
+        logContext = new LogContext();
+
+        Properties p = new Properties();
+        p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+        p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+        p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+        ConsumerConfig config = new ConsumerConfig(p);
+
+        subscriptions = createSubscriptionState(config, logContext);
+
+        Metrics metrics = createMetrics(config, time);
+        metricsManager = createFetchMetricsManager(metrics);
+    }
+
+    /**
+     * Verifies the basics: we can add buffered data to the queue, peek to view them, and poll to remove them.
+     */
+    @Test
+    public void testBasicPeekAndPoll() {
+        try (FetchBuffer fetchBuffer = new FetchBuffer(logContext)) {
+            CompletedFetch completedFetch = completedFetch(topicAPartition0);
+            assertTrue(fetchBuffer.isEmpty());
+            fetchBuffer.add(completedFetch);
+            assertTrue(fetchBuffer.hasCompletedFetches(p -> true));
+            assertFalse(fetchBuffer.isEmpty());
+            assertNotNull(fetchBuffer.peek());
+            assertSame(completedFetch, fetchBuffer.peek());
+            assertSame(completedFetch, fetchBuffer.poll());
+            assertNull(fetchBuffer.peek());
+        }
+    }
+
+    /**
+     * Verifies {@link FetchBuffer#close()}} closes the buffered data for both the queue and the next-in-line buffer.
+     */
+    @Test
+    public void testCloseClearsData() {
+        FetchBuffer fetchBuffer = null;
+
+        try {
+            fetchBuffer = new FetchBuffer(logContext);
+            assertNull(fetchBuffer.nextInLineFetch());
+            assertTrue(fetchBuffer.isEmpty());
+
+            fetchBuffer.add(completedFetch(topicAPartition0));
+            assertFalse(fetchBuffer.isEmpty());
+
+            fetchBuffer.setNextInLineFetch(completedFetch(topicAPartition0));
+            assertNotNull(fetchBuffer.nextInLineFetch());
+        } finally {
+            if (fetchBuffer != null)
+                fetchBuffer.close();
+        }
+
+        assertNull(fetchBuffer.nextInLineFetch());
+        assertTrue(fetchBuffer.isEmpty());
+    }
+
+    /**
+     * Tests that the buffer returns partitions for both the queue and the next-in-line buffer.
+     */
+    @Test
+    public void testPartitions() {

Review Comment:
   testBufferedPartitions?



##########
clients/src/main/java/org/apache/kafka/common/internals/IdempotentCloser.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.internals;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Supplier;
+
+/**
+ * {@code IdempotentCloser} encapsulates some basic logic to ensure that a given resource is only closed once.
+ * It provides methods to invoke callbacks (via optional {@link Runnable}s) for either the <em>initial</em> close
+ * and/or any <em>subsequent</em> closes.
+ *
+ * <p/>
+ *
+ * Here's an example:
+ *
+ * <pre>
+ *
+ * public class MyDataFile implements Closeable {
+ *
+ *     private final IdempotentCloser closer = new IdempotentCloser();
+ *
+ *     private final File file;
+ *
+ *     . . .
+ *
+ *     public boolean write() {
+ *         closer.maybeThrowIllegalStateException(() -> String.format("Data file %s already closed!", file));
+ *         writeToFile();
+ *     }
+ *
+ *
+ *     public boolean isClosed() {
+ *         return closer.isClosed();
+ *     }
+ *
+ *     &#064;Override
+ *     public void close() {
+ *         Runnable onInitialClose = () -> {
+ *             cleanUpFile(file);
+ *             log.debug("Data file {} closed", file);
+ *         };
+ *         Runnable onSubsequentClose = () -> {
+ *             log.warn("Data file {} already closed!", file);
+ *         };
+ *         closer.close(onInitialClose, onSubsequentClose);
+ *     }
+ * }
+ * </pre>
+ *
+ * Note that the callbacks are optional and if unused operates as a simple means to ensure resources
+ * are only closed once.
+ */
+public class IdempotentCloser implements AutoCloseable {
+
+    private final AtomicBoolean isClosed;
+
+    /**
+     * Creates an {@code IdempotentCloser} that is not yet closed.
+     */
+    public IdempotentCloser() {
+        this(false);
+    }
+
+    /**
+     * Creates an {@code IdempotentCloser} with the given initial state.
+     *
+     * @param isClosed Initial value for underlying state
+     */
+    public IdempotentCloser(boolean isClosed) {
+        this.isClosed = new AtomicBoolean(isClosed);
+    }
+
+    public void maybeThrowIllegalStateException(Supplier<String> message) {

Review Comment:
   Could we describe what the method does?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchBufferTest.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.Time;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createFetchMetricsManager;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createMetrics;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createSubscriptionState;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * This tests the {@link FetchBuffer} functionality in addition to what {@link FetcherTest} tests during the course
+ * of its tests. One of the main concerns of these tests are that we correctly handle both places that data is held
+ * internally:
+ *
+ * <ol>
+ *     <li>A special "next in line" buffer</li>

Review Comment:
   Are these comments relevant to this class?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java:
##########
@@ -421,14 +309,45 @@ Node selectReadReplica(final TopicPartition partition, final Node leaderReplica,
                         " using the leader instead.", nodeId, partition);
                 // Note that this condition may happen due to stale metadata, so we clear preferred replica and
                 // refresh metadata.
-                requestMetadataUpdate(partition);
+                requestMetadataUpdate(metadata, subscriptions, partition);
                 return leaderReplica;
             }
         } else {
             return leaderReplica;
         }
     }
 
+    protected Map<Node, FetchSessionHandler.FetchRequestData> prepareCloseFetchSessionRequests() {

Review Comment:
   This is only used inside AbstractFetch. Could this be private?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325058583


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchUtils.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+
+/**
+ * {@code FetchUtils} provides a place for disparate parts of the fetch logic to live.
+ */
+public class FetchUtils {
+
+    /**
+     * Performs two combined actions based on the state related to the {@link TopicPartition}:
+     *
+     * <ol>
+     *     <li>
+     *         Invokes {@link ConsumerMetadata#requestUpdate(boolean)} to signal that the metadata is incorrect and
+     *         needs to be updated
+     *     </li>
+     *     <li>
+     *         Invokes {@link SubscriptionState#clearPreferredReadReplica(TopicPartition)} to clear out any read replica
+     *         information that may be present.
+     *     </li>
+     * </ol>
+     *
+     * This utility method should be invoked if the client detects (or is told by a node in the broker) that an
+     * attempt was made to fetch from a node that isn't the leader or preferred replica.
+     *
+     * @param metadata {@link ConsumerMetadata} for which to request an update
+     * @param subscriptions {@link SubscriptionState} to clear any internal read replica node
+     * @param topicPartition {@link TopicPartition} for which this state change is related
+     */
+    static void requestMetadataUpdate(final ConsumerMetadata metadata,
+                                      final SubscriptionState subscriptions,
+                                      final TopicPartition topicPartition) {
+        metadata.requestUpdate(true);

Review Comment:
   You're right. I double-checked the (very helpful) comments in `requestUpdate()` and I agree, so I set it to `false`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325174944


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NodeStatusDetector.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.KafkaClient;
+import org.apache.kafka.clients.NetworkClientUtils;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.utils.Time;
+
+/**
+ * Use {@code NodeStatusDetector} to determine the status of a given broker {@link Node}. It's also
+ * possible to check for previous authentication errors if the node isn't available.
+ *
+ * @see ConsumerNetworkClient
+ * @see NetworkClientDelegate
+ */
+public interface NodeStatusDetector {
+
+    /**
+     * Check if the node is disconnected and unavailable for immediate reconnection (i.e. if it is in
+     * reconnect backoff window following the disconnect).
+     *
+     * @param node {@link Node} to check for availability
+     * @see NetworkClientUtils#isUnavailable(KafkaClient, Node, Time)
+     */
+    boolean isUnavailable(Node node);
+
+    /**
+     * Checks for an authentication error on a given node and throws the exception if it exists.
+     *
+     * @param node {@link Node} to check for a previous {@link AuthenticationException}; if found it is thrown
+     * @see NetworkClientUtils#maybeThrowAuthFailure(KafkaClient, Node)
+     */
+    void maybeThrowAuthFailure(Node node);
+
+    /**
+     * Initiate a connection if currently possible. This is only really useful for resetting

Review Comment:
   Per the above comments, I've removed `NodeStatusDetector` for now, so this should be moot for the time being.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325171010


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {

Review Comment:
   Yes—`tryConnect` and `maybeTryConnect` were intended to be used together. The reason the latter wasn't in the interface was because of the difference in _when_ we perform network I/O. In the existing `Consumer`, we perform it immediately, whereas in the refactored `Consumer`, we delay that I/O until the last step. So the intent was to for the new `Consumer` to queue up the connection attempts and perform them in one step in `maybeTryConnect` vs. immediately when `tryConnect` was called.
   
   That said, a) it's confusing, and b) it's not needed in this particular PR, so I've removed the whole `NodeStatusDetector` for now.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -63,6 +68,31 @@ public NetworkClientDelegate(
         this.unsentRequests = new ArrayDeque<>();
         this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
         this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
+        this.tryConnectNodes = new HashSet<>();
+    }
+
+    @Override
+    public boolean isUnavailable(Node node) {
+        return NetworkClientUtils.isUnavailable(client, node, time);
+    }
+
+    @Override
+    public void maybeThrowAuthFailure(Node node) {
+        NetworkClientUtils.maybeThrowAuthFailure(client, node);
+    }
+
+    @Override
+    public void tryConnect(Node node) {

Review Comment:
   Yes—`tryConnect` and `maybeTryConnect` were intended to be used together. The reason the latter wasn't in the interface was because of the difference in _when_ we perform network I/O. In the existing `Consumer`, we perform it immediately, whereas in the refactored `Consumer`, we delay that I/O until the last step. So the intent was to for the new `Consumer` to queue up the connection attempts and perform them in one step in `maybeTryConnect` vs. immediately when `tryConnect` was called.
   
   That said, a) it's confusing, and b) it's not needed in this particular PR, so I've removed the whole `NodeStatusDetector` for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1323772849


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -50,7 +55,8 @@
 public class Fetcher<K, V> extends AbstractFetch<K, V> {
 
     private final Logger log;
-    private final AtomicBoolean isClosed = new AtomicBoolean(false);
+    private final FetchCollector<K, V> fetchCollector;
+    private final ConsumerNetworkClient client;

Review Comment:
   I agree, it's ugly 😦
   
   `client` and `nodeStatusDetector` both point at the same object in `Fetcher` and `AbstractFetch`, respectively, but the two classes use them for different purposes.
   
   The `ConsumerNetworkClient` used in the `Fetcher` invokes methods like `poll()`, `send()`, and `disableWakeups()` to perform network I/O "directly." The forthcoming `FetchRequestManager` adheres to the design principle that request managers don't perform network I/O directly. To enforce that, we introduced this intentionally limited `NodeStatusDetector` to allow for checking basic state of nodes.
   
   I'll keep looking at it a bit to see if I can come up with something cleaner because I don't like it either.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java:
##########
@@ -50,7 +55,8 @@
 public class Fetcher<K, V> extends AbstractFetch<K, V> {
 
     private final Logger log;
-    private final AtomicBoolean isClosed = new AtomicBoolean(false);
+    private final FetchCollector<K, V> fetchCollector;
+    private final ConsumerNetworkClient client;

Review Comment:
   I agree, it's ugly 😦
   
   `client` and `nodeStatusDetector` both point at the same object in `Fetcher` and `AbstractFetch`, respectively, but the two classes use them for different purposes.
   
   The `ConsumerNetworkClient` used in the `Fetcher` invokes methods like `poll()`, `send()`, and `disableWakeups()` to perform network I/O "directly." The forthcoming `FetchRequestManager` adheres to the design principle that request managers don't perform network I/O directly. To enforce that, we introduced this intentionally limited `NodeStatusDetector` to allow for checking basic state of nodes.
   
   I'll keep looking at it a bit to see if I can come up with something cleaner because I don't like it either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1325038436


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.internals.IdempotentCloser;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.function.Predicate;
+
+/**
+ * {@code FetchBuffer} buffers up the results from the broker responses as they are received. It is essentially a
+ * wrapper around a {@link java.util.Queue} of {@link CompletedFetch}.
+ *
+ * <p/>
+ *
+ * <em>Note</em>: this class is not thread-safe and is intended to only be used from a single thread.
+ */
+public class FetchBuffer implements Closeable {
+
+    private final Logger log;
+    private final ConcurrentLinkedQueue<CompletedFetch> completedFetches;
+    private final IdempotentCloser idempotentCloser = new IdempotentCloser();
+
+    private CompletedFetch nextInLineFetch;
+
+    public FetchBuffer(final LogContext logContext) {
+        this.log = logContext.logger(FetchBuffer.class);
+        this.completedFetches = new ConcurrentLinkedQueue<>();
+    }
+
+    /**
+     * Returns {@code true} if there are no completed fetches pending to return to the user.
+     *
+     * @return {@code true} if the buffer is empty, {@code false} otherwise
+     */
+    boolean isEmpty() {
+        return completedFetches.isEmpty();
+    }
+
+    /**
+     * Return whether we have any completed fetches pending return to the user. This method is thread-safe. Has
+     * visibility for testing.
+     *
+     * @return {@code true} if there are completed fetches that match the {@link Predicate}, {@code false} otherwise
+     */
+    boolean hasCompletedFetches(Predicate<CompletedFetch> predicate) {
+        return completedFetches.stream().anyMatch(predicate);
+    }
+
+    void add(CompletedFetch completedFetch) {
+        completedFetches.add(completedFetch);
+    }
+
+    void addAll(Collection<CompletedFetch> completedFetches) {
+        this.completedFetches.addAll(completedFetches);
+    }
+
+    CompletedFetch nextInLineFetch() {
+        return nextInLineFetch;
+    }
+
+    void setNextInLineFetch(CompletedFetch completedFetch) {
+        this.nextInLineFetch = completedFetch;
+    }
+
+    CompletedFetch peek() {
+        return completedFetches.peek();
+    }
+
+    CompletedFetch poll() {
+        return completedFetches.poll();
+    }
+
+    /**
+     * Updates the buffer to retain only the fetch data that corresponds to the given partitions. Any previously
+     * {@link CompletedFetch fetched data} is removed if its partition is not in the given set of partitions.
+     *
+     * @param partitions {@link Set} of {@link TopicPartition}s for which any buffered data should be kept
+     */
+    void retainAll(final Set<TopicPartition> partitions) {
+        completedFetches.removeIf(cf -> maybeDrain(partitions, cf));
+
+        if (maybeDrain(partitions, nextInLineFetch))
+            nextInLineFetch = null;
+    }
+
+    boolean maybeDrain(final Set<TopicPartition> partitions, final CompletedFetch completedFetch) {
+        if (completedFetch != null && !partitions.contains(completedFetch.partition)) {
+            log.debug("Removing {} from buffered fetch data as it is not in the set of partitions to retain ({})", completedFetch.partition, partitions);
+            completedFetch.drain();
+            return true;
+        } else {
+            return false;
+        }
+    }
+
+    Set<TopicPartition> partitions() {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on a diff in pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #14359:
URL: https://github.com/apache/kafka/pull/14359#discussion_r1326506750


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchBufferTest.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.Time;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createFetchMetricsManager;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createMetrics;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createSubscriptionState;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * This tests the {@link FetchBuffer} functionality in addition to what {@link FetcherTest} tests during the course
+ * of its tests. One of the main concerns of these tests are that we correctly handle both places that data is held
+ * internally:
+ *
+ * <ol>
+ *     <li>A special "next in line" buffer</li>

Review Comment:
   Yes, but apparently, they're confusing, though 😄
   
   I tried to reword the comments, but I'm not sure if it was enough. The idea was to emphasize that the tests are trying to ensure that we keep the internal state that is split between the main buffer queue and the 'next in line' buffer consistent.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchBufferTest.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.Time;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createFetchMetricsManager;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createMetrics;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createSubscriptionState;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * This tests the {@link FetchBuffer} functionality in addition to what {@link FetcherTest} tests during the course
+ * of its tests. One of the main concerns of these tests are that we correctly handle both places that data is held
+ * internally:
+ *
+ * <ol>
+ *     <li>A special "next in line" buffer</li>

Review Comment:
   Yes, but apparently, they're confusing, though 😄
   
   I tried to reword the comments, but I'm not sure if it was enough. The idea was to emphasize that the tests are trying to ensure that we keep the internal state that is split between the main buffer queue and the 'next in line' buffer consistent.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchBufferTest.java:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.Time;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createFetchMetricsManager;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createMetrics;
+import static org.apache.kafka.clients.consumer.internals.ConsumerUtils.createSubscriptionState;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * This tests the {@link FetchBuffer} functionality in addition to what {@link FetcherTest} tests during the course
+ * of its tests. One of the main concerns of these tests are that we correctly handle both places that data is held
+ * internally:
+ *
+ * <ol>
+ *     <li>A special "next in line" buffer</li>
+ *     <li>The remainder of the buffers in a queue</li>
+ * </ol>
+ */
+public class FetchBufferTest {
+
+    private final Time time = new MockTime(0, 0, 0);
+    private final TopicPartition topicAPartition0 = new TopicPartition("topic-a", 0);
+    private final TopicPartition topicAPartition1 = new TopicPartition("topic-a", 1);
+    private final TopicPartition topicAPartition2 = new TopicPartition("topic-a", 2);
+    private final Set<TopicPartition> allPartitions = partitions(topicAPartition0, topicAPartition1, topicAPartition2);
+    private LogContext logContext;
+
+    private SubscriptionState subscriptions;
+
+    private FetchMetricsManager metricsManager;
+
+    @BeforeEach
+    public void setup() {
+        logContext = new LogContext();
+
+        Properties p = new Properties();
+        p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+        p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+        p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
+        ConsumerConfig config = new ConsumerConfig(p);
+
+        subscriptions = createSubscriptionState(config, logContext);
+
+        Metrics metrics = createMetrics(config, time);
+        metricsManager = createFetchMetricsManager(metrics);
+    }
+
+    /**
+     * Verifies the basics: we can add buffered data to the queue, peek to view them, and poll to remove them.
+     */
+    @Test
+    public void testBasicPeekAndPoll() {
+        try (FetchBuffer fetchBuffer = new FetchBuffer(logContext)) {
+            CompletedFetch completedFetch = completedFetch(topicAPartition0);
+            assertTrue(fetchBuffer.isEmpty());
+            fetchBuffer.add(completedFetch);
+            assertTrue(fetchBuffer.hasCompletedFetches(p -> true));
+            assertFalse(fetchBuffer.isEmpty());
+            assertNotNull(fetchBuffer.peek());
+            assertSame(completedFetch, fetchBuffer.peek());
+            assertSame(completedFetch, fetchBuffer.poll());
+            assertNull(fetchBuffer.peek());
+        }
+    }
+
+    /**
+     * Verifies {@link FetchBuffer#close()}} closes the buffered data for both the queue and the next-in-line buffer.
+     */
+    @Test
+    public void testCloseClearsData() {
+        FetchBuffer fetchBuffer = null;
+
+        try {
+            fetchBuffer = new FetchBuffer(logContext);
+            assertNull(fetchBuffer.nextInLineFetch());
+            assertTrue(fetchBuffer.isEmpty());
+
+            fetchBuffer.add(completedFetch(topicAPartition0));
+            assertFalse(fetchBuffer.isEmpty());
+
+            fetchBuffer.setNextInLineFetch(completedFetch(topicAPartition0));
+            assertNotNull(fetchBuffer.nextInLineFetch());
+        } finally {
+            if (fetchBuffer != null)
+                fetchBuffer.close();
+        }
+
+        assertNull(fetchBuffer.nextInLineFetch());
+        assertTrue(fetchBuffer.isEmpty());
+    }
+
+    /**
+     * Tests that the buffer returns partitions for both the queue and the next-in-line buffer.
+     */
+    @Test
+    public void testPartitions() {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue closed pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager
URL: https://github.com/apache/kafka/pull/14359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao merged pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao merged PR #14359:
URL: https://github.com/apache/kafka/pull/14359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kirktrue commented on pull request #14359: KAFKA-14274 [2-5/7]: Introduction of more infrastructure for forthcoming fetch request manager

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #14359:
URL: https://github.com/apache/kafka/pull/14359#issuecomment-1712393353

   Test failures are unrelated:
   
   - `kafka.api.ConsumerBounceTest.testClose()`
   - `kafka.server.DescribeClusterRequestTest.testDescribeClusterRequestIncludingClusterAuthorizedOperations()`: KAFKA-15419
   - `o.a.k.controller.QuorumControllerTest.testBalancePartitionLeaders()`: KAFKA-15052
   - `o.a.k.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerTest.testRemoteLogSizeCalculationWithSegmentsOfTheSameEpoch()`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org