You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhuoxi2000 (via GitHub)" <gi...@apache.org> on 2026/08/04 23:49:23 UTC

[PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Zhuoxi2000 opened a new pull request, #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290

   ## What is the purpose of the change
   
   Fixes [FLINK-36434](https://issues.apache.org/jira/browse/FLINK-36434): the `KafkaPartitionSplitReader` is constructed on the source-reader thread — or, when `KafkaSourceFetcherManager#commitOffsets` re-creates a fetcher after an idle shutdown, on the checkpoint thread — and its constructor eagerly created the `KafkaConsumer` (and registered consumer metrics) there. The consumer is then used almost exclusively on the split fetcher thread, but `KafkaConsumer` is not thread-safe; as described in the ticket, this broken threading model blocks Kafka client upgrades (sporadic `CorrelationIdMismatchException`).
   
   This PR makes the consumer lazy inside the reader, so it is created on the first consumer-touching call — which by construction happens on the split fetcher thread (`fetch`, `handleSplitsChanges`, `pauseOrResumeSplits`, the offset-commit task, and `close()` in the fetcher's shutdown path all run there). Design points, as discussed on the ticket:
   
   - The constructor only computes `consumerProps` (client-id derivation unchanged) and stores them; no consumer, no metric registration.
   - A private `ensureConsumer()` runs at the entry of every consumer-touching method; consumer metric registration moves there as well.
   - `wakeUp()` remains the only cross-thread entry point (the one call `KafkaConsumer` documents as thread-safe). `wakeUp()` and creation synchronize on a small lock: if the consumer does not exist yet, the wakeup is recorded and applied right after creation, so the first blocking call still observes it — matching the semantics of a wakeup against an eagerly-created consumer. The worst case is one spurious `WakeupException` on the first consumer call, which the existing `retryOnWakeup` handling already tolerates.
   - `close()` before first use is a no-op.
   
   `KafkaPartitionSplitReaderWrapper` (DynamicKafkaSource) subclasses this reader and only wraps `fetch()` output, so the fix covers the `DynamicKafkaSource` path as well.
   
   Deferring `splitReaderFactory.get()` itself onto the fetcher thread in flink-connector-base would fix this class of issue for all connectors; with the lazy consumer this connector no longer depends on that, so I will file it as a separate ticket (as discussed on FLINK-36434, it is orthogonal).
   
   ## Brief change log
   
     - `KafkaPartitionSplitReader` no longer creates the `KafkaConsumer` (nor registers consumer metrics) in its constructor; both happen in a new `ensureConsumer()` on the first consumer-touching call, i.e. on the split fetcher thread
     - `wakeUp()` before the consumer exists records a pending wakeup that is applied immediately after creation; `close()` before first use is a no-op
     - A protected `@VisibleForTesting createConsumer(Properties)` seam allows tests to observe consumer creation
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing).
   
   This change added tests and can be verified as follows:
   
     - Added `KafkaPartitionSplitReaderConsumerThreadTest` (pure unit tests, no Kafka cluster needed): the consumer is not created at construction time and is created on the thread that first uses the reader; a `wakeUp()` arriving before creation is applied on creation exactly once; a `wakeUp()` after creation delegates directly; `close()` without prior use never creates a consumer
     - Existing `KafkaPartitionSplitReaderTest` covers the eager-path behaviors against a real cluster and passes unchanged in CI (not run locally: no Docker environment on this machine); `mvn test` for the module (compilation, checkstyle, spotless, and the new tests) passes locally
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no — `KafkaPartitionSplitReader` is `@Internal`)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no — `fetch()` gains a single volatile read on its fast path; consumer creation is one-time)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Claude Code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on code in PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#discussion_r4056325153


##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -295,7 +355,7 @@ void setConsumerClientRack(Properties consumerProps, String rackId) {
     }
 
     long getConsumerPosition(TopicPartition tp, String msg) {
-        return retryOnWakeup(() -> consumer.position(tp), msg);
+        return retryOnWakeup(() -> ensureConsumer().position(tp), msg);

Review Comment:
   Done the helpers now take the consumer directly, and `ensureConsumer()` is only used at the SplitReader entry points and in the test accessor.
   



##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -94,17 +104,49 @@ public KafkaPartitionSplitReader(
         consumerProps.putAll(props);
         consumerProps.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, createConsumerClientId(props));
         setConsumerClientRack(consumerProps, rackIdSupplier);
-        this.consumer = new KafkaConsumer<>(consumerProps);
+        this.consumerProps = consumerProps;
         this.stoppingOffsets = new HashMap<>();
         this.groupId = consumerProps.getProperty(ConsumerConfig.GROUP_ID_CONFIG);
+    }
+
+    /**
+     * Returns the consumer, creating it on the calling thread on first use.
+     *
+     * <p>The reader is constructed on the source-reader (or, when a fetcher is re-created for an
+     * offset commit, the checkpoint) thread, while the consumer is used almost exclusively on the
+     * split fetcher thread. Creating the consumer eagerly in the constructor therefore puts it on
+     * the wrong thread. All consumer-touching {@link SplitReader} methods run on the fetcher
+     * thread, so deferring creation to the first such call keeps construction and use on the same
+     * thread. The only cross-thread entry point remains {@link #wakeUp()}, which is the one call
+     * {@link KafkaConsumer} documents as thread-safe.
+     */
+    private KafkaConsumer<byte[], byte[]> ensureConsumer() {
+        // Single-writer: only the fetcher thread calls this, so the creation branch needs no
+        // guard; the field is volatile solely for the cross-thread reads in wakeUp() and close().
+        KafkaConsumer<byte[], byte[]> currentConsumer = this.consumer;
+        if (currentConsumer == null) {
+            currentConsumer = createConsumer(consumerProps);
+            maybeRegisterKafkaConsumerMetrics(
+                    consumerProps, kafkaSourceReaderMetrics, currentConsumer);
+            kafkaSourceReaderMetrics.registerNumBytesIn(currentConsumer);
+            this.consumer = currentConsumer;

Review Comment:
   Done  the field is assigned before metric registration now, with a regression test covering the failure case.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5344189390

   The architecture test caught one issue on my side: the new `@VisibleForTesting` annotation on `createConsumer` trips the freezing ArchUnit rule, since it only allows the class’s existing violations.
   
   I removed the annotation and kept the seam package-private, with the javadoc explaining why it exists. `ProductionCodeArchitectureTest` and the module tests pass locally now.
   
   @AHeise could you approve the CI run once more when you get a chance?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5443037466

   I looked at how #293 interacts with the lazy consumer, and I think the two changes compose cleanly:
   
   * #293 avoids touching the consumer during offset reconciliation. It uses `lastFetchedOffsets` / `lastKnownPositions`, which are populated from `fetch()`. Both fetching and offset commits run on the split fetcher executor, so this doesn’t introduce any new cross-thread access.
   * With lazy creation, a commit before the first fetch creates the consumer on that same fetcher thread, which is exactly what FLINK-36434 wants. Since nothing has been fetched yet, there are no known positions to reconcile, so the offsets pass through unchanged.
   * The overlap is mostly mechanical. If #293 lands first, the rebase here should just change `consumer.commitAsync(...)` to `ensureConsumer().commitAsync(...)`.
   
   Happy to rebase either way once the landing order is settled with @dalelane.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5391953419

   > > One simplification on the locking, though: I don't think `pendingWakeup`/`consumerLock` are reachable in practice, so we can drop them (and the lock) entirely.
   > 
   > cc @dalelane since #293 introduces another consumer access from the main thread (to determine the right position to commit on checkpoint) @AHeise wdyt
   
   Let's work it out with @dalelane before merging. Imho it should be fine: if there is no consumer, there are no offsets to commit. I'll also comment on that PR; I think that approach is unnecessarily complex.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5317134839

   Thanks for tracking down FLINK-36434 — lazy creation on the fetcher thread is the right fix, and confining it to `KafkaPartitionSplitReader` is the right call for this ticket (I looked at pushing the fix into `SplitFetcherManager#createSplitFetcher`/`SplitFetcher` instead, but `getSplitReader()` is public and several callers, including `KafkaSourceFetcherManager#commitOffsets`, capture the reader reference before the fetcher thread ever runs — that's a much bigger, cross-connector change and belongs in its own ticket as you noted).
   
   One simplification on the locking, though: I don't think `pendingWakeup`/`consumerLock` are reachable in practice, so we can drop them (and the lock) entirely.
   
   `SplitReader#wakeUp()`'s contract is narrow: "wake up the split reader in case the fetcher thread is blocking in `fetch()`". In the whole of flink-connector-base + flink-connector-kafka, `splitReader.wakeUp()` has exactly one call site: `FetchTask#wakeUp()`, which only fires when `SplitFetcher`'s `runningTask == fetchTask`. `fetchTask` is only ever selected once `assignedSplits` is non-empty, and `assignedSplits` is populated in exactly one place — `AddSplitsTask#run()` — which unconditionally calls `handleSplitsChanges()` (i.e. `ensureConsumer()`) immediately after, on the same fetcher thread, before that thread ever loops back to pick a next task. So by construction, `wakeUp()` can never reach `KafkaPartitionSplitReader` before `ensureConsumer()` has already run once.
   
   That means a plain `volatile` field is enough — no `AtomicReference`, no `synchronized`, no pending-wakeup bookkeeping:
   
   ```java
   /** Created lazily on first use by the fetcher thread — see #ensureConsumer(). */
   private volatile KafkaConsumer<byte[], byte[]> consumer;
   
   private KafkaConsumer<byte[], byte[]> ensureConsumer() {
       KafkaConsumer<byte[], byte[]> c = consumer;
       if (c == null) {
           c = createConsumer(consumerProps);
           maybeRegisterKafkaConsumerMetrics(consumerProps, kafkaSourceReaderMetrics, c);
           kafkaSourceReaderMetrics.registerNumBytesIn(c);
           consumer = c;
       }
       return c;
   }
   
   @Override
   public void wakeUp() {
       // wakeUp() only ever reaches a running fetch task (see SplitReader#wakeUp javadoc),
       // which the base fetcher can only schedule after handleSplitsChanges() has run at
       // least once, so consumer is always non-null here in practice.
       KafkaConsumer<byte[], byte[]> c = consumer;
       if (c != null) {
           c.wakeup();
       }
   }
   
   @Override
   public void close() throws Exception {
       KafkaConsumer<byte[], byte[]> c = consumer;
       if (c != null) {
           c.close();
       }
   }
   ```
   
   `ensureConsumer()`'s creation branch needs no guard because it's single-writer (fetcher thread only calls it); `wakeUp()`'s cross-thread read just needs the `volatile` for visibility. Worth noting this leans on an implementation invariant of `SplitFetcher`/`AddSplitsTask` rather than a documented contract beyond `wakeUp()`'s own javadoc scoping — stable code, but that's why I'd keep the comment explaining it.
   
   This also removes the exact thing @Efrat19 flagged as feeling weird (the `@GuardedBy` split between `ensureConsumer`/`createConsumer`).
   
   The `KafkaPartitionSplitReaderLazyConsumerCreationTest` case for "wakeUp before creation is applied on creation" would need to flip to asserting that a pre-creation `wakeUp()` is safely dropped instead of deferred.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Efrat19 (via GitHub)" <gi...@apache.org>.
Efrat19 commented on code in PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#discussion_r3751430369


##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -66,13 +66,28 @@ public class KafkaPartitionSplitReader
     private static final Logger LOG = LoggerFactory.getLogger(KafkaPartitionSplitReader.class);
     private static final long POLL_TIMEOUT = 10000L;
 
-    private final KafkaConsumer<byte[], byte[]> consumer;
+    private final Properties consumerProps;
     private final Map<TopicPartition, Long> stoppingOffsets;
     private final String groupId;
     private final int subtaskId;
 
     private final KafkaSourceReaderMetrics kafkaSourceReaderMetrics;
 
+    /** Guards lazy creation of {@link #consumer} and the {@link #pendingWakeup} flag. */
+    private final Object consumerLock = new Object();
+
+    /**
+     * The Kafka consumer. Created lazily on the first thread that actually uses it — the split
+     * fetcher thread — rather than on the thread constructing this reader (the source-reader or
+     * checkpoint thread). {@link KafkaConsumer} is not thread-safe, so it must live on the thread
+     * that uses it (FLINK-36434). Access outside {@link #ensureConsumer()} / {@link #wakeUp()} is
+     * only safe after a call to {@link #ensureConsumer()} on the same thread.
+     */
+    @Nullable private volatile KafkaConsumer<byte[], byte[]> consumer;

Review Comment:
   `@GuardedBy("lock")`



##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -94,17 +109,56 @@ public KafkaPartitionSplitReader(
         consumerProps.putAll(props);
         consumerProps.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, createConsumerClientId(props));
         setConsumerClientRack(consumerProps, rackIdSupplier);
-        this.consumer = new KafkaConsumer<>(consumerProps);
+        this.consumerProps = consumerProps;
         this.stoppingOffsets = new HashMap<>();
         this.groupId = consumerProps.getProperty(ConsumerConfig.GROUP_ID_CONFIG);
+    }
+
+    /**
+     * Returns the consumer, creating it on the calling thread on first use.
+     *
+     * <p>The reader is constructed on the source-reader (or, when a fetcher is re-created for an
+     * offset commit, the checkpoint) thread, while the consumer is used almost exclusively on the
+     * split fetcher thread. Creating the consumer eagerly in the constructor therefore puts it on
+     * the wrong thread. All consumer-touching {@link SplitReader} methods run on the fetcher
+     * thread, so deferring creation to the first such call keeps construction and use on the same
+     * thread. The only cross-thread entry point remains {@link #wakeUp()}, which is the one call
+     * {@link KafkaConsumer} documents as thread-safe.
+     */
+    private KafkaConsumer<byte[], byte[]> ensureConsumer() {
+        KafkaConsumer<byte[], byte[]> currentConsumer = this.consumer;
+        if (currentConsumer != null) {
+            return currentConsumer;
+        }
+        synchronized (consumerLock) {
+            currentConsumer = this.consumer;
+            if (currentConsumer == null) {
+                currentConsumer = createConsumer(consumerProps);
+                maybeRegisterKafkaConsumerMetrics(
+                        consumerProps, kafkaSourceReaderMetrics, currentConsumer);
+                kafkaSourceReaderMetrics.registerNumBytesIn(currentConsumer);
+                if (pendingWakeup) {
+                    // A wakeUp() arrived before the consumer existed. Apply it now so that the
+                    // first blocking call still observes it, matching the behavior of a wakeup
+                    // against an eagerly-created consumer.
+                    currentConsumer.wakeup();
+                    pendingWakeup = false;
+                }
+                this.consumer = currentConsumer;
+            }
+            return currentConsumer;
+        }
+    }
 
-        // Metric registration
-        maybeRegisterKafkaConsumerMetrics(props, kafkaSourceReaderMetrics, consumer);
-        this.kafkaSourceReaderMetrics.registerNumBytesIn(consumer);
+    /** Creates the {@link KafkaConsumer}. Overridable for tests to observe the creation. */
+    @VisibleForTesting
+    protected KafkaConsumer<byte[], byte[]> createConsumer(Properties consumerProps) {

Review Comment:
   It feels a bit weird to have the `ensureConsumer` wrapper private and `createConsumer` inner call `protected`, wdyt?
   If this is just for the tests you could probably call consumer.poll with a long timeout and expect it to throw a wakeUpException
   Or at least make package-private



##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -66,13 +66,28 @@ public class KafkaPartitionSplitReader
     private static final Logger LOG = LoggerFactory.getLogger(KafkaPartitionSplitReader.class);
     private static final long POLL_TIMEOUT = 10000L;
 
-    private final KafkaConsumer<byte[], byte[]> consumer;
+    private final Properties consumerProps;
     private final Map<TopicPartition, Long> stoppingOffsets;
     private final String groupId;
     private final int subtaskId;
 
     private final KafkaSourceReaderMetrics kafkaSourceReaderMetrics;
 
+    /** Guards lazy creation of {@link #consumer} and the {@link #pendingWakeup} flag. */
+    private final Object consumerLock = new Object();
+
+    /**
+     * The Kafka consumer. Created lazily on the first thread that actually uses it — the split
+     * fetcher thread — rather than on the thread constructing this reader (the source-reader or
+     * checkpoint thread). {@link KafkaConsumer} is not thread-safe, so it must live on the thread
+     * that uses it (FLINK-36434). Access outside {@link #ensureConsumer()} / {@link #wakeUp()} is
+     * only safe after a call to {@link #ensureConsumer()} on the same thread.
+     */
+    @Nullable private volatile KafkaConsumer<byte[], byte[]> consumer;
+
+    /** Set when {@link #wakeUp()} is called before the consumer exists; guarded by the lock. */
+    private boolean pendingWakeup;

Review Comment:
   `@GuardedBy("lock")`



##########
flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReaderConsumerThreadTest.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.source.reader;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.kafka.source.KafkaSourceOptions;
+import org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics;
+import org.apache.flink.connector.testutils.source.reader.TestingReaderContext;
+import org.apache.flink.metrics.groups.UnregisteredMetricsGroup;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.jupiter.api.Test;
+
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
+
+/**
+ * Unit tests for FLINK-36434: the {@link KafkaConsumer} inside {@link KafkaPartitionSplitReader}
+ * must be created lazily on the thread that uses it (the split fetcher thread), not on the thread
+ * that constructs the reader (the source-reader or checkpoint thread), because the consumer is not
+ * thread-safe.
+ *
+ * <p>These tests need no running Kafka cluster: constructing a {@link KafkaConsumer} performs no
+ * network I/O, and {@link KafkaPartitionSplitReader#fetch()} tolerates the {@code
+ * IllegalStateException}/{@code WakeupException} that polling without an assignment raises.
+ */
+class KafkaPartitionSplitReaderConsumerThreadTest {

Review Comment:
   Nit:
   ```suggestion
   class KafkaPartitionSplitReaderLazyConsumerCreationTest {
   ```
   or even move to `KafkaPartitionSplitReaderTest`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5323210689

   > Thanks for tracking down [FLINK-36434](https://issues.apache.org/jira/browse/FLINK-36434) — lazy creation on the fetcher thread is the right fix, and confining it to `KafkaPartitionSplitReader` is the right call for this ticket (I looked at pushing the fix into `SplitFetcherManager#createSplitFetcher`/`SplitFetcher` instead, but `getSplitReader()` is public and several callers, including `KafkaSourceFetcherManager#commitOffsets`, capture the reader reference before the fetcher thread ever runs — that's a much bigger, cross-connector change and belongs in its own ticket as you noted).
   > 
   > One simplification on the locking, though: I don't think `pendingWakeup`/`consumerLock` are reachable in practice, so we can drop them (and the lock) entirely.
   > 
   > `SplitReader#wakeUp()`'s contract is narrow: "wake up the split reader in case the fetcher thread is blocking in `fetch()`". In the whole of [flink-connector-base](https://issues.apache.org/jira/browse/FLINK-connector-base) + [flink-connector-kafka](https://issues.apache.org/jira/browse/FLINK-connector-kafka), `splitReader.wakeUp()` has exactly one call site: `FetchTask#wakeUp()`, which only fires when `SplitFetcher`'s `runningTask == fetchTask`. `fetchTask` is only ever selected once `assignedSplits` is non-empty, and `assignedSplits` is populated in exactly one place — `AddSplitsTask#run()` — which unconditionally calls `handleSplitsChanges()` (i.e. `ensureConsumer()`) immediately after, on the same fetcher thread, before that thread ever loops back to pick a next task. So by construction, `wakeUp()` can never reach `KafkaPartitionSplitReader` before `ensureConsumer()` has already run once.
   > 
   > That means a plain `volatile` field is enough — no `AtomicReference`, no `synchronized`, no pending-wakeup bookkeeping:
   > 
   > ```java
   > /** Created lazily on first use by the fetcher thread — see #ensureConsumer(). */
   > private volatile KafkaConsumer<byte[], byte[]> consumer;
   > 
   > private KafkaConsumer<byte[], byte[]> ensureConsumer() {
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c == null) {
   >         c = createConsumer(consumerProps);
   >         maybeRegisterKafkaConsumerMetrics(consumerProps, kafkaSourceReaderMetrics, c);
   >         kafkaSourceReaderMetrics.registerNumBytesIn(c);
   >         consumer = c;
   >     }
   >     return c;
   > }
   > 
   > @Override
   > public void wakeUp() {
   >     // wakeUp() only ever reaches a running fetch task (see SplitReader#wakeUp javadoc),
   >     // which the base fetcher can only schedule after handleSplitsChanges() has run at
   >     // least once, so consumer is always non-null here in practice.
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c != null) {
   >         c.wakeup();
   >     }
   > }
   > 
   > @Override
   > public void close() throws Exception {
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c != null) {
   >         c.close();
   >     }
   > }
   > ```
   > 
   > `ensureConsumer()`'s creation branch needs no guard because it's single-writer (fetcher thread only calls it); `wakeUp()`'s cross-thread read just needs the `volatile` for visibility. Worth noting this leans on an implementation invariant of `SplitFetcher`/`AddSplitsTask` rather than a documented contract beyond `wakeUp()`'s own javadoc scoping — stable code, but that's why I'd keep the comment explaining it.
   > 
   > This also removes the exact thing @Efrat19 flagged as feeling weird (the `@GuardedBy` split between `ensureConsumer`/`createConsumer`).
   > 
   > The `KafkaPartitionSplitReaderLazyConsumerCreationTest` case for "wakeUp before creation is applied on creation" would need to flip to asserting that a pre-creation `wakeUp()` is safely dropped instead of deferred.
   
   @AHeise Thanks  agreed on both points, and I’ve updated it as suggested.
   
   `consumerLock` and `pendingWakeup` are gone. `consumer` is now a plain `volatile`, only created by `ensureConsumer()` on the fetcher thread, while `wakeUp()` and `close()` just do a null-checked read. I kept the comment in `wakeUp()` explaining the `SplitFetcher` / `AddSplitsTask` invariant, including that we’re relying on current implementation behavior rather than a documented contract.
   
   I updated the test as well: `testWakeUpBeforeConsumerCreationIsAppliedOnCreation` is now `testWakeUpBeforeConsumerCreationIsSafelyDropped`. A `wakeUp()` before consumer creation is simply ignored — it doesn’t throw, create the consumer, or get replayed later.
   
   Also refreshed the PR description to remove the old pending-wakeup design.
   
   Could you approve the CI run when you get a chance?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#discussion_r4054244363


##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -94,17 +104,49 @@ public KafkaPartitionSplitReader(
         consumerProps.putAll(props);
         consumerProps.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, createConsumerClientId(props));
         setConsumerClientRack(consumerProps, rackIdSupplier);
-        this.consumer = new KafkaConsumer<>(consumerProps);
+        this.consumerProps = consumerProps;
         this.stoppingOffsets = new HashMap<>();
         this.groupId = consumerProps.getProperty(ConsumerConfig.GROUP_ID_CONFIG);
+    }
+
+    /**
+     * Returns the consumer, creating it on the calling thread on first use.
+     *
+     * <p>The reader is constructed on the source-reader (or, when a fetcher is re-created for an
+     * offset commit, the checkpoint) thread, while the consumer is used almost exclusively on the
+     * split fetcher thread. Creating the consumer eagerly in the constructor therefore puts it on
+     * the wrong thread. All consumer-touching {@link SplitReader} methods run on the fetcher
+     * thread, so deferring creation to the first such call keeps construction and use on the same
+     * thread. The only cross-thread entry point remains {@link #wakeUp()}, which is the one call
+     * {@link KafkaConsumer} documents as thread-safe.
+     */
+    private KafkaConsumer<byte[], byte[]> ensureConsumer() {
+        // Single-writer: only the fetcher thread calls this, so the creation branch needs no
+        // guard; the field is volatile solely for the cross-thread reads in wakeUp() and close().
+        KafkaConsumer<byte[], byte[]> currentConsumer = this.consumer;
+        if (currentConsumer == null) {
+            currentConsumer = createConsumer(consumerProps);
+            maybeRegisterKafkaConsumerMetrics(
+                    consumerProps, kafkaSourceReaderMetrics, currentConsumer);
+            kafkaSourceReaderMetrics.registerNumBytesIn(currentConsumer);
+            this.consumer = currentConsumer;

Review Comment:
   Assigning the field straight after `createConsumer` would be a bit safer I think: if metric registration throws, `close()` sees null and never closes the consumer.



##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -295,7 +355,7 @@ void setConsumerClientRack(Properties consumerProps, String rackId) {
     }
 
     long getConsumerPosition(TopicPartition tp, String msg) {
-        return retryOnWakeup(() -> consumer.position(tp), msg);
+        return retryOnWakeup(() -> ensureConsumer().position(tp), msg);

Review Comment:
   These helpers only run after `fetch()` or `handleSplitsChanges()` already ensured the consumer, and `fetch()` reaches this one per assigned partition per poll rather than once. Passing the consumer down keeps creation to the four entry points.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5750368127

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5256253870

   @Efrat19 Thanks for the review! All three addressed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5185871633

   Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Efrat19 (via GitHub)" <gi...@apache.org>.
Efrat19 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5329176562

   > Thanks for tracking down [FLINK-36434](https://issues.apache.org/jira/browse/FLINK-36434) — lazy creation on the fetcher thread is the right fix, and confining it to `KafkaPartitionSplitReader` is the right call for this ticket (I looked at pushing the fix into `SplitFetcherManager#createSplitFetcher`/`SplitFetcher` instead, but `getSplitReader()` is public and several callers, including `KafkaSourceFetcherManager#commitOffsets`, capture the reader reference before the fetcher thread ever runs — that's a much bigger, cross-connector change and belongs in its own ticket as you noted).
   > 
   > One simplification on the locking, though: I don't think `pendingWakeup`/`consumerLock` are reachable in practice, so we can drop them (and the lock) entirely.
   > 
   > `SplitReader#wakeUp()`'s contract is narrow: "wake up the split reader in case the fetcher thread is blocking in `fetch()`". In the whole of [flink-connector-base](https://issues.apache.org/jira/browse/FLINK-connector-base) + [flink-connector-kafka](https://issues.apache.org/jira/browse/FLINK-connector-kafka), `splitReader.wakeUp()` has exactly one call site: `FetchTask#wakeUp()`, which only fires when `SplitFetcher`'s `runningTask == fetchTask`. `fetchTask` is only ever selected once `assignedSplits` is non-empty, and `assignedSplits` is populated in exactly one place — `AddSplitsTask#run()` — which unconditionally calls `handleSplitsChanges()` (i.e. `ensureConsumer()`) immediately after, on the same fetcher thread, before that thread ever loops back to pick a next task. So by construction, `wakeUp()` can never reach `KafkaPartitionSplitReader` before `ensureConsumer()` has already run once.
   > 
   > That means a plain `volatile` field is enough — no `AtomicReference`, no `synchronized`, no pending-wakeup bookkeeping:
   > 
   > ```java
   > /** Created lazily on first use by the fetcher thread — see #ensureConsumer(). */
   > private volatile KafkaConsumer<byte[], byte[]> consumer;
   > 
   > private KafkaConsumer<byte[], byte[]> ensureConsumer() {
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c == null) {
   >         c = createConsumer(consumerProps);
   >         maybeRegisterKafkaConsumerMetrics(consumerProps, kafkaSourceReaderMetrics, c);
   >         kafkaSourceReaderMetrics.registerNumBytesIn(c);
   >         consumer = c;
   >     }
   >     return c;
   > }
   > 
   > @Override
   > public void wakeUp() {
   >     // wakeUp() only ever reaches a running fetch task (see SplitReader#wakeUp javadoc),
   >     // which the base fetcher can only schedule after handleSplitsChanges() has run at
   >     // least once, so consumer is always non-null here in practice.
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c != null) {
   >         c.wakeup();
   >     }
   > }
   > 
   > @Override
   > public void close() throws Exception {
   >     KafkaConsumer<byte[], byte[]> c = consumer;
   >     if (c != null) {
   >         c.close();
   >     }
   > }
   > ```
   > 
   > `ensureConsumer()`'s creation branch needs no guard because it's single-writer (fetcher thread only calls it); `wakeUp()`'s cross-thread read just needs the `volatile` for visibility. Worth noting this leans on an implementation invariant of `SplitFetcher`/`AddSplitsTask` rather than a documented contract beyond `wakeUp()`'s own javadoc scoping — stable code, but that's why I'd keep the comment explaining it.
   > 
   > This also removes the exact thing @Efrat19 flagged as feeling weird (the `@GuardedBy` split between `ensureConsumer`/`createConsumer`).
   > 
   > The `KafkaPartitionSplitReaderLazyConsumerCreationTest` case for "wakeUp before creation is applied on creation" would need to flip to asserting that a pre-creation `wakeUp()` is safely dropped instead of deferred.
   
   cc @dalelane since https://github.com/apache/flink-connector-kafka/pull/293 introduces another consumer access from the main thread


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5748188441

   Thanks for the thorough review and for running the 50-run probe. I've addressed the feedback: rebased onto current `main`, fixed the consumer ownership around metric registration with a regression test, and cleaned up the `ensureConsumer()` usage so it's only called at the SplitReader entry points. I also squashed the changes and linked FLINK-40396 for the connector-base follow-up. The relevant unit tests pass locally; the Testcontainers suites will need CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36434][Connectors/Kafka] Create the KafkaConsumer lazily on the split fetcher thread [flink-connector-kafka]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
Zhuoxi2000 commented on PR #290:
URL: https://github.com/apache/flink-connector-kafka/pull/290#issuecomment-5256266169

   On CI: the main workflow is still awaiting first-time-contributor approval  @AHeise  could you approve the CI run when you get a chance?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org