[jira] [Created] (FLINK-40396) Create SplitReaders on the split fetcher thread instead of the caller thread

["Edson Wang (Jira)" <ji...@apache.org>]

Edson Wang created FLINK-40396:
----------------------------------

             Summary: Create SplitReaders on the split fetcher thread instead of the caller thread
                 Key: FLINK-40396
                 URL: https://issues.apache.org/jira/browse/FLINK-40396
             Project: Flink
          Issue Type: Improvement
            Reporter: Edson Wang


Follow-up promised in https://github.com/apache/flink-connector-kafka/pull/290 (FLINK-36434).

SplitFetcherManager currently invokes the split-reader supplier on the thread that creates
the SplitFetcher (the source reader thread), not on the fetcher thread that will use the
reader. Clients constructed inside a SplitReader therefore capture the wrong thread at
construction time. For Kafka this meant the KafkaConsumer was created on a thread it never
runs on afterwards; FLINK-36434 fixed it locally by making the consumer lazy inside
KafkaPartitionSplitReader.

Deferring the supplier call itself onto the fetcher thread in flink-connector-base would fix
this class of issue for all connectors, so no connector needs its own laziness workaround.

Sketch: move the supplier invocation from SplitFetcher construction into the fetcher
thread's startup (first run), while keeping construction failures propagating through the
existing fetcher error-handling path (today a failing supplier throws synchronously on the
caller thread — connectors relying on that would see the failure asynchronously instead,
which needs to be called out in the compatibility notes).

I'm happy to work on this once assigned.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)