You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2026/07/06 15:21:07 UTC

[PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

1996fanrui opened a new pull request, #28661:
URL: https://github.com/apache/flink/pull/28661

   This PR depends on https://github.com/apache/flink/pull/28651, https://github.com/apache/flink/pull/28652, https://github.com/apache/flink/pull/28659 and https://github.com/apache/flink/pull/28660
   
   Please only review the last 5 commits (prefixed [FLINK-39524]).
   
   ## What is the purpose of the change
   
   [FLINK-39524] Fetched channel state: spill files, filtered write, reader.
   
   It's part 5 of FLINK-38544 (checkpointing during recovery of unaligned checkpoint state). This PR introduces the on-disk subsystem for recovered channel state: an append-only segmented spill-file format written by the recovery read path (optionally through the rescale record filter), a ref-counted file container with snapshot/resume semantics, and a forward-only reader. Runtime behavior is unchanged: the handler factory does not select the new handlers, so no production path produces or consumes spill files.
   
   ## Brief change log
   
   - [FLINK-39524][checkpoint] Add FetchedChannelState spill-file container
   - [FLINK-39524][checkpoint] Spill-writing handlers: segmented on-disk format
   - [FLINK-39524][checkpoint] Rewrite ChannelStateFilteringHandler to emit into a spill segment
   - [FLINK-39524][checkpoint] Forward-only spill reader with snapshot/resume
   - [FLINK-39524][checkpoint] SequentialChannelStateReader#readInputData returns Optional<FetchedChannelState>
   
   ## Verifying this change
   
   - New unit tests: FetchedChannelStateTest, FetchedChannelStateRefCountTest, FetchedChannelStateReaderTest, AbstractSpillingHandlerTest, ChannelStateFilteringHandlerTest, GateFilterHandlerTest, GateFilterHandlerBufferOwnershipTest, RecoveredChannelStateHandlerFilterRoutingTest, SequentialChannelStateReaderImplTest
   - RecoveredStateFilteringLargeRecordITCase stays green (the in-memory filtering path is untouched)
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3728240568


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   This value is only used as a presence flag today: if it's present, StreamTask requests partitions with needsRecovery=true and builds a drainer; if it's empty, it does neither. It never carries data — the in-memory handler pushes the recovered buffers straight into the channels' queues.
   
   The `Spilling*` handlers are introduced here but not selected by the factory yet, so getProducedChannelState() is always null at this point; it starts carrying real data in the next PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3736557082


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   My understanding is the opposite of the expectation here: this would add code rather than remove it. I do agree the interfaces themselves would read clearer — but that clarity comes with more types and more code, not less.
   
   Almost everything the two paths do is shared: file rolling, position tracking, header parsing, the bounded body view, fail-loud EOF, close and the lifecycle grant. That is essentially the whole class.
   
   What is unique to each side is a couple of members: `commit()`/`snapshot()` on the drain side, and the mid-body resume entry on the checkpoint side (the drain reader starts at offset 0 and never uses it). So splitting the implementation gives a base class carrying nearly all of the code plus two very thin subclasses.
   
   The interface split isn't free on its own either, because `commit()` lives on `SpillSegment`. Hiding it from the checkpoint side needs either a generic reader, or a second segment type, or moving `commit()` up onto the drain-side reader; each of those touches every call site while the implementation stays a single class.
   
   So I'd rather keep one interface and one implementation and state the contract in the javadoc (only the main reader commits) — especially since the resume point moving into the snapshot already removed most of the contractual comments the split would have enforced. Let me know if you see it differently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3605821064


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/ChannelState.java:
##########
@@ -98,4 +111,26 @@ public ChannelState emptyState() {
         sequenceNumberInAnnouncedChannels.clear();
         return this;
     }
+
+    /**
+     * Dispatches checkpoint start: inserts recovery-checkpoint barriers into in-recovery channels
+     * through the trigger, then notifies every input. (FLINK-38544 transitional: the spilling
+     * backend adds a third step handing the trigger's snapshot reader to the channel-state writer.)
+     */
+    public void onCheckpointStartedForAllInputs(CheckpointBarrier barrier)
+            throws CheckpointException, IOException {
+        long cpId = barrier.getId();
+        try {
+            recoveryCheckpointTrigger.snapshotAndInsertBarriers(cpId);
+
+            for (CheckpointableInput input : inputs) {
+                input.checkpointStarted(barrier);
+            }
+        } catch (Throwable t) {
+            if (t instanceof CheckpointException) {
+                throw (CheckpointException) t;
+            }
+            ExceptionUtils.rethrowIOException(t);
+        }
+    }

Review Comment:
   Don't we need to call `channelStateWriter.addInputDataFromSpill` in this method?
   
   Like here:
   https://github.com/apache/flink/pull/28554/changes#diff-8104c9791f592faf61f8613d92c48aefcb89dae2e6d89ad02dff96bc80b94c18R136



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3745637786


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   thanks, I created https://issues.apache.org/jira/browse/FLINK-40359 to follow it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #28661:
URL: https://github.com/apache/flink/pull/28661#issuecomment-5209491522

   > One more thing found while reading, flagged here instead of inline since it isn't part of this diff (pre-existing from #28651, in `FilteringHandler.recoverWithFiltering`'s catch block): `onRecoveredStateBuffer` only throws after the buffer's ownership has already transferred into the channel's queue, so the catch loop's `for (int j = i; j < filteredBuffers.size(); j++)` recycles the buffer at index `i` while it's already live in that queue. Might be worth a fix there (starting at `j = i + 1`), separately from this PR.
   
   Hey @rkhachatryan ,  good catch, created a separate pr to fix it https://github.com/apache/flink/pull/28936
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3724924102


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,

Review Comment:
   I would prefer the code to be self-documenting (so that there's no need to read the javadoc).
   How about `advanceAndGetNextSegment`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link

Review Comment:
   TBH, drain reader is also not very clear to me :) 
   I'm struggling to come up with a better name though. Maybe main reader?
   
   (this is very NIT anyways)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds

Review Comment:
   Sorry but it doesn't tell me much.
   Could you clarify in the javadoc - who calls commit and when?
   
   > before anything is committed it's simply the reader's start position; now defined in the interface javadoc.
   
   👍 



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   > added the TODO about returning getProducedChannelState() once the Spilling* handlers are selected.
   
   Can't we return it already? We already return - just the wrong value:
   ```
               return filterContext.isCheckpointingDuringRecoveryEnabled() && readAny
                       ? Optional.of(new FetchedChannelState(java.util.Collections.emptyList()))
   ```



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   I think this interface split already makes sense - it makes it clear how each call site uses the interface.
   I don't understand why the implementation is the same? Can't we split it as well?
   
   > What does bring real simplification is your suggestion in #28661 (comment): once the snapshot carries the resume segment, most of the contractual comments this split would enforce disappear anyway.
   
   Probably we should discuss it offline.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Yes, this comment is about `skipOnStream`, sorry for misplacing.
   I think `skipOnStream` can be replaced by `IOUtils#skipFully`.
   
   `readFully` as well - it was the [next](https://github.com/apache/flink/pull/28661#discussion_r3693955986) comment



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {

Review Comment:
   Do we still need to catch the IOException after switching to IOUtils.readFully?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #28661:
URL: https://github.com/apache/flink/pull/28661


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3728622738


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,

Review Comment:
   Renamed to `advanceAndGetNextSegment`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;

Review Comment:
   Removed the field — a private `files()` accessor keeps the call sites short.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds

Review Comment:
   Clarified in the javadoc — the drainer's drain loop calls it once per delivered buffer, under the same lock as `snapshot()`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);

Review Comment:
   Added non-negativity checks; the upper bounds can only be checked by the consumer, which owns the gate structures.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");

Review Comment:
   No — each handler writes into its own `flink-channel-spill-<UUID>` directory, so the `spill-segment-N.bin` numbering is per-directory and can't collide across tasks.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;

Review Comment:
   Not exceptional — the header is written before filtering runs, so a channel whose records are all filtered out ends up empty. Dropping it avoids a header-only segment; added a comment saying so.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link

Review Comment:
   Renamed to "main reader".



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   Today nothing is produced here — the factory only builds the in-memory handlers — so there is nothing to close yet; who releases the grant and deletes the files is part of the handoff that comes with the drainer in the next PR.
   
   If you'd rather have it airtight already, I can add a `checkState(stateHandler.getProducedChannelState() == null)` after the handler is closed, so flipping the factory without wiring the drain path fails loudly instead of leaking files — happy to do that.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {

Review Comment:
   No — the catch only wrapped the exception to add the file/offset, removed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3739202386


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   Is ^^^ this a correct change?
   It wasn't clear to me what's the scope/lifecycle of `BoundedSegmentStream` and it's relation to `current` stream; which clearly signals bad design.
   
   If the above diff is a correct change (which I believe it is) can you please apply it?
   Otherwise, let's have a follow up to revisit this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3693955946


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelState.java:
##########
@@ -22,42 +22,124 @@
 
 import java.io.Closeable;
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
+import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
- * Sealed container for fetched recovered channel-state data.
+ * Sealed container for recovered channel-state data written to spill files.
+ *
+ * <p>Holds a list of file paths (in write order). Segment boundaries are self-described in disk
+ * segment headers ([4B gateIdx][4B channelIdx][4B bufferLength]), so no in-memory segment locator
+ * table is maintained. The reader scans files sequentially, reading each 12-byte header to obtain
+ * the channel info and body length.
+ *
+ * <p>The file list grows as the writer rotates to new files (one rotation per 64 MB soft limit),

Review Comment:
   This seems to contradict the next paragraph ("The file list grows as the writer
   rotates to new files..."). By the time this container exists, is the list actually
   fixed, with the growth happening earlier in the writer, before this object is
   constructed? Might be worth rewording the paragraph to make that explicit.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds

Review Comment:
   What exactly is the "committed position"? And what does it resolve to if nothing
   has been read yet?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {

Review Comment:
   This duplicates the `openFileAndSeek()` + bounds-check block above — can we fold
   both branches into one loop/helper?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();

Review Comment:
   This interface is `@Internal public`, but `snapshot()` returns
   `FetchedChannelStateSnapshot`, which is package-private with only package-private
   methods — an external caller in a different package can obtain the object but can't
   do anything useful with it. Is that intentional?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");
+        currentStream =
+                new OffsetAwareOutputStream(
+                        new BufferedOutputStream(new FileOutputStream(filePath.toFile())), 0L);

Review Comment:
   Can we open the file for writing in WRITE_IF_NOT_EXISTS mode to fail loudly on
   collisions?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;

Review Comment:
   Is a zero-body segment an exceptional situation? If not, should we still write the
   zero length rather than dropping the segment silently?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");

Review Comment:
   Can't we get a file name collision here if other tasks are using the same tmp dir?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,

Review Comment:
   This whole javadoc comment is difficult to read. Could we rename `nextSegment` to
   `advanceAndGet` and remove this comment?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link

Review Comment:
   Is there still a distinct "root reader" concept, or is that just informal
   terminology now that there's a single reader implementation for both the drain and
   snapshot paths? (Relates to the interface-split suggestion further down.)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   Could we enforce this contract by splitting this interface into separate drain-side
   and checkpoint-side interfaces, so `commit()` isn't even callable from the
   snapshot/checkpoint path? I believe this would simplify a lot of the code in this
   PR.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;

Review Comment:
   This can't be a local — `nextSegment()`'s entry check needs the previous call's
   state. But do we need both this field and `Segment.body` holding the same object?
   Could we store the last `Segment` instead and drop one of the two?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;

Review Comment:
   Why do we need this field? Could `openFileAndSeek()` just return the file size
   instead of storing it?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;

Review Comment:
   Why do we need this field if we already have `snapshot`? (`snapshot.channelState()`
   is accessible.)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;

Review Comment:
   NIT: rename to `currentFileStream` for consistency with `current` /
   `currentFileSize` / `currentBody`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;

Review Comment:
   Same question — why cache this separately when `snapshot.channelState().files()`
   gives us the same list?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateDrainer.java:
##########
@@ -71,10 +71,14 @@ public void drain() throws IOException, InterruptedException {
      * must persist is already inside the channels' queues, so the snapshot is inherently empty.
      */
     @Override
-    public void snapshotAndInsertBarriers(long checkpointId) throws IOException {
+    public FetchedChannelStateReader snapshotAndInsertBarriers(long checkpointId)

Review Comment:
   Why is this method called `snapshot...` but returns a `reader`? Shouldn't it return
   a snapshot instead?
   
   (Separately: the purpose and naming of `drain()` are unclear to me.)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;

Review Comment:
   NIT: could `rollToNextFile()` return a boolean instead, so the caller doesn't need
   this separate bounds check?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {

Review Comment:
   Is it intentional that this stream isn't buffered?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");

Review Comment:
   Why can't we have multiple readers per snapshot? Is it because the snapshot is mutable? That shouldn't be the case - snapshot is immutable by definition IMO.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**

Review Comment:
   Should this comment be removed?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   Could this comment be trimmed down to just the "FLINK-38544 transitional"
   paragraph below? The paragraph above feels redundant with what the code already
   expresses.
   
   Separately: Today this is harmless since no production path spills real files yet, but once
   the handler factory starts selecting the `Spilling*` handlers, this line will
   silently drop the real `FetchedChannelState` instead of returning it
   (`readInputData` always returns `Optional.empty()` or a fresh empty container,
   never `stateHandler.getProducedChannelState()`). Worth a TODO here pointing at
   whatever ticket wires this up?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);

Review Comment:
   Should we validate `gateIdx`/`channelIdx` here too, not just `bufferLength`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");
+        currentStream =
+                new OffsetAwareOutputStream(
+                        new BufferedOutputStream(new FileOutputStream(filePath.toFile())), 0L);
+        files.add(filePath);
+    }
+
+    @Override
+    @Nullable
+    FetchedChannelState getProducedChannelState() {

Review Comment:
   Confirmed unused outside tests — is wiring this into `readInputData` tracked as a
   follow-up (JIRA/TODO), or should this PR add that already? The PR description says
   no production path produces/consumes spill files yet, so I assume this is
   intentional — worth a TODO here referencing the follow-up ticket?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {

Review Comment:
   Could we delete this field and instead carry the parsed header in
   `FetchedChannelStateSnapshot` (nullable for the root reader), so we don't need to
   seek back to the header? I think this would simplify the code a lot.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;

Review Comment:
   A channel that never switches never calls `sealCurrentSegment()` — so its segment
   accumulates unboundedly in heap (`segmentSerializer`), not just missing out on disk
   rotation. Should we force-seal based on accumulated size here?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);

Review Comment:
   Could `skipOnStream` look up the file name itself (`files.get(current.fileIndex)`)
   instead of taking `path` as a parameter?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Can we use `org.apache.flink.util.IOUtils#skipFully` here instead of hand-rolling
   this loop?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   I understand this class is needed for tracking bytes read, but I'm not sure it's
   still necessary if the other suggestions here are applied. If we keep it: (1) why
   not delegate/wrap and ask `FetchedChannelStateReaderImpl` for the tracked value
   instead? (2) could we make it static to make its purpose clearer? (3) if (1) is the
   right read, the class javadoc is now inaccurate.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");
+        readerOpened = true;
+        return new FetchedChannelStateReaderImpl(this);
+    }
+
+    /**
+     * Releases the lifecycle grant held by this snapshot. Called by the reader on close; must not
+     * be called directly by any other party.

Review Comment:
   If `reader()` is never called on a snapshot, is the lifecycle grant acquired in the
   constructor ever released? Looks like it would leak.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {

Review Comment:
   Can we use `org.apache.flink.util.IOUtils#readFully` here instead of hand-rolling
   this loop?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3731948725


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateDrainer.java:
##########
@@ -71,10 +71,14 @@ public void drain() throws IOException, InterruptedException {
      * must persist is already inside the channels' queues, so the snapshot is inherently empty.
      */
     @Override
-    public void snapshotAndInsertBarriers(long checkpointId) throws IOException {
+    public FetchedChannelStateReader snapshotAndInsertBarriers(long checkpointId)

Review Comment:
   Returns the snapshot now, so the name fits — the caller opens the reader. `drain()` is transitional in-memory; the real drainer lands in the next PR.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;

Review Comment:
   Kept — `openFileAndSeek()` returns early when the stream is already open, so the size belongs with the stream.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");
+        readerOpened = true;
+        return new FetchedChannelStateReaderImpl(this);
+    }
+
+    /**
+     * Releases the lifecycle grant held by this snapshot. Called by the reader on close; must not
+     * be called directly by any other party.

Review Comment:
   Fixed — the snapshot is closable now, releasing the grant when no reader was opened; the integrated path always opens one immediately.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");

Review Comment:
   We don't need multiple readers — a snapshot is read once — so rather than refcounting per reader, opening one transfers the snapshot's single grant; the reader's close returns it.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {

Review Comment:
   Done via 53a8d71f6397b26c815d6f30896157000004d57d, and re-defined FetchedChannelStateSnapshot.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {

Review Comment:
   Kept explicit: the roll can only happen once, and the two branches say so. A loop would read as if it could repeat arbitrarily — correct me if I'm wrong?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;

Review Comment:
   Left as is — `Position` doesn't know the file count, and injecting it just to return a boolean would blur that boundary.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   Done for (1) and (3) — the view no longer counts anything, it takes the bound from the reader's position, and the javadoc is rewritten. Kept non-static since it needs the reader's `readBody` and position.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Dropped the skipping instead: the reader now seeks the channel directly. `skipFully` would spin forever on truncation — `skip()` returns 0 at EOF, it only checks negatives.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #28661:
URL: https://github.com/apache/flink/pull/28661#issuecomment-5245777987

   @rkhachatryan Thanks for the confirmation, merging


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3724881811


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Yes, this comment is about `skipOnStream`, sorry for misplacing.
   I think `skipOnStream` can be replaced by `IOUtils#skipFully`.
   
   `readFully` as well - it was the [next](https://github.com/apache/flink/pull/28661#discussion_r3693955986) comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3719813640


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link

Review Comment:
   It was just informal terminology — replaced "root reader" everywhere with "drain reader" and defined it in the interface javadoc.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;

Review Comment:
   Right, removed the field in favor of `snapshot.channelState()`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();

Review Comment:
   Not intentional — made `FetchedChannelStateSnapshot` `@Internal public` with a public `reader()`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {

Review Comment:
   Not intentional — wrapped it in a `BufferedInputStream`



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   Compressed the first paragraph to the non-obvious part (why the condition is "readAny" rather than "a filter ran") and added the TODO about returning `getProducedChannelState()` once the `Spilling*` handlers are selected.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");
+        currentStream =
+                new OffsetAwareOutputStream(
+                        new BufferedOutputStream(new FileOutputStream(filePath.toFile())), 0L);

Review Comment:
   Done — the file is now opened with `Files.newOutputStream(CREATE_NEW, WRITE)` to fail loudly on collisions.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,

Review Comment:
   Trimmed the javadoc down to two paragraphs; kept the name `nextSegment` since it reads more domain-specific to me than `advanceAndGet`. 
   
   wdyt?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**

Review Comment:
   Compressed it to one line; I'd keep the "no skipping, the stream already sits on the next header" part since that invariant isn't obvious from the code.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelState.java:
##########
@@ -22,42 +22,124 @@
 
 import java.io.Closeable;
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
+import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
- * Sealed container for fetched recovered channel-state data.
+ * Sealed container for recovered channel-state data written to spill files.
+ *
+ * <p>Holds a list of file paths (in write order). Segment boundaries are self-described in disk
+ * segment headers ([4B gateIdx][4B channelIdx][4B bufferLength]), so no in-memory segment locator
+ * table is maintained. The reader scans files sequentially, reading each 12-byte header to obtain
+ * the channel info and body length.
+ *
+ * <p>The file list grows as the writer rotates to new files (one rotation per 64 MB soft limit),

Review Comment:
   I think it is caused by multi-refactor. 
   
   Good catch — reworded: the list is fixed at construction (the growth happens earlier, in the writer), and dropped the stale "Mutations" paragraph as well.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds

Review Comment:
   The committed position is the delivered boundary recorded by `SpillSegment#commit()` — before anything is committed it's simply the reader's start position; now defined in the interface javadoc.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;

Review Comment:
   Renamed.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;
+        }
+        if (currentChannel != null) {
+            sealCurrentSegment();
+        }
+        segmentSerializer.clear();
+        segmentSerializer.writeInt(channelInfo.getGateIdx());
+        segmentSerializer.writeInt(channelInfo.getInputChannelIdx());
+        segmentSerializer.writeInt(0); // bufferLength placeholder
+        currentChannel = channelInfo;
+    }
+
+    /**
+     * Backfills the body length into the segment header and flushes the whole segment to the file
+     * stream. Empty segments (filtered out entirely, or a zero-byte pass-through) are dropped
+     * without opening a file, so no empty file is created.
+     */
+    private void sealCurrentSegment() throws IOException {
+        if (currentChannel == null) {
+            return;
+        }
+        currentChannel = null;
+        int totalBytes = segmentSerializer.length();
+        int bodyBytes = totalBytes - SEGMENT_HEADER_BYTES;
+        if (bodyBytes == 0) {
+            return;
+        }
+        // Math.toIntExact guards against the unlikely case of a single segment > 2 GB.
+        segmentSerializer.writeIntUnsafe(Math.toIntExact(bodyBytes), BUFFER_LENGTH_HEADER_OFFSET);
+        ensureFileOpen();
+        currentStream.write(segmentSerializer.getSharedBuffer(), 0, totalBytes);
+    }
+
+    /**
+     * Ensures an output stream is ready for the next segment, rotating to a fresh file first if the
+     * current one reached the size bound. Rotation happens here, between sealed segments, so a
+     * segment is never split across files.
+     */
+    private void ensureFileOpen() throws IOException {
+        if (currentStream != null && currentStream.getLength() >= maxFileSizeBytes) {
+            currentStream.flush();
+            currentStream.close();
+            currentStream = null;
+        }
+        if (currentStream != null) {
+            return;
+        }
+        // create the spill dir on the first file; no-op afterwards
+        Files.createDirectories(baseDir);
+        Path filePath = baseDir.resolve("spill-segment-" + files.size() + ".bin");
+        currentStream =
+                new OffsetAwareOutputStream(
+                        new BufferedOutputStream(new FileOutputStream(filePath.toFile())), 0L);
+        files.add(filePath);
+    }
+
+    @Override
+    @Nullable
+    FetchedChannelState getProducedChannelState() {

Review Comment:
   Yes, intentional — added a TODO referencing FLINK-38544; the wiring lands with the spilling-backend follow-up (#28662).



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);

Review Comment:
   Done — `skipOnStream` now looks up the path itself.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Is this comment in the wrong line? IIUC, it should be commented on `private void readFully(byte[] buf)`.
   
   Done — switched to `IOUtils.readFully`, wrapping the exception to keep the file/offset context in the message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3732107287


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   Exactly the next PR's version, but it would fail here.
   
   The in-memory handlers produce nothing, so it always returns empty; StreamTask then passes `needsRecovery=false` to requestPartitions, and `RecoveredInputChannel#toInputChannel `asserts `receivedBuffers.isEmpty() `— which the in-memory handler just filled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3724850829


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   I prototyped this: since `commit()` sits on `SpillSegment`, a clean split first requires moving it up onto the drain-side reader (a segment is only valid until the next `nextSegment()`, so "the current segment" is unambiguous). It would look like:
   
   ```java
   // checkpoint side: read-only
   public interface FetchedChannelStateReader extends Closeable {
       Optional<SpillSegment> nextSegment(); // SpillSegment: channelInfo/bodyStream/length, no commit
   }
   
   // drain side
   public interface DrainingChannelStateReader extends FetchedChannelStateReader {
       void commitDelivered(); // was SpillSegment#commit
       FetchedChannelStateSnapshot snapshot();
   }
   ```
   
   However, the split itself doesn't reduce the amount of code at all: the implementation stays a single class, every call site keeps the same number of lines (`segment.commit()` merely becomes `reader.commitDelivered()`), and we end up with one more interface than before — the only gain is that the checkpoint path can no longer call `commit()`/`snapshot()` by construction. I'm happy to do this refactoring, but given that it doesn't actually simplify anything — WDYT?
   
   What does bring real simplification is your suggestion in https://github.com/apache/flink/pull/28661#discussion_r3693955989: once the snapshot carries the resume segment, most of the contractual comments this split would enforce disappear anyway.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3724881811


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {

Review Comment:
   Yes, this comment is about `skipOnStream`, sorry for misplacing.
   I think `skipOnStream` can be replaced by `IOUtils#skipFully`.
   
   `readFully` as well - it was the [next](https://github.com/apache/flink/pull/28661#discussion_r3693955986) comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738825052


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;

Review Comment:
   I still see
   ```
   private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
       if (channelInfo.equals(currentChannel)) {
           return;
       }
       ...
       // size check
   ```
   
   so if there's no change we still keep accumulating in memory?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");
+        readerOpened = true;
+        return new FetchedChannelStateReaderImpl(this);
+    }
+
+    /**
+     * Releases the lifecycle grant held by this snapshot. Called by the reader on close; must not
+     * be called directly by any other party.

Review Comment:
   Thanks. But It seems unclear now how to properly close it: only the reader, the snapshot, or both?
   And closing both would double-free fetched channel state.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   >  Kept non-static since it needs the reader's readBody and position.
   
   Is there any reason not to pass the necessary objects into its contructor?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   Why can't the classes share the common part of the implementation (via inheritance or encapsulation)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738993269


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {

Review Comment:
   How about something like this:
   ```java
     private boolean openCurrentFile() throws IOException {
         boolean rolled = false;
         while (!isLastFile()) {
             openFileAndSeek();
             if (fileHasMore()) {
                 return true;
             }
             checkState(!rolled, "Rolled past more than one empty file");
             closeFileStream();
             current.rollToNextFile();
             rolled = true;
         }
         return false;
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3736227164


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;

Review Comment:
   Yes — added a 1 MB soft bound: a new segment now starts on a channel switch or once the open one outgrew it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3745437253


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   Applied, thanks. I kept the check that only the current segment can be read — reading an older one fails loud.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738928852


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;

Review Comment:
   I assume you did not pull the latest code, `switchChannelIfNeeded` is changed to `startNewSegmentIfNeeded` in e0942ba2fce7f291f157a1b026b7e7eaee5b1183, it introduced `DEFAULT_MAX_SEGMENT_SIZE_BYTES`, the segment will be sealed when channel is changed or segment size is great than 1 MB.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3735939179


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;

Review Comment:
   Done — the reader now stores the last Segment. It only exists for the guard: reads and commits must come from the latest segment, older ones fail loud.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3739122528


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/RecoveredChannelStateHandler.java:
##########
@@ -214,9 +234,368 @@ public void recover(
     }
 }
 
+/**
+ * Intermediate abstract base for the two spilling variants. Owns the on-disk spill format end to
+ * end: a single reusable {@link DataOutputSerializer} accumulates one channel's segment, the
+ * segment header is backfilled with the body length at seal time, and sealed segments are flushed
+ * to the current file stream with 64 MB-bounded rotation.
+ *
+ * <h3>Disk format</h3>
+ *
+ * <pre>
+ * [ 4B BE int: gate idx      ]   segment header: written once per channel segment
+ * [ 4B BE int: channel idx   ]
+ * [ 4B BE int: buffer length ]   segment body byte count (backfilled at segment seal)
+ *   [ 4B BE int: record length ]  repeated for every record in this segment
+ *   [ N bytes: serialized record ]
+ * [ 4B BE int: gate idx      ]   next segment header (channel switch or post-rotation)
+ * ...
+ * </pre>
+ *
+ * <p>The body byte count is only known after the whole segment is written, so each segment is first
+ * accumulated in {@link #segmentSerializer} (header written at open with a zero placeholder) and
+ * {@link DataOutputSerializer#writeIntUnsafe} backfills the length at seal. A segment is one
+ * uninterrupted run of records for a single channel; file rotation happens only after a segment is
+ * fully sealed, so a segment never crosses a file boundary.
+ */
+abstract class AbstractSpillingHandler extends AbstractInputChannelRecoveredStateHandler {
+
+    /** Byte offset of the {@code bufferLength} field within a segment's header. */
+    static final int BUFFER_LENGTH_HEADER_OFFSET = 2 * Integer.BYTES;
+
+    /** Total size of the segment header in bytes: gateIdx + channelIdx + bufferLength. */
+    static final int SEGMENT_HEADER_BYTES = 3 * Integer.BYTES;
+
+    final String[] spillTmpDirectories;
+
+    public static final long DEFAULT_SPILL_FILE_SIZE_BYTES = 64L * 1024 * 1024;
+
+    /** Soft per-file size bound that triggers rotation between segments. */
+    private final long maxFileSizeBytes;
+
+    /**
+     * Accumulates the current segment: the header followed by the body, which is either
+     * length-prefixed filtered records or verbatim pass-through bytes, depending on the subclass.
+     * Reused across segments via {@code clear()}.
+     */
+    private final DataOutputSerializer segmentSerializer = new DataOutputSerializer(256);
+
+    /**
+     * Spill files written so far, in order. The {@link FetchedChannelState} handoff is built from
+     * this list once writing is sealed; an empty list means the handler never spilled any bytes, so
+     * it produces no state.
+     */
+    private final List<Path> files = new ArrayList<>();
+
+    /**
+     * Unique directory for this handler's spill files; created lazily when the first file opens.
+     */
+    private final Path baseDir;
+
+    /**
+     * Output stream to the current spill file; tracks the bytes written so far via {@link
+     * OffsetAwareOutputStream#getLength()} to decide when to rotate. Null before the first segment
+     * is flushed.
+     */
+    @Nullable private OffsetAwareOutputStream currentStream;
+
+    /** Channel whose segment is currently open; null when no segment is in progress. */
+    @Nullable private InputChannelInfo currentChannel;
+
+    @Nullable private FetchedChannelState producedChannelState;
+
+    AbstractSpillingHandler(
+            InputGate[] inputGates,
+            InflightDataRescalingDescriptor channelMapping,
+            String[] spillTmpDirectories,
+            long maxFileSizeBytes) {
+        // FLINK-38544 transitional: the base's third ctor arg is removed when the spilling backend
+        // lands (spilling always implies checkpointing-during-recovery enabled).
+        super(inputGates, channelMapping, true);
+        checkArgument(
+                checkNotNull(spillTmpDirectories).length > 0,
+                "spillTmpDirectories must not be empty");
+        checkArgument(
+                maxFileSizeBytes > 0, "maxFileSizeBytes must be positive: %s", maxFileSizeBytes);
+        this.spillTmpDirectories = spillTmpDirectories;
+        this.maxFileSizeBytes = maxFileSizeBytes;
+        this.baseDir =
+                Paths.get(spillTmpDirectories[0], "flink-channel-spill-" + UUID.randomUUID());
+    }
+
+    /**
+     * Opens (or switches to) the segment for {@code channelInfo} and returns its buffer for the
+     * caller to append the body into. The caller must not seal the segment.
+     */
+    DataOutputSerializer segmentSerializerFor(InputChannelInfo channelInfo) throws IOException {
+        switchChannelIfNeeded(channelInfo);
+        return segmentSerializer;
+    }
+
+    private void switchChannelIfNeeded(InputChannelInfo channelInfo) throws IOException {
+        if (channelInfo.equals(currentChannel)) {
+            return;

Review Comment:
   Yes, sorry, my IDE doesn't work well with worktrees.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3729044885


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,

Review Comment:
   Thanks!



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link

Review Comment:
   Thanks!



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   Why can't we simply change
   ```
   return filterContext.isCheckpointingDuringRecoveryEnabled() && readAny
           ? Optional.of(new FetchedChannelState(java.util.Collections.emptyList()))
           : Optional.empty();
   ```
   to
   ```
   return filterContext.isCheckpointingDuringRecoveryEnabled() && readAny
           ? Optional.ofNullable(stateHandler.getProducedChannelState())
           : Optional.empty();
   ```
   so we don't need to worry about some intermediate states between PRs?
   Or am I missing something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28661:
URL: https://github.com/apache/flink/pull/28661#issuecomment-4894617185

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c66e8e2889545ac983ae275345c8fdc33c5fe13",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c66e8e2889545ac983ae275345c8fdc33c5fe13",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c66e8e2889545ac983ae275345c8fdc33c5fe13 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3739191767


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   ```diff
   diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java
   index 79a4936379b..fd166d1eaf2 100644
   --- a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java
   +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java
   @@ -129,7 +129,9 @@ final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
            openFileAndSeek();
            currentSegment =
                    new Segment(
   -                        this, current.channel, new BoundedSegmentStream(this, current.remaining));
   +                        this,
   +                        current.channel,
   +                        new BoundedSegmentStream(currentFileStream, current, current.remaining));
            return Optional.of(currentSegment);
        }
    
   @@ -144,7 +146,8 @@ final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
                    new Segment(
                            this,
                            header.channelInfo,
   -                        new BoundedSegmentStream(this, header.bufferLength));
   +                        new BoundedSegmentStream(
   +                                currentFileStream, current, header.bufferLength));
            return Optional.of(currentSegment);
        }
    
   @@ -246,18 +249,6 @@ final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
            current.advanceReadOffset(buf.length);
        }
    
   -    /**
   -     * Reads up to {@code len} body bytes from the current file; called only by the body view. The
   -     * only place body progress moves, so {@code readOffset} and {@code remaining} cannot drift.
   -     */
   -    private int readBody(byte[] buf, int off, int len) throws IOException {
   -        int n = currentFileStream.read(buf, off, len);
   -        if (n > 0) {
   -            current.advanceBody(n);
   -        }
   -        return n;
   -    }
   -
        private void closeFileStream() throws IOException {
            if (currentFileStream != null) {
                currentFileStream.close();
   @@ -392,11 +383,13 @@ final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
         * file; the reader owns it.
         */
        private static final class BoundedSegmentStream extends InputStream {
   -        private final FetchedChannelStateReaderImpl reader;
   +        private final InputStream fileStream;
   +        private final Position current;
            private final int length;
    
   -        private BoundedSegmentStream(FetchedChannelStateReaderImpl reader, int length) {
   -            this.reader = reader;
   +        private BoundedSegmentStream(InputStream fileStream, Position current, int length) {
   +            this.fileStream = fileStream;
   +            this.current = current;
                this.length = length;
            }
    
   @@ -414,18 +407,18 @@ final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
    
            @Override
            public int read(byte[] buf, int off, int len) throws IOException {
   -            checkState(
   -                    reader.currentSegment != null && reader.currentSegment.body == this,
   -                    "Reading a segment that is no longer the current one");
   -            if (reader.current.remaining == 0) {
   +            if (current.remaining == 0) {
                    return -1;
                }
   -            int toRead = Math.min(len, reader.current.remaining);
   -            int n = reader.readBody(buf, off, toRead);
   +            int toRead = Math.min(len, current.remaining);
   +            int n = fileStream.read(buf, off, toRead);
   +            if (n > 0) {
   +                current.advanceBody(n);
   +            }
                if (n < 0) {
                    throw new EOFException(
                            "Unexpected EOF in segment body after "
   -                                + (length - reader.current.remaining)
   +                                + (length - current.remaining)
                                    + "/"
                                    + length
                                    + " bytes");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3740378603


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   SGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3630862997


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/ChannelState.java:
##########
@@ -98,4 +111,26 @@ public ChannelState emptyState() {
         sequenceNumberInAnnouncedChannels.clear();
         return this;
     }
+
+    /**
+     * Dispatches checkpoint start: inserts recovery-checkpoint barriers into in-recovery channels
+     * through the trigger, then notifies every input. (FLINK-38544 transitional: the spilling
+     * backend adds a third step handing the trigger's snapshot reader to the channel-state writer.)
+     */
+    public void onCheckpointStartedForAllInputs(CheckpointBarrier barrier)
+            throws CheckpointException, IOException {
+        long cpId = barrier.getId();
+        try {
+            recoveryCheckpointTrigger.snapshotAndInsertBarriers(cpId);
+
+            for (CheckpointableInput input : inputs) {
+                input.checkpointStarted(barrier);
+            }
+        } catch (Throwable t) {
+            if (t instanceof CheckpointException) {
+                throw (CheckpointException) t;
+            }
+            ExceptionUtils.rethrowIOException(t);
+        }
+    }

Review Comment:
   This pr introduced Fetched channel state, but they are not integrated yet. addInputDataFromSpill is not needed for Memory buffer.
   
   They will be done in the next pr https://github.com/apache/flink/pull/28662



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738957300


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   Could be, but needs more efforts, especially `commit()` lives on `SpillSegment`.
   
   > The interface split isn't free on its own either, because commit() lives on SpillSegment. Hiding it from the checkpoint side needs either a generic reader, or a second segment type, or moving commit() up onto the drain-side reader; each of those touches every call site while the implementation stays a single class.
   
   This comment likes a refactor or interface polish, how about treating it as a follow up? I am happy to create a jira and pick it if you agree.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738943902


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");
+        readerOpened = true;
+        return new FetchedChannelStateReaderImpl(this);
+    }
+
+    /**
+     * Releases the lifecycle grant held by this snapshot. Called by the reader on close; must not
+     * be called directly by any other party.

Review Comment:
   The ownership will be passed from snapshot to reader once reader is created from snapshot.
   
   - Closing the snapshot is a no-op once a reader was opened.
   - Snapshot releases it when non-reader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3739058222


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {

Review Comment:
   Nice, adopted — thanks.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReaderImpl.java:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.channel.FetchedChannelStateReader.SpillSegment;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.runtime.checkpoint.channel.AbstractSpillingHandler.SEGMENT_HEADER_BYTES;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The single {@link FetchedChannelStateReader} implementation over a {@link FetchedChannelState}'s
+ * spill files.
+ *
+ * <p>Reading is strictly sequential and never seeks mid-iteration. There is exactly one place that
+ * skips bytes: the very first {@link #nextSegment()} call, where a snapshot reader started mid-body
+ * discards the already-delivered prefix to land on the not-yet-delivered remainder. Every later
+ * call does no skipping at all — the previous body was read to its end, so the stream already sits
+ * on the next segment's header. This "skip only on first positioning" rule is what keeps the
+ * steady-state path free of any seek/skip.
+ *
+ * <p>The reader holds <b>two</b> {@link Position}s and nothing else duplicates them:
+ *
+ * <ul>
+ *   <li>{@code current} — the live read position; its {@code readOffset} is exactly where the open
+ *       file stream sits, advancing as the header and the consumer's body reads consume bytes (the
+ *       latter outside the drainer lock).
+ *   <li>{@code committed} — the delivered boundary; {@link SpillSegment#commit()} advances it from
+ *       {@code current} (under the drainer lock). {@link #snapshot()} derives a new reader from it.
+ * </ul>
+ *
+ * <p>The "previous body fully read before advancing" rule is checked at the {@link #nextSegment()}
+ * entry (the first call is exempt — there is no previous segment). Body ownership is handed to the
+ * consumer, so the reader does not track body progress except through {@code current}.
+ */
+@Internal
+final class FetchedChannelStateReaderImpl implements FetchedChannelStateReader {
+
+    private final FetchedChannelStateSnapshot snapshot;
+    private final FetchedChannelState channelState;
+    private final List<Path> files;
+
+    /** Live read position; {@code readOffset} is where the open stream physically sits. */
+    private final Position current;
+
+    /** Delivered boundary; {@link SpillSegment#commit()} advances it from {@link #current}. */
+    private final Position committed;
+
+    /** Open stream over {@code current.fileIndex}, or {@code null} before the first read. */
+    @Nullable private InputStream fileStream;
+
+    /** Size of the file currently open. */
+    private long currentFileSize;
+
+    /** Body view of the segment returned by the last {@link #nextSegment()}, or {@code null}. */
+    @Nullable private BoundedSegmentStream currentBody;
+
+    private boolean positioned;
+    private boolean closed;
+
+    FetchedChannelStateReaderImpl(FetchedChannelStateSnapshot snapshot) {
+        this.snapshot = snapshot;
+        this.channelState = snapshot.channelState();
+        this.files = channelState.files();
+        // Must copy the position so that this reader's commits do not mutate the snapshot's state.
+        this.committed = snapshot.position().copy();
+        this.current = committed.copy();
+    }
+
+    @Override
+    public Optional<SpillSegment> nextSegment() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        checkState(
+                currentBody == null || currentBody.remaining() == 0,
+                "Previous segment body not fully consumed before advancing: %s bytes left",
+                currentBody == null ? 0 : currentBody.remaining());
+        try {
+            if (!positioned) {
+                positioned = true;
+                return firstSegment();
+            }
+            return followingSegment();
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read segment", e);
+        }
+    }
+
+    /**
+     * First positioning — the only path that may skip bytes. Opens the file at the committed header
+     * offset and reads the header. A snapshot may resume in the middle of a segment: the committed
+     * {@code readOffset} says how many body bytes were already delivered, and that prefix is
+     * skipped so the returned body starts at the not-yet-delivered remainder. If the segment was
+     * already fully delivered (prefix == whole body), it is exhausted here and we move on to the
+     * next one.
+     */
+    private Optional<SpillSegment> firstSegment() throws IOException {
+        // The committed read offset may sit mid-body (after a partial commit), but the header lives
+        // at segmentStartOffset. Capture how much was already delivered, then rewind the live read
+        // offset to the header so we open the file there and read the header, not mid-body.
+        int deliveredPrefix = (int) current.deliveredBodyBytes();
+        current.rewindToSegmentStart();
+
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        checkState(
+                deliveredPrefix <= header.bufferLength,
+                "Delivered offset %s exceeds segment length %s",
+                deliveredPrefix,
+                header.bufferLength);
+
+        if (deliveredPrefix == header.bufferLength) {
+            // This segment was already fully delivered before the snapshot; nothing remains in it.
+            // Skip its whole body to reach the next segment's header, then take the steady path.
+            skipBody(header.bufferLength);
+            return followingSegment();
+        }
+
+        // Discard the already-delivered prefix (the one and only skip in this class), then hand out
+        // the remainder. alreadyDelivered is carried so commit() records the boundary from the
+        // head.
+        skipBody(deliveredPrefix);
+        currentBody =
+                new BoundedSegmentStream(header.bufferLength - deliveredPrefix, deliveredPrefix);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    /**
+     * Steady-state path — no skipping. The previous body was read to its end, so the stream sits
+     * exactly on this segment's header (or at the current file's end, in which case we roll to the
+     * next file). Reads the header and returns the whole-body view.
+     */
+    private Optional<SpillSegment> followingSegment() throws IOException {
+        if (!openCurrentFile()) {
+            return Optional.empty();
+        }
+        SegmentHeader header = readHeaderAtCurrent();
+        currentBody = new BoundedSegmentStream(header.bufferLength);
+        return Optional.of(new Segment(header.channelInfo, currentBody));
+    }
+
+    @Override
+    public FetchedChannelStateSnapshot snapshot() {
+        checkState(!closed, "FetchedChannelStateReader is closed");
+        return new FetchedChannelStateSnapshot(channelState, committed.copy());
+    }
+
+    @Override
+    public void close() throws IOException {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        try {
+            closeFileStream();
+        } finally {
+            snapshot.release();
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Sequential IO over the spill files; all of it advances current.readOffset / current.fileIndex
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * Ensures a file is open with the stream positioned at {@code current}'s read offset, ready to
+     * read this segment's header. Rolls to the next file when the current one is exhausted. Returns
+     * false when no segment remains.
+     *
+     * <p>{@code current.segmentStartOffset} is set to where the header begins, so a later {@link
+     * SpillSegment#commit()} records the right segment for the snapshot to resume from.
+     */
+    private boolean openCurrentFile() throws IOException {
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        // Current file fully read: move to the next file's first segment.
+        closeFileStream();
+        current.rollToNextFile();
+        if (current.fileIndex >= files.size()) {
+            return false;
+        }
+        openFileAndSeek();
+        if (current.readOffset < currentFileSize) {
+            current.startSegmentHere();
+            return true;
+        }
+        return false;
+    }
+
+    /** Reads the 12-byte header at the current read offset; advances past it. */
+    private SegmentHeader readHeaderAtCurrent() throws IOException {
+        byte[] headerBytes = new byte[SEGMENT_HEADER_BYTES];
+        readFully(headerBytes);
+        DataInputStream h = new DataInputStream(new ByteArrayInputStream(headerBytes));
+        int gateIdx = h.readInt();
+        int channelIdx = h.readInt();
+        int bufferLength = h.readInt();
+        checkState(bufferLength >= 0, "negative segment length: %s", bufferLength);
+        return new SegmentHeader(new InputChannelInfo(gateIdx, channelIdx), bufferLength);
+    }
+
+    /**
+     * Ensures the file at {@code current.fileIndex} is open with the stream positioned at {@code
+     * current.readOffset}. If a stream is already open it is left as-is: sequential reading
+     * guarantees it is already there.
+     */
+    private void openFileAndSeek() throws IOException {
+        if (fileStream != null) {
+            return;
+        }
+        Path path = files.get(current.fileIndex);
+        currentFileSize = Files.size(path);
+        InputStream in = Files.newInputStream(path);
+        try {
+            skipOnStream(in, current.readOffset, path);
+        } catch (IOException e) {
+            in.close();
+            throw e;
+        }
+        fileStream = in;
+    }
+
+    /** Skips {@code count} body bytes on the open stream, advancing the read offset. */
+    private void skipBody(long count) throws IOException {
+        if (count > 0) {
+            skipOnStream(fileStream, count, files.get(current.fileIndex));
+            current.advanceReadOffset(count);
+        }
+    }
+
+    /** Skips exactly {@code count} bytes on {@code in}, failing loud if the file ends early. */
+    private void skipOnStream(InputStream in, long count, Path path) throws IOException {
+        long skipped = 0;
+        while (skipped < count) {
+            long s = in.skip(count - skipped);
+            if (s <= 0) {
+                // skip can return 0 near EOF; read-and-discard as a fallback.
+                if (in.read() < 0) {
+                    throw new EOFException(
+                            "Cannot position to offset " + count + " in spill file " + path);
+                }
+                skipped++;
+            } else {
+                skipped += s;
+            }
+        }
+    }
+
+    private void readFully(byte[] buf) throws IOException {
+        int read = 0;
+        while (read < buf.length) {
+            int n = fileStream.read(buf, read, buf.length - read);
+            if (n < 0) {
+                throw new EOFException(
+                        "Truncated segment header in file "
+                                + files.get(current.fileIndex)
+                                + " at offset "
+                                + current.readOffset
+                                + ": expected "
+                                + buf.length
+                                + " bytes, got "
+                                + read);
+            }
+            read += n;
+            current.advanceReadOffset(n);
+        }
+    }
+
+    /** Reads up to {@code len} body bytes from the current file; called only by the body view. */
+    private int readBody(byte[] buf, int off, int len) throws IOException {
+        int n = fileStream.read(buf, off, len);
+        if (n > 0) {
+            current.advanceReadOffset(n);
+        }
+        return n;
+    }
+
+    private void closeFileStream() throws IOException {
+        if (fileStream != null) {
+            fileStream.close();
+            fileStream = null;
+        }
+    }
+
+    // -------------------------------------------------------------------------------------------
+    // Position: the three progress values locating where a snapshot resumes
+    // -------------------------------------------------------------------------------------------
+
+    /**
+     * One progress point, expressed with the three values that fully locate where a snapshot
+     * resumes. The reader holds two of these: {@code current} (live read position) and {@code
+     * committed} (delivered boundary). They are the same shape but different in meaning; neither
+     * keeps a shadow copy of the other.
+     *
+     * <ul>
+     *   <li>{@code fileIndex} — file holding the current segment.
+     *   <li>{@code segmentStartOffset} — byte offset of the current segment's header. A snapshot
+     *       reads the segment metadata (channel, full body length) from here, because the segment
+     *       may already be partially delivered yet the snapshot still needs the whole-segment
+     *       header.
+     *   <li>{@code readOffset} — for {@code current}, exactly where the open stream sits (the live
+     *       read offset); for {@code committed}, the delivered boundary. For a brand-new segment
+     *       {@code readOffset == segmentStartOffset} (header not yet passed); after the header and
+     *       n body bytes it is {@code segmentStartOffset + SEGMENT_HEADER_BYTES + n}.
+     * </ul>
+     */
+    static final class Position {
+        private int fileIndex;
+        private long segmentStartOffset;
+        private long readOffset;
+
+        Position(int fileIndex, long segmentStartOffset, long readOffset) {
+            this.fileIndex = fileIndex;
+            this.segmentStartOffset = segmentStartOffset;
+            this.readOffset = readOffset;
+        }
+
+        static Position atStart() {
+            return new Position(0, 0L, 0L);
+        }
+
+        Position copy() {
+            return new Position(fileIndex, segmentStartOffset, readOffset);
+        }
+
+        /**
+         * Already-delivered body bytes of the current segment, clamped to 0 before the header is
+         * crossed (where {@code readOffset <= segmentStartOffset}). Only the first-positioning path
+         * uses it, to size the prefix a snapshot must discard.
+         */
+        long deliveredBodyBytes() {
+            return Math.max(0L, readOffset - segmentStartOffset - SEGMENT_HEADER_BYTES);
+        }
+
+        /** Advances the live read offset by {@code delta} bytes just read/skipped. */
+        void advanceReadOffset(long delta) {
+            readOffset += delta;
+        }
+
+        /**
+         * Rewinds the live read offset back to the current segment's header. Used only on first
+         * positioning: a snapshot's committed offset may sit mid-body, but the header must be read
+         * first, so the read offset returns to {@code segmentStartOffset} before opening the file.
+         */
+        void rewindToSegmentStart() {
+            readOffset = segmentStartOffset;
+        }
+
+        /**
+         * Marks the current read offset as the start of the segment about to be read (its header
+         * begins here). Called right before reading a header.
+         */
+        void startSegmentHere() {
+            segmentStartOffset = readOffset;
+        }
+
+        /** Rolls to the start of the next file once the current one is exhausted. */
+        void rollToNextFile() {
+            fileIndex++;
+            segmentStartOffset = 0L;
+            readOffset = 0L;
+        }
+
+        /**
+         * Copies this position into {@code target} but pins {@code target}'s read offset to {@code
+         * deliveredBody} body bytes of the current segment — i.e. {@code commit} records "delivered
+         * up to here", not "physically read up to here".
+         */
+        void copyAsDelivered(Position target, long deliveredBody) {
+            target.fileIndex = fileIndex;
+            target.segmentStartOffset = segmentStartOffset;
+            target.readOffset = segmentStartOffset + SEGMENT_HEADER_BYTES + deliveredBody;
+        }
+    }
+
+    /** Parsed segment header: channel and full body length. */
+    private static final class SegmentHeader {
+        private final InputChannelInfo channelInfo;
+        private final int bufferLength;
+
+        private SegmentHeader(InputChannelInfo channelInfo, int bufferLength) {
+            this.channelInfo = channelInfo;
+            this.bufferLength = bufferLength;
+        }
+    }
+
+    /**
+     * The single {@link SpillSegment} implementation. Exposes one segment's channel, body, and
+     * length; {@link #commit()} advances the reader's {@code committed} position to however many
+     * body bytes have been read. Reading the body and committing are separate steps so the consumer
+     * can read outside the drainer lock and commit inside it.
+     *
+     * <p>Only the root (drain) reader commits.
+     */
+    private final class Segment implements SpillSegment {
+        private final InputChannelInfo channelInfo;
+        private final BoundedSegmentStream body;
+
+        private Segment(InputChannelInfo channelInfo, BoundedSegmentStream body) {
+            this.channelInfo = channelInfo;
+            this.body = body;
+        }
+
+        @Override
+        public InputChannelInfo channelInfo() {
+            return channelInfo;
+        }
+
+        @Override
+        public InputStream bodyStream() {
+            return body;
+        }
+
+        @Override
+        public int length() {
+            return body.deliverableLength();
+        }
+
+        @Override
+        public void commit() {
+            current.copyAsDelivered(committed, body.deliveredFromSegmentHead());
+        }
+    }
+
+    /**
+     * A forward-only, bounded view over one segment's not-yet-delivered body remainder. It hands
+     * out {@code remainingLength} bytes and reaches EOF after them; it never reads into the next
+     * segment or file. The underlying stream is already positioned at the first byte this view
+     * hands out (the prefix, if any, was skipped before construction). If the file ends before the
+     * segment end, an {@link EOFException} is thrown (fail-loud). Closing this view does not close
+     * the underlying file; the reader owns it.
+     */
+    private final class BoundedSegmentStream extends InputStream {

Review Comment:
   Done — both are static now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3738943902


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateSnapshot.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * An immutable resume point for a {@link FetchedChannelState} reader. It captures the {@link
+ * FetchedChannelStateReaderImpl.Position} at which reading should start and holds one lifecycle
+ * grant on the underlying {@link FetchedChannelState} (acquired in the constructor).
+ *
+ * <p>A snapshot is a one-shot, single-reader handle:
+ *
+ * <ul>
+ *   <li>Exactly one {@link FetchedChannelStateReader} may be opened from it via {@link #reader()}.
+ *   <li>When that reader is closed, it releases the lifecycle grant via {@link #release()}.
+ * </ul>
+ *
+ * <p>The 1:1 reader constraint is enforced fail-loud: calling {@link #reader()} a second time
+ * throws {@link IllegalStateException} immediately.
+ */
+final class FetchedChannelStateSnapshot {
+
+    private final FetchedChannelState channelState;
+    private final FetchedChannelStateReaderImpl.Position position;
+
+    /** True once {@link #reader()} has been called; prevents opening a second reader. */
+    private boolean readerOpened;
+
+    /**
+     * Creates a snapshot at {@code position} within {@code channelState}. Acquires one lifecycle
+     * grant on {@code channelState}; the grant is released when the reader returned by {@link
+     * #reader()} is closed.
+     */
+    FetchedChannelStateSnapshot(
+            FetchedChannelState channelState, FetchedChannelStateReaderImpl.Position position) {
+        this.channelState = channelState;
+        this.position = position;
+        channelState.acquire();
+    }
+
+    /**
+     * Opens the reader for this snapshot. May be called at most once; a second call fails loud to
+     * enforce the 1:1 snapshot-to-reader invariant.
+     *
+     * @return a new reader starting from this snapshot's position; caller must close it when done
+     */
+    FetchedChannelStateReader reader() {
+        checkState(!readerOpened, "A reader has already been opened from this snapshot");
+        readerOpened = true;
+        return new FetchedChannelStateReaderImpl(this);
+    }
+
+    /**
+     * Releases the lifecycle grant held by this snapshot. Called by the reader on close; must not
+     * be called directly by any other party.

Review Comment:
   The ownership will be passed from snapshot to reader once reader is created from snapshot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3724975334


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/FetchedChannelStateReader.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Closeable;
+import java.io.InputStream;
+import java.util.Collections;
+import java.util.Optional;
+
+/**
+ * Forward reader over a {@link FetchedChannelState}'s spill files. This is our own segment reader,
+ * on purpose <em>not</em> a Java {@link java.util.Iterator}: our access pattern ("a body must be
+ * fully read before the next segment", "body ownership is handed to the consumer", "consume and
+ * commit are separate steps") does not fit the {@code hasNext/next} contract.
+ *
+ * <p>This interface is the contract callers depend on; {@link FetchedChannelStateReaderImpl} holds
+ * the implementation (the live file stream, the two progress positions, the bounded body view).
+ *
+ * <p>Reading is strictly sequential: a reader is positioned once (offset 0 for the root reader, or
+ * the committed position for a {@link #snapshot()}), then consumes forward only via {@link
+ * #nextSegment()}. It never seeks backward and never re-positions mid-iteration.
+ *
+ * <p>The drain thread reads the root reader front to back and commits via {@link
+ * SpillSegment#commit()}; each checkpoint derives a fresh {@link #snapshot()} that resumes from the
+ * committed position. {@link #snapshot()} and {@link SpillSegment#commit()} must be called under
+ * the drainer lock; disk reads happen outside it.
+ */
+@Internal
+public interface FetchedChannelStateReader extends Closeable {
+
+    /**
+     * Advances to the next segment and returns it, or {@link Optional#empty()} when no segment
+     * remains. Advancing and probing are one step; there is no separate {@code hasNext}.
+     *
+     * <p>Entry rule (the first call is exempt): the previous segment's body must be fully read,
+     * otherwise this is a contract violation and fails loud (no skip-ahead).
+     */
+    Optional<SpillSegment> nextSegment();
+
+    /**
+     * Derives an independent resume point starting from the committed position. The snapshot holds
+     * its own {@link FetchedChannelState} lifecycle grant; the caller must open a reader from it
+     * via {@link FetchedChannelStateSnapshot#reader()} and close that reader when done.
+     *
+     * <p>Must be called under the drainer lock so that the copied position reflects the latest
+     * committed state.
+     *
+     * @return a snapshot capturing the current committed position; caller must open and close a
+     *     reader from it
+     */
+    FetchedChannelStateSnapshot snapshot();
+
+    /**
+     * Returns a reader with no segments — its first {@link #nextSegment()} is empty. Each call
+     * hands out a fresh instance: readers have independent lifecycle and {@link #close()} is
+     * single-use, so a shared instance would let one consumer's close break later consumers. Used
+     * wherever there is nothing to snapshot (e.g. after drain finished, or the no-op recovery
+     * trigger).
+     */
+    static FetchedChannelStateReader emptyReader() {
+        return new FetchedChannelState(Collections.emptyList()).reader();
+    }
+
+    /**
+     * One per-channel segment produced by {@link #nextSegment()}.
+     *
+     * <p>The segment body bytes are opaque to the reader; record framing is handled by the
+     * consumer's deserializer. A consumer reads {@link #bodyStream()} to EOF (after {@link
+     * #length()} bytes), and the drain consumer additionally calls {@link #commit()} under the
+     * drainer lock after each delivery so that a later {@link FetchedChannelStateReader#snapshot()}
+     * resumes from the delivered boundary.
+     *
+     * <p>Ownership of {@link #bodyStream()} passes to the consumer: the reader no longer tracks how
+     * far it has been read. The "previous body must be fully read" rule (no skip-ahead) is enforced
+     * at the next {@link FetchedChannelStateReader#nextSegment()} call, not here.
+     *
+     * <p>A segment is valid only until the next {@code nextSegment()} call on the parent reader.
+     */
+    interface SpillSegment {
+
+        /** The channel whose data this segment contains. */
+        InputChannelInfo channelInfo();
+
+        /**
+         * Returns an {@link InputStream} bounded to this segment's body. Reading returns {@code -1}
+         * (EOF) after {@link #length()} bytes; it never reads into the next segment or the next
+         * file.
+         *
+         * <p>The stream is single-use, not thread-safe, and must be fully consumed before the next
+         * {@link FetchedChannelStateReader#nextSegment()}.
+         */
+        InputStream bodyStream();
+
+        /**
+         * Number of body bytes this segment hands out before EOF. For the snapshot path this is the
+         * not-yet-delivered remainder used as the length prefix when writing to the checkpoint
+         * stream. Bounded by the spill file size limit, so it always fits in an {@code int}.
+         */
+        int length();
+
+        /**
+         * Advances the reader's committed position to match how many body bytes have been read from
+         * {@link #bodyStream()} so far. Must be called under the drainer lock after each buffer
+         * delivery so that a subsequent {@link FetchedChannelStateReader#snapshot()} sees the
+         * correct delivered boundary. Only the drain (root) reader commits; the snapshot reader

Review Comment:
   I think this interface split already makes sense - it makes it clear how each call site uses the interface.
   I don't understand why the implementation is the same? Can't we split it as well?
   
   > What does bring real simplification is your suggestion in #28661 (comment): once the snapshot carries the resume segment, most of the contractual comments this split would enforce disappear anyway.
   
   Probably we should discuss this offline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39524][checkpoint] Fetched channel state: spill files, filtered write, reader [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28661:
URL: https://github.com/apache/flink/pull/28661#discussion_r3728495379


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImpl.java:
##########
@@ -110,7 +111,7 @@ public Optional<FetchedChannelState> readInputData(
             // only signals "there is state to recover". The spilling backend returns a real,
             // file-backed container here.

Review Comment:
   But shouldn't it be closed anyways?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org