You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2026/07/01 21:18:44 UTC
[PR] [FLINK-40016][checkpoint] Prevent recovered buffers from being persisted twice in LocalInputChannel [flink]
1996fanrui opened a new pull request, #28605:
URL: https://github.com/apache/flink/pull/28605
## What is the purpose of the change
`LocalInputChannel.getBufferAndAvailability()` was calling `channelStatePersister.maybePersist(buffer)` unconditionally for all buffers, including recovered buffers that were already spilled to channel state by `checkpointStarted()`. This caused recovered buffers to be persisted twice during a checkpoint.
## Brief change log
- Add `fromUpstream` parameter to `getBufferAndAvailability()` to distinguish buffers pulled from the upstream `ResultSubpartitionView` vs. recovered buffers from `toBeConsumedBuffers`.
- Only call `maybePersist()` when `fromUpstream == true`, since recovered buffers are already fully spilled by `checkpointStarted()`.
- Add unit test `testRecoveredBuffersNotPersistedAgainWhenConsumedDuringCheckpoint` that verifies recovered buffers are not re-added to the channel state writer when consumed during a checkpoint.
## Verifying this change
- Added `testRecoveredBuffersNotPersistedAgainWhenConsumedDuringCheckpoint` in `LocalInputChannelTest`: starts a checkpoint, verifies 2 recovered buffers are persisted by `checkpointStarted`, consumes the recovered buffers, then asserts the total persisted count is still 2 (not 4).
## Does this pull request potentially affect one of the following areas
- **Network stack / checkpointing**: yes — `LocalInputChannel` channel-state persistence path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40016][checkpoint] Prevent recovered buffers from being persisted twice in LocalInputChannel [flink]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28605:
URL: https://github.com/apache/flink/pull/28605#discussion_r3509501277
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannel.java:
##########
@@ -329,10 +329,10 @@ public Optional<BufferAndAvailability> getNextBuffer() throws IOException {
seq++));
}
- return getBufferAndAvailability(toBeConsumedBuffers.removeFirst());
+ return getBufferAndAvailability(toBeConsumedBuffers.removeFirst(), true);
}
- return getBufferAndAvailability(next);
+ return getBufferAndAvailability(next, true);
Review Comment:
Thanks for the quick review!
Make sense, and updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40016][checkpoint] Prevent recovered buffers from being persisted twice in LocalInputChannel [flink]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28605:
URL: https://github.com/apache/flink/pull/28605#discussion_r3509554387
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannel.java:
##########
@@ -329,10 +329,10 @@ public Optional<BufferAndAvailability> getNextBuffer() throws IOException {
seq++));
}
- return getBufferAndAvailability(toBeConsumedBuffers.removeFirst());
+ return getBufferAndAvailability(toBeConsumedBuffers.removeFirst(), true);
}
- return getBufferAndAvailability(next);
+ return getBufferAndAvailability(next, true);
Review Comment:
and `channelStatePersister.checkForBarrier` is moved to call side as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40016][checkpoint] Prevent recovered buffers from being persisted twice in LocalInputChannel [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28605:
URL: https://github.com/apache/flink/pull/28605#issuecomment-4860177676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "da409ff0d2a11f9d59a36e46df3c17dfb69e784a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da409ff0d2a11f9d59a36e46df3c17dfb69e784a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da409ff0d2a11f9d59a36e46df3c17dfb69e784a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40016][checkpoint] Prevent recovered buffers from being persisted twice in LocalInputChannel [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28605:
URL: https://github.com/apache/flink/pull/28605#discussion_r3509354778
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannel.java:
##########
@@ -329,10 +329,10 @@ public Optional<BufferAndAvailability> getNextBuffer() throws IOException {
seq++));
}
- return getBufferAndAvailability(toBeConsumedBuffers.removeFirst());
+ return getBufferAndAvailability(toBeConsumedBuffers.removeFirst(), true);
}
- return getBufferAndAvailability(next);
+ return getBufferAndAvailability(next, true);
Review Comment:
I think it would be a bit cleaner and more robust if we just pull `maybePersist` to the call-site.
That would work correctly for FullyFilledBuffer in case it ever supports checkpointing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org