You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "infocusmodereal (via GitHub)" <gi...@apache.org> on 2026/02/04 17:25:28 UTC
[PR] [FLINK-39000] Avoid redundant seeks during operator list state restore [flink]
infocusmodereal opened a new pull request, #27527:
URL: https://github.com/apache/flink/pull/27527
This PR addresses FLINK-39000.
Problem
- Operator list state restore performs FSDataInputStream.seek(offset) for every element in OperatorStateRestoreOperation.deserializeOperatorStateValues, even when offsets are already sequential.
- On object stores (S3/GCS/Ceph/etc.) and for some stream wrappers (e.g. compressed input streams), repeated seeks can be expensive and dominate restore time for large operator list state.
Changes
- Track the current stream position and only call seek() when the desired offset differs.
- Add a unit test that wraps the state handle input stream to count seek() calls and asserts that sequential offsets result in minimal seeks (covers snapshot compression enabled/disabled).
Tests
- ./mvnw -pl flink-runtime -Dtest=OperatorStateRestoreOperationTest test -Djdk17 -Pjava17-target
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #27527:
URL: https://github.com/apache/flink/pull/27527#discussion_r3000040457
##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateRestoreOperation.java:
##########
@@ -225,15 +225,19 @@ private <S> void deserializeOperatorStateValues(
OperatorStateHandle.StateMetaInfo metaInfo)
throws IOException {
- if (null != metaInfo) {
+ if (metaInfo != null) {
long[] offsets = metaInfo.getOffsets();
- if (null != offsets) {
+ if (offsets != null) {
DataInputView div = new DataInputViewStreamWrapper(in);
TypeSerializer<S> serializer =
stateListForName.getStateMetaInfo().getPartitionStateSerializer();
+ long currentPos = in.getPos();
for (long offset : offsets) {
- in.seek(offset);
+ if (currentPos != offset) {
+ in.seek(offset);
+ }
stateListForName.add(serializer.deserialize(div));
+ currentPos = in.getPos();
Review Comment:
I tend to say that `getPos` and `seek` contract must be fine on all filesystem implementations so we're on the safe side. WDYT @Zakelly? The only concern is that if there are FS implementations which are not always fulfilling this but I've not met such so I would say good to go and planning to merge if no objections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #27527:
URL: https://github.com/apache/flink/pull/27527#discussion_r3005602295
##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateRestoreOperation.java:
##########
@@ -225,15 +225,19 @@ private <S> void deserializeOperatorStateValues(
OperatorStateHandle.StateMetaInfo metaInfo)
throws IOException {
- if (null != metaInfo) {
+ if (metaInfo != null) {
long[] offsets = metaInfo.getOffsets();
- if (null != offsets) {
+ if (offsets != null) {
DataInputView div = new DataInputViewStreamWrapper(in);
TypeSerializer<S> serializer =
stateListForName.getStateMetaInfo().getPartitionStateSerializer();
+ long currentPos = in.getPos();
for (long offset : offsets) {
- in.seek(offset);
+ if (currentPos != offset) {
+ in.seek(offset);
+ }
stateListForName.add(serializer.deserialize(div));
+ currentPos = in.getPos();
Review Comment:
I think we are safe here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] Avoid redundant seeks during operator list state restore [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-3848793835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35c3e81211ad631f638f3346b5ee0236a34b55c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35c3e81211ad631f638f3346b5ee0236a34b55c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35c3e81211ad631f638f3346b5ee0236a34b55c3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-4119662737
Hmmm, that looks promising. I must find some to test it internally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #27527:
URL: https://github.com/apache/flink/pull/27527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "infocusmodereal (via GitHub)" <gi...@apache.org>.
infocusmodereal commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-4110156316
Herw is the follow up with a more production-shaped validation for this patch:
This was run from a state derived from actual production checkpoint metadata and object store settings from two running Flink CDC jobs on our internal Kubernetes cluster, with checkpoints stored in Ceph/S3A.
I inspected the latest checkpoints of two production jobs. In those checkpoints, the relevant `managed operator state` handled by `OperatorStateRestoreOperation` had:
- job A: `395` total operator-state partitions (offsets)
- job B: `510` total operator-state partitions (offsets)
In both cases, the non-empty operator-state handles were mostly concentrated in a single merged task-owned state object. So the patch-relevant restore pattern is a few hundred sequential offsets in merged operator-state handles.
To benchmarked it, I built a synthetic job whose operator list-state shape was derived from the larger real checkpoint (`510` offsets total). I used a Ceph/S3A-backed checkpoint store and prod-like restore settings:
- RocksDB state backend
- incremental checkpoints
- file merging enabled
- same S3A/Ceph configuration family as the production jobs
I ran `patched` and `baseline` jobs sequentially, and used separate storage prefixes for each run to avoid object-store contention. I added temporary logging in `OperatorStateRestoreOperation` to count performed vs skipped seeks and to measure the restore phase directly.
Results:
| Load | Offsets | Restore span patched | Restore span baseline | Restore improvement | Deploy->Running patched | Deploy->Running baseline | End-to-end improvement |
|---|---:|---:|---:|---:|---:|---:|---:|
| 20% | 101 | 634 ms | 1055 ms | 39.9% | 3084 ms | 3344 ms | 7.8% |
| 40% | 203 | 630 ms | 1140 ms | 44.7% | 3267 ms | 3351 ms | 2.5% |
| 60% | 307 | 1306 ms | 1892 ms | 31.0% | 3562 ms | 4081 ms | 12.7% |
| 80% | 409 | 1432 ms | 2089 ms | 31.5% | 4485 ms | 5188 ms | 13.6% |
| 100% | 510 | 1624 ms | 2453 ms | 33.8% | 5095 ms | 5889 ms | 13.5% |
- The patch behaved exactly as intended: baseline performed one seek per offset, while the patched build reduced that to `0` performed seeks in these runs.
- The direct restore-phase improvement was consistent across the full matrix.
- The end-to-end task startup improvement was relatively small, but still signicficant at the higher loads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] Avoid redundant seeks during operator list state restore [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-4037948630
Thanks for the contribution, please fill the original PR form.
Have you tested it with real world workload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "infocusmodereal (via GitHub)" <gi...@apache.org>.
infocusmodereal commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-4091963522
I updated the PR description to use the standard Flink PR template.
On the workload question: not yet with a real end-to-end object-store-backed workload. So far I have verified this with:
- the added regression test in `OperatorStateRestoreOperationTest`
- a synthetic restore experiment on Flink's real snapshot/restore path that injects 1 ms latency per `FSDataInputStream.seek()` to approximate object-store seek/reset overhead
In that synthetic experiment with a 10k-element operator `ListState` and snapshot compression enabled, restore time dropped from about 12.7s with 10,000 seeks to about 18.8ms with 0 seeks.
I agree that an end-to-end validation against a real object-store-backed checkpoint would be stronger evidence, and I can follow up with that if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39000] [runtime] Avoid redundant seeks during operator list state restore [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #27527:
URL: https://github.com/apache/flink/pull/27527#issuecomment-4154976128
Intended to merge in couple of days unless further comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org