You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "dybyte (via GitHub)" <gi...@apache.org> on 2026/07/19 12:30:45 UTC

[PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

dybyte opened a new pull request, #11494:
URL: https://github.com/apache/seatunnel/pull/11494

   ### Purpose of this pull request
   
   This PR reduces memory usage during file-based IMap WAL recovery.
   
   Previously, WAL files and all deserialized mutations were retained in memory and sorted. This could cause high heap usage as WAL history grows.
   
   This PR:
   
   - Reads WAL records incrementally.
   - Retains only the latest mutation for each key.
   - Removes the full mutation list and global sorting.
   
   This reduces recovery memory usage from being proportional to the total number of WAL mutations to the number of unique keys.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   Added unit tests covering:
   
   - Retaining only one latest mutation per key.
   - Selecting the latest mutation regardless of WAL file order.
   - Selective key loading.
   - Recovery with an incomplete trailing WAL record.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5190591222

   Thanks @SEZ9 for staying on this one too — I want to push back gently on the Issue 1 conclusion this time, with the evidence for why.
   
   I checked the current `dev` baseline for the two methods this concern is about:
   
   - `DefaultReader.readData(Path path)` is already `private` on `dev` (`DefaultReader.java:89`). A private method has no external callers by definition, so a signature change here — streaming via `forEachData`/`RecordConsumer` instead of returning a `List` — cannot be a public API break; it's an internal implementation detail of one class.
   - `IFileReader.readAllData(Path parentPath)` stays on the interface as a default method specifically so that any third-party `IFileReader` implementation that only overrode the old method keeps working — this is exactly the "Backward-compat path" your own 2026-07-22 review traced (`IFileReader.forEachData` default impl falling back to `readAllData`). That's the one place a real downstream implementer could be affected, and this PR deliberately keeps it in place rather than removing it.
   
   `WALReader` itself is internal `seatunnel-engine-storage` plugin code, not a `SeaTunnelSource`/`SeaTunnelSink` SPI surface, so there's no connector-facing contract riding on its method shapes either.
   
   Given that, I'm keeping my 2026-08-04 approval as Daniel's current conclusion: no open source-side blocker from my side on this head (`bccb2761`), recovery semantics unchanged, WAL format unchanged, checks green. If you're aware of a concrete external caller of `DefaultReader.readData` or a specific `IFileReader` implementation outside this module that would break, I'd genuinely like to see it — happy to reopen the blocker with that evidence. Note I only have read-level review permission here, so a write-capable maintainer still needs to handle the actual merge/approval-state step regardless of how this discussion resolves.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5534466953

   Thanks for running the benchmarks and diagnostics, @nzw921rx — good to have the numbers documented on the thread.
   
   One flag while I'm here: the current head (`f75ab5036`) has a fresh `Build` failure in the fork CI (`transform-v2-it-part-1 (8, ubuntu-latest)`, run finished 2026-09-03T15:08Z), which is newer than my last check-in on this thread. @dybyte, could you take a look and confirm whether that's a known flake in that IT bucket or something this change actually touches? I'm not treating it as a new blocker yet since I haven't dug into the failure log myself, but wanted to surface it before it gets buried under the benchmark discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5502443158

   Thanks @nzw921rx for taking a look and for the `+1 LGTM`!
   
   For the record, since a couple of reviews landed close together on this thread: my own conclusion on this PR is unchanged and still stands at **APPROVED** (submitted 2026-08-04 on head `bccb2761b`). I re-verified that on 2026-08-26 after @SEZ9's latest full pass — none of that round's findings carried forward as a blocking item either. The two items worth the author's attention going forward are non-blocking follow-ups:
   - Issue 3: negative/corrupted `dataLength` not guarded in WAL record parsing
   - Issue 6: `loadAllKeys` still retaining full `IMapFileData` (including value bytes) rather than a key-only footprint
   
   Neither blocks merge on the current head. So from a source-review standpoint there's no new blocker here — the remaining step is just going through the normal merge process. Thanks again for the extra eyes on this one!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5038686552

   > Thanks for the update. I re-reviewed the complete current-head diff and the affected runtime path from scratch. The direction is useful, but the following items still block merge:
   > 
   > 1. **Blocker 1: Equal-timestamp WAL mutations still have no stable total order** (`seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/common/LatestMutationAccumulator.java:47-55`) — The latest commit switched to `mutation.compareTo(current) < 0`, but `IMapFileData.compareTo()` returns `-1` even for equal objects. Both directions compare as smaller and `x.compareTo(x) != 0`, so recovery now chooses the last filesystem traversal result rather than a durable latest mutation. Please persist a cross-writer/region comparable epoch plus sequence, use a deterministic per-key order, and test equal timestamps across files in opposite traversal orders.
   > 
   > The current Build is `PENDING`. Please rerun the required checks after the source changes.
   > 
   > Thanks again for working through this — happy to re-review the next head.
   > 
   > Queue note: this head is `diverged` against `dev` (ahead_by=3, behind_by=4). I am recording that fact separately; it is not by itself a request to sync unless the CI guidance above explicitly says so.
   
   The equal-timestamp ordering issue is pre-existing in the current WAL recovery path and is caused by the current IMapFileData.compareTo() implementation.
   The comparator contract and deterministic ordering for equal-timestamp mutations are being fixed separately in #11465. This PR intentionally reuses IMapFileData.compareTo() so that, once #11465 is merged and this branch is updated, the streaming recovery path will preserve the same ordering semantics as the full-sort recovery path.
   Persisting a cross-writer/region epoch and sequence would require a broader WAL format and writer protocol change. That limitation already exists in the current implementation and is not introduced by this PR. If strict causal ordering across writers is required, I believe it should be tracked separately from this memory-usage optimization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5090089683

   @DanielLeens This compatibility issue has already been addressed in the current head bccb2761b


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5496380651

   <img width="4070" height="3090" alt="b0384c84bb3f360f05362bdaec1c208d" src="https://github.com/user-attachments/assets/26667dad-518c-497e-b450-9f7265215bf2" />
   
   A great optimization once🚀


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5092874750

   Thanks for the clarification. I checked the latest head `bccb2761b` and the current `Build` check is green now. From Daniel's side, I will not keep the earlier compatibility concern as an active source blocker on this head.\n\nThe remaining gate is the repository review/merge gate: my permission here is read-only, so I can leave this technical follow-up but cannot turn the GitHub review summary into a formal approval. A write-capable maintainer still needs to handle the final approval/merge step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5419378026

   Thanks for the fresh full pass, @SEZ9 — and thanks for continuing to engage on this one.
   
   First, on the earlier Issue 1 (possible public API break on `DefaultReader.readData`): this new review doesn't carry it forward, which lines up with the evidence in my 2026-08-05 reply — `readData` is `private` on the current head `bccb2761b`, so I'll take that as resolved unless you want to keep it open explicitly.
   
   I re-verified the current head against the eight new findings before replying:
   
   - Issue 3 (negative/corrupted `dataLength` not guarded) checks out. `WALDataUtils.byteArrayToInt` does a signed left-shift on the top byte, so a corrupted/torn length can produce a negative `int`. The `dataLength > remainingBytes` check compares an `int` against a `long` and never catches a negative value, so `new byte[dataLength]` throws `NegativeArraySizeException` and aborts the whole recovery instead of skipping a bad tail record — exactly the kind of input WAL recovery has to tolerate.
   - Issue 6 (`loadAllKeys` retaining full `IMapFileData` incl. value bytes) also checks out. `LatestMutationAccumulator.latestMutations` is `Map<SerializedKey, IMapFileData>`, and `WALReader.loadAllKeys` only reads `mutation.getKey()`/`getKeyClassName()` off it, so peak heap for a key-only listing still scales with total live value bytes rather than key count — that meaningfully undercuts this PR's own memory-optimization claim for that code path.
   - Issues 1/2/5/7 (serialized-byte vs deserialized-object key identity): the code matches your read — `SerializedKey.equals` is `Arrays.equals` on raw bytes. Worth noting `IMapFileData.key` was already a raw `byte[]` on `dev` before this PR, so the byte-level storage itself isn't new; what's new is using raw-byte identity for dedup/tombstone matching in the accumulator. I agree this is a real, worth-documenting property of the new path, though I'd want to see a concrete non-canonical serializer actually in use in this codebase before calling it a live correctness bug rather than a documented assumption.
   - Issue 8 (inverted `compareTo` needs a comment): confirmed. `IMapDataComparator.compare` orders the newer record first, so `mutation.compareTo(current) < 0` in the accumulator does mean "mutation is newer" — a one-line comment there is cheap insurance against a future flip.
   
   None of these land in a "Blocking items" section this round (there isn't one), so I'm reading this as follow-ups rather than a renewed hold, and my 2026-08-04 approval stands as-is. Issues 3 and 6 look like the most valuable ones for the author to pick up given they touch correctness and the memory-optimization goal directly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5539323470

   Follow-up on my comment above: the `transform-v2-it-part-1 (8, ubuntu-latest)` failure I flagged (fork run finished 2026-09-03T15:08Z) turned out to be transient. A rerun on this same, unchanged head (`f75ab5036`) started right after at 15:13Z and completed successfully at 18:44Z (https://github.com/apache/seatunnel/runs/100702174541). The `Build` check is green end-to-end now.
   
   No action needed from @dybyte after all — my 2026-08-04 "Ready to merge" approval stands.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494#issuecomment-5528752997

   benchmarks compare result: https://github.com/apache/seatunnel/actions/runs/33771937553


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Zeta] Optimize WAL recovery memory usage [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11494:
URL: https://github.com/apache/seatunnel/pull/11494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org