You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2026/08/12 14:30:48 UTC
[PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
zhangshenghang opened a new pull request, #11781:
URL: https://github.com/apache/seatunnel/pull/11781
## Purpose
Fixes #9928.
Job status classes were moved from `engine-core` to `engine-common`, but file-backed IMap WAL records preserve their original class names. Restarting a newer SeaTunnel version therefore failed to load persisted job state from older versions.
## Changes
- Resolve legacy job model class names to their current `engine-common` locations before WAL deserialization.
- Cover `JobResult`, `JobStatus`, and `JobStatusData`, which moved together.
- Add a regression test that reads a WAL record carrying the legacy `JobStatus` class name.
## Verification
- `./mvnw -pl seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file -am -Dtest=WALReaderAndWriterTest -Dsurefire.failIfNoSpecifiedTests=false package`
- `./mvnw -q -DskipTests verify`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6117104368
Thanks for the CI update on `452cd144` — CI is not the blocker here; the remaining items from my previous review are. To close this out:
**Must-fix before I can approve**
1. **PR11781-F1 / PR11781-F3 (Test coverage of the legacy mappings)** – `WALReaderAndWriterTest` currently replays only the `JobStatus` mapping and asserts `getClass()`/`toString()`. Please extend it to cover all three legacy names (`JobStatus`, `JobResult`, `JobStatusData`) and assert on the actual deserialized value (enum identity / field values), not just the class name. For the two non-enum targets, please also add a compile-time check of the mapped target class names via the test-scoped `seatunnel-engine-common` dependency, so a future rename of the target classes fails the build instead of silently breaking the shim.
2. **PR11781-F2 (Diagnosability after rewrite)** – when the rewritten name still throws `ClassNotFoundException`, the error only reports the original WAL class name. Please include both the legacy name and the mapped target in the exception message (or log) so a stale/wrong mapping is diagnosable from the restart failure.
**Nice-to-have / can be a follow-up**
3. **PR11781-F4 / PR11781-F5 (Docs)** – a short entry in `incompatible-changes.md` describing the #9689 relocation, that this shim restores reading of pre-relocation WAL files, and the downgrade semantics (WAL written by the new version is not readable by old versions) would help users on file-based IMap storage.
4. **PR11781-F6 (pom.xml)** – a one-line comment on the new test-scoped `seatunnel-engine-common` dependency explaining that production code intentionally references the class names as strings.
5. **PR11781-F7 (Fixture)** – if feasible, check in a small WAL frame produced by a pre-#9689 build, or document why generating it with the current `WALWriter` is equivalent.
6. **PR11781-F8 (Lookup design)** – either resolve the mapping transitively (loop until no further rewrite) or adjust the Javadoc so it does not suggest that simply extending the map is sufficient for a future relocation.
Once items 1 and 2 are addressed, please ping me and I'll re-review promptly. If you'd rather split 3–6 into a follow-up PR, just say so and I'll track it separately.
<!-- streview-comment:1692 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6149834430
Hi @SEZ9, yes — exactly: my F5 was referring to your F7, i.e. replaying a WAL frame actually written by a pre-#9689 release rather than one produced by the current WALWriter. Producing a genuine old-format fixture needs a build from the pre-relocation release line, so I'd prefer to keep it as a follow-up once this shim lands. Thanks for the careful review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-5903298554
Thanks @nzw921rx and @dybyte for the approvals.
Before I merge, I'd like to close out the open points from my earlier review — as far as I can tell from the thread, none of them have been addressed on `835b264f3b` yet. If some already were, please point me to where and I'll re-check.
Remaining asks, in priority order:
1. **Test coverage of the legacy mapping** (`WALReaderAndWriterTest.java`): the regression test currently only asserts `getClass()`/`toString()` on the replayed enum and exercises just the `JobStatus` mapping. Please extend it to cover all three legacy names (`JobStatus`, `JobResult`, `JobStatusData`) and assert on the actual deserialized values, not just the class/string form. Otherwise the `JobResult`/`JobStatusData` target names remain unverified string literals the compiler can't check.
2. **Error reporting after rewrite** (`WALReader.java`): when the class lookup fails after a legacy-name rewrite, the exception still only reports the original WAL class name. Please include the rewritten target name too, so a wrong or stale mapping is diagnosable rather than just aborting the cluster restart with a misleading message.
3. **Docs**: please add a short user-facing note (and an entry in `incompatible-changes.md`) describing the #9689 relocation and what this shim does on upgrade/downgrade for clusters using file-based IMap storage. Javadoc alone won't reach operators.
4. **Minor**: a one-line comment on the new test-scoped `seatunnel-engine-common` dependency in `pom.xml` explaining why production code deliberately references the class names as strings; and, if feasible, consider making the lookup handle chained relocations (or adjust the "extend it" Javadoc guidance so it matches the single-hop behaviour). Ideally the fixture would also be a WAL frame captured from a pre-#9689 build rather than one produced by the current `WALWriter`, but I won't block on that one.
Items 1 and 2 are what I'd like to see before merging; 3 and 4 can be follow-ups if you prefer, just let me know. Thanks for the patience on this.
<!-- streview-comment:1426 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-5881964302
Thanks for the `+1`, @dybyte! Nothing new to verify on my end since my last comment — head is still `835b264f3b`, CI is green, and `mergeable_state` is `clean` with two prior approvals plus yours now. This one's ready to merge whenever a maintainer with merge rights picks it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6313279148
Done in 5dbf76a83: the TODO in `WALReaderAndWriterTest` is replaced with the justification (identical field layout and enum ordinals before and after #9689; the static `serialVersionUID` added by #11257 is skipped by `RuntimeSchema`; the only wire difference is the `valueClassName` string, which the legacy fixtures set to the old names explicitly). The same argument is now in the PR description. A captured byte fixture stays a welcome follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6159753096
Hi @zhangshenghang, thanks for clarifying — that lines up, your F5 and my F7 are the same point: replaying a frame actually written by a pre-#9689 release, not one produced by the current WALWriter.
Agreed on deferring that. Producing a genuine old-format fixture does need a build from the pre-relocation release line, and that's not something I'd block this shim on. Two small asks so it doesn't get lost:
- Please add a short note (a TODO in `WALReaderAndWriterTest.java` or a line in the PR description) stating that the legacy fixture is currently generated by the current WALWriter and that a real pre-#9689 frame fixture is planned as a follow-up.
- Once this lands, please open the follow-up issue and link it back here so we have the trail.
On the rest of the earlier review, could you confirm where things stand so I can do a final pass:
- F1/F3: whether the regression test now asserts on the actual enum value (not only `getClass()`/`toString()`) and covers all three legacy mappings, including the `JobResult` and `JobStatusData` targets, so the string literals in `WALReader.java` are verified by a test rather than left unchecked.
- F2: whether a `ClassNotFoundException` after the legacy-name rewrite now reports the rewritten target name alongside the original WAL class name.
- F4/F5: whether an upgrade note was added for file-based IMap storage users and whether the #9689 relocation plus this shim's upgrade/downgrade semantics are recorded in `incompatible-changes.md`.
- F6: a brief comment on the test-scoped `seatunnel-engine-common` dependency in `pom.xml` explaining why production code references the class names as strings.
- F8: whether the lookup now handles chained relocations, or if you'd prefer to adjust the Javadoc wording to match the single-hop `getOrDefault` behavior.
If any of these were addressed in a later push, just point me at it and I'll re-review.
<!-- streview-comment:1738 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6266777395
The CI fact cited in my previous review has changed: Build for the current unchanged head is now passing. CI was previously the only remaining blocker; CI is now passing, so my technical conclusion changes to approve. I do not see a new source-side blocker from Daniel's review. This branch is behind the latest dev base, which I have recorded as merge metadata only; because the current head CI is green, that fact alone does not call for an additional source change. I have read-only review controls, so this COMMENT does not change GitHub's displayed review decision; a write-capable maintainer must perform any final approval or dismissal action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6282264699
Thanks for the update. With CI reported passing on `f6869ab09850` (run 38504983404), there is no build-side blocker from my side.
My earlier review points are still open, though. Here is what I'd still like to see before approval:
**Tests (`WALReaderAndWriterTest.java`)**
- The regression test only asserts `getClass()`/`toString()` on the replayed enum and covers one of the three legacy mappings. Please cover all three legacy names and assert on the deserialized value, not just the class.
- The legacy fixture is produced by the current `WALWriter`, so the test never reads a frame actually written by a release that used the old class names. A checked-in byte fixture or a hand-built frame with the old class name would make this a real compatibility test.
**Robustness (`WALReader.java`)**
- When `ClassNotFoundException` is thrown after the legacy-name rewrite, the error still reports only the original WAL class name. Please include the rewritten target name so a wrong or stale mapping is diagnosable, and consider whether a single bad frame should abort the whole cluster restart.
- The `JobResult` and `JobStatusData` target names are unverified string literals. Since `seatunnel-engine-common` is already a test-scoped dependency, a test that resolves every mapping target would catch typos.
- The single-hop `getOrDefault` lookup means a future relocation requires rewriting every existing target, which contradicts the "extend it" guidance in the Javadoc. Either resolve mappings transitively or adjust the Javadoc.
**Docs**
- Please add a short user-facing upgrade note for clusters using file-based IMap storage, and record the class relocation plus this shim's upgrade/downgrade semantics in `incompatible-changes.md`.
**Style (`pom.xml`)**
- A one-line comment on the test-scoped `seatunnel-engine-common` dependency explaining why production code deliberately references class names as strings would help future readers.
If you've decided against any of these, a short reply is fine and we can close them out. Happy to re-review once a new push lands.
<!-- streview-comment:1893 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6306793003
Thanks for the detailed mapping of the checklist against `f6869ab09850`. The description lines up with what I asked for on the test coverage (all three legacy names with value assertions), the `ClassNotFoundException` message showing both the recorded and resolved class names, the mapping-target test, the `pom.xml` comment, the single-level Javadoc on `LEGACY_CLASS_NAME_MAPPINGS`, and the `incompatible-changes.md` entries. I have not re-walked the diff for each of those yet, so I will confirm them against the actual changes rather than close them from the summary alone; if anything there is still in progress rather than already on this head, please say so.
One small correction on the mapping-target test: a comparison of string literals against `JobStatus.class.getName()` and friends catches a stale target at test time, not at compile time. That is still a good improvement over an unverified literal, I just want the description to be precise about it.
On the fixture point (the TODO acknowledging the frames are produced by the current `WALWriter`): I don't want to block on this, but before merge I'd like one of:
- a small checked-in WAL file (or hard-coded byte array) captured from a pre-#9689 build and replayed by a test; or
- if the argument is that `ProtoStuffSerializer`'s `RuntimeSchema` is field-number based and the class-name string is therefore the only thing that differs on the wire, please spell that out in the test comment and the PR description so the gap is explicitly justified rather than left as a TODO.
Your last comment looks cut off partway through item 1 of the "still open" list. If there were further open items, could you post them so I can review everything in one pass?
<!-- streview-comment:1911 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6160039440
Hi @SEZ9, both asks are done:
- **TODO note**: added on `f6869ab0` — a short comment on `LEGACY_JOB_STATUS_PATH` in `WALReaderAndWriterTest.java` stating that the fixtures are generated by the current `WALWriter` (only the recorded class-name strings are pre-#9689) and that replaying a frame actually written by a pre-relocation release is planned as a follow-up.
- **Follow-up issue**: will open it and link it back here once this lands.
On the earlier items, where things stand on the current head:
- **F1/F3**: `testReaderLoadsAllLegacyJobModelClassNames` writes all three legacy names and asserts on the deserialized values — the enum value via `name()` for `JobStatus`, `getStatus().name()` plus the error string for `JobResult`, and the class for `JobStatusData`. `testLegacyMappingsTargetCurrentClassNames` binds every `LEGACY_CLASS_NAME_MAPPINGS` target to the live class via `JobStatus.class.getName()` etc., so the string literals in `WALReader.java` are verified by a test rather than left unchecked.
- **F2**: `testUnknownClassNameSurfacesInExceptionMessage` replays a record whose value class is `org.apache.seatunnel.engine.core.job.NeverExistedJobStatus` and asserts the `IMapStorageException` message contains both the recorded name and the resolved name.
- **F4/F5**: both `incompatible-changes.md` (en/zh) carry the entry describing the #9689 relocation, the transparent rewrite on read, and the one-directional downgrade caveat.
- **F6**: the test-scoped `seatunnel-engine-common` dependency in `imap-storage-file/pom.xml` has the comment explaining why production code references the class names as strings.
- **F8**: `resolveClassName` is a single-hop `getOrDefault`, and the `LEGACY_CLASS_NAME_MAPPINGS` Javadoc documents resolution as single-level with one entry per legacy name, as we agreed.
These landed across `446ae128` → `9658666b`; the only newer commits are `dev` merges and the TODO note above. The full CI run on `6c771bdd` was green (24 passed, 6 skipped by path filters), and `f6869ab0` has just re-triggered the pipeline — it is a comment-only change. Thanks for the careful review, happy to get the final pass whenever you have a moment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6136697464
Thanks @zhangshenghang for the rundown of `99827177`.
- **F1/F3**: a test that binds every `LEGACY_CLASS_NAME_MAPPINGS` target to the live class via `JobStatus.class.getName()` etc. is exactly the compile-time guard I was after, and the three-mapping coverage with deserialized-value assertions from `452cd144` addresses the enum-only concern.
- **F2**: including both the original WAL name and `resolveClassName(className)` in the `IMapStorageException` message is sufficient for diagnosing a stale mapping target.
- **F4/F5**: the `incompatible-changes.md` (en/zh) entry with the one-directional downgrade semantics covers both the upgrade note and the incompatible-changes record.
- **F6**: the pom comment is what I asked for.
- **F8**: documenting resolution as single-level with one entry per intermediate name is fine with me; no need for multi-hop resolution.
One note on numbering: your nice-to-have list is offset from my finding IDs — your F3 covers my F4/F5, your F4 is my F6, and your F6 is my F8. I read your F5 ("follow-up if still wanted") as my F7 (reading a WAL frame actually written by a pre-relocation release rather than one produced by the current `WALWriter`). Can you confirm that's what you meant? If so, a follow-up is fine and it does not block this PR.
I'll go through the diff on `99827177` to confirm the above and then approve.
<!-- streview-comment:1702 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6310056067
Thanks @SEZ9, and sorry about the confusion. My previous comment was not cut off. It listed exactly three items under "Still open", and nothing else was pending:
1. The fixture is produced by the current `WALWriter` (acknowledged by the TODO at `WALReaderAndWriterTest.java:62-64`).
2. One bad frame aborts the whole restart (unchanged pre-existing fail-fast in `loadAllData`; I would keep it).
3. A Low nit about reflection and fully qualified names in `testLegacyMappingsTargetCurrentClassNames`.
If the comment rendered truncated on your side, those three are the complete list.
**Correction accepted on the mapping-target test.** "Fails the build" should have said "fails the unit-test job". `testLegacyMappingsTargetCurrentClassNames` compares the string values with `JobStatus.class.getName()` and friends, so a stale or mistyped target is caught when the test runs in CI, not at compile time. The strings stay literals on purpose, because production code must not depend on the old class locations.
**On the fixture, here is the evidence for your second option (justify the gap explicitly).** I compared the three relocated types before and after the move, using the parent of `e7de54e4e7a` (#9689, the commit that moved them out of `seatunnel-engine-core`) against current `dev`:
- `JobStatus`: the same 13 constants in the same order (`INITIALIZING` ... `UNKNOWABLE`), so the enum ordinals are unchanged.
- `JobResult`: the same fields in the same order, `status` and `error`.
- `JobStatusData`: the same fields in the same order and types, `jobId`, `jobName`, `jobStatus`, `submitTime`, `startTime`, `finishTime`.
- The only structural difference is the `private static final long serialVersionUID = 1L` that #11257 added later. `ProtoStuffSerializer` uses `RuntimeSchema`, which skips static and transient fields, so it does not change the bytes.
None of the three types has an `Object`-typed or abstract field, so no other class name is embedded in the value bytes either. That means a frame written by a pre-#9689 release differs from the frame the current `WALWriter` produces only in the `valueClassName` string of `IMapFileData`, and the legacy tests already set that string to the old name explicitly. This is a source-level comparison, not a replay of a real old WAL file. I did not run anything, and a captured byte fixture would still be the strongest proof if someone has an old WAL at hand.
So I would suggest @zhangshenghang replaces the TODO in `WALReaderAndWriterTest` with a short comment stating this argument (identical field layout and enum ordinals before and after #9689, class name being the only difference on the wire), and puts the same two sentences in the PR description. That satisfies your second option without a new fixture. If a checked-in fixture is easy to obtain, it is a welcome follow-up, but I would not hold the merge for it.
Nothing else has changed on my side: the head is still `f6869ab0`, the fork `Build` is green, and I have no source-level blocker. The PR still needs a write-capable maintainer to finish the approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6116687925
Quick follow-up on my earlier approval: the CI note in that review is now out of date. At that time the fork `Build` run was still in progress with one failure so far (`all-connectors-it-7 (8)`). On this same head (`452cd144`), the apache-side `Build`, `labeler` and `Notify test workflow` checks are now all `SUCCESS`, so that failure did not persist. My source-side conclusion is unchanged. The only remaining gate is the required approving review from a committer with write access.
Thanks again for the patience!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-5812713701
Thanks for the extra `+1`, @nzw921rx! Nothing new to add on my end — I re-verified this fix (and its `dev`-sync commit) from scratch across a few rounds, CI is green, and it's ready to merge as far as I can tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-5924175273
Thanks for the follow-up commit. I haven't been able to verify the new head (446ae12807) myself yet, so rather than guess at what changed since 835b264f3b, could you confirm the status of the earlier points?
**Error reporting (F2)**: Does the `ClassNotFoundException` path in `WALReader` now report both the recorded WAL class name and the resolved target name, so a wrong or stale mapping entry is diagnosable?
**Docs (F4, F5)**: If `incompatible-changes.md` (en and zh) is part of the change, does it record the #9689 relocation and the shim's upgrade/downgrade semantics for clusters using file-based IMap storage?
**Test coverage (F1, F3)**: Does the regression test now cover all three legacy names (`JobResult`, `JobStatus`, `JobStatusData`) and assert deserialized field values rather than only `getClass()`/`toString()`? Since the target names are string literals the compiler cannot check, a test that resolves every mapping target against the real class on the classpath would close F3.
**Legacy fixture (F7)**: Is the fixture still produced by the current `WALWriter`? A checked-in byte fixture, or at minimum a hand-built frame with the `org.apache.seatunnel.engine.core.job.*` name, would exercise what a pre-#9689 WAL actually contains. Not blocking if the frame format is unchanged, but please state that explicitly in the test.
**Multi-hop lookup (F8)**: If the lookup is still a single-hop `getOrDefault`, a future relocation would require rewriting every existing target. Either loop until the name is stable (with a bound) or adjust the Javadoc so the "extend it" guidance matches single-hop behavior.
**Pom comment (F6)**: A one-line comment on the test-scoped `seatunnel-engine-common` dependency explaining that production code intentionally references the class names as strings would help the next reader.
None of these change the approach; the shim at the single `deserializeData` choke point is the right fix. Once the above are confirmed or pushed, I'm happy to merge.
<!-- streview-comment:1447 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6305096513
Thanks @SEZ9 for the thorough checklist. I re-read it against the current head `f6869ab09850` (unchanged since my last review), and most of the items are already in place on this head. Mapping them so we can close what is closed and isolate what is genuinely left:
**Already addressed on this head**
- *Cover all three legacy names and assert on the deserialized value, not only the class:* `WALReaderAndWriterTest#testReaderLoadsAllLegacyJobModelClassNames` (L301) writes `JobStatus`, `JobResult` and `JobStatusData` frames with the old `org.apache.seatunnel.engine.core.job.*` names and asserts the replayed values (`status` name, `JobResult` status plus error text, and every `JobStatusData` field: id, name, status, submit/start/finish time). `testReaderLoadsJobStatusWrittenWithLegacyClassName` (L277) covers the single-enum case.
- *Show the rewritten target name when `ClassNotFoundException` happens:* `WALReader.deserializeData` (`WALReader.java:142-165`) now reports both the recorded and the resolved class name, and the stray `{}`/`s%` placeholders were fixed to `%s`. `testUnknownClassNameSurfacesInExceptionMessage` (L368) asserts both names are present in the message.
- *Resolve every mapping target:* `testLegacyMappingsTargetCurrentClassNames` (L259) compares each map value to `JobStatus.class.getName()`, `JobResult.class.getName()` and `JobStatusData.class.getName()`, so a renamed or mistyped target fails the build. This is why `seatunnel-engine-common` is a test-scoped dependency, and the `pom.xml` now carries a comment explaining that production code deliberately references the old names as strings.
- *Single-hop lookup vs. the Javadoc:* the Javadoc on `LEGACY_CLASS_NAME_MAPPINGS` (`WALReader.java:46-62`) now states resolution is single-level and that every intermediate name needs its own entry, so it no longer promises transitive resolution.
- *User-facing upgrade note:* both `incompatible-changes.md` files (en and zh) gained the "File-based IMap storage: legacy job model class names in WAL files are remapped on startup" entry, including the one-way semantics (a WAL written by a current build cannot be read by an older release, so a downgrade needs the persisted WAL cleared or rewritten).
**Still open, and my read on each**
1. *Fixture is produced by the current `WALWriter`.* This is acknowledged in the TODO at `WALReaderAndWriterTest.java:62-64`. One technical point that narrows the gap: `ProtoStuffSerializer` uses `RuntimeSchema` (field-number based), so the value/key bytes do not embed any class name. The only place the old name lives in a frame is the `valueClassName` string of `IMapFileData`, which these tests set to the old name explicitly. A frame from a pre-relocation release therefore differs from the test frame only if the field layout of `JobStatus`/`JobResult`/`JobStatusData` changed since then, and that is exactly what a checked-in byte fixture would prove. I agree a real fixture is the strongest evidence, but I would call it a recommended follow-up, not a merge blocker, because the failure mode it guards (a field-layout drift in these three classes) is independent of this shim.
2. *One bad frame aborts the whole restart.* That is unchanged pre-existing behavior (`ClassNotFoundException` is wrapped into `IMapStorageException` and propagates through `loadAllData`). I would keep it: silently skipping a job-status frame during restore would drop job state without a trace, which is worse than a clear fail-fast with both class names in the message.
3. *Nit (Low):* `testLegacyMappingsTargetCurrentClassNames` reads the private map through reflection and uses fully qualified `java.lang.reflect.Field` / `java.util.Map` in the method body. The other tests already prove the mapping through `loadAllData`, so this one is a duplicate guard; if you prefer, a package-private accessor or dropping the reflection would be cleaner. Not blocking.
On CI: the `Build` check on `f6869ab09850` is green (fork run 38504983404), and nothing from my side blocks. I only have read-level review rights here, so a write-capable maintainer needs to give the final approval. Thanks @zhangshenghang for all the iterations on this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][Zeta] Restore legacy job status WAL entries [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11781:
URL: https://github.com/apache/seatunnel/pull/11781#issuecomment-6117244001
Thanks @SEZ9 — addressed on head `99827177`:
**Item 1 (F1/F3)**: Added `testLegacyMappingsTargetCurrentClassNames`, which binds every `LEGACY_CLASS_NAME_MAPPINGS` target to the live `engine-common` class via compile-time references (`JobStatus.class.getName()` etc.), so a future rename of a target class now fails the build instead of silently breaking the shim. The three-name coverage with real deserialized-value assertions (`JobStatus` by enum name, `JobResult` by status/error fields, `JobStatusData` by all six fields) was already in place at `452cd144`.
**Item 2 (F2)**: Already satisfied on the current head — the `ClassNotFoundException` handler throws `IMapStorageException` with both the original name and `resolveClassName(className)` in the message.
**Nice-to-haves**: F3 — `incompatible-changes.md` (en/zh) already carries the #9689 relocation entry including the one-directional downgrade semantics. F4 — added the explanatory pom comment. F5 — will handle as a follow-up if still wanted. F6 — clarified the javadoc: resolution is single-level; chained relocations need an entry per intermediate name.
Spotless clean, `WALReaderAndWriterTest` 8/8 green locally. Ready for re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org