You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/07/23 02:09:31 UTC

[PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

goutamadwant opened a new pull request, #11538:
URL: https://github.com/apache/seatunnel/pull/11538

   ### Purpose of this pull request
   
   Closes #9764.
   
   This draft adds the local-only first slice of Zeta debug mode as `--dry-run sample`.
   
   - runs the configured source and transform chain with a bounded row sample
   - prints source and transform schemas and sample rows
   - replaces configured sinks before sink factory creation and save-mode handling
   - disables checkpoints and uses source parallelism 1 for deterministic per-source limits
   - rejects cluster, async, restore, and savepoint execution for sample mode
   - keeps sink DDL preview out of this slice until a no-write preview contract is agreed
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes.
   
   Users can run a local sample preview with:
   
   ```shell
   bin/seatunnel.sh --master local --config config/v2.batch.config.template --dry-run sample --sample-limit 10
   ```
   
   The command reads up to the configured number of rows from each source and runs the transform chain without creating the configured sink or writing to the target system. Source reads are real and may access external systems.
   
   ### How was this patch tested?
   
   Added coverage for:
   
   - CLI parsing, defaults, validation, and local execution
   - replacement of configured sinks before sink runtime creation
   - bounded source collection and normal completion
   - bounded output reporting across a multi-transform chain
   - a local FakeSource job confirming that the configured InMemory sink is not used
   
   Verified with:
   
   ```shell
   ./mvnw -pl seatunnel-core/seatunnel-starter -Dtest=ClientCommandArgsTest test
   ./mvnw -pl seatunnel-engine/seatunnel-engine-client -Dtest=MultipleTableJobConfigParserTest test
   ./mvnw -pl seatunnel-engine/seatunnel-engine-server -Dtest=SeaTunnelSourceCollectorDryRunSampleTest,TransformFlowLifeCycleDryRunSampleTest test
   ./mvnw -pl seatunnel-engine/seatunnel-engine-common,seatunnel-engine/seatunnel-engine-core,seatunnel-engine/seatunnel-engine-server,seatunnel-engine/seatunnel-engine-client,seatunnel-core/seatunnel-starter -DskipTests package
   ```
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   
   No new binary dependency, incompatible change, or connector plugin change is included.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5230720145

   Thanks @goutamadwant for the thorough follow-ups — this is shaping up well.
   
   Confirming the points that now look closed to me:
   
   - **Parallelism**: forcing source and transform parallelism to 1 in `MultipleTableJobConfigParser`, with the two-source parallelism-3 regression test, resolves my determinism concern about multiple sample collectors.
   - **CLI contract**: moving sample mode onto the validated CLI path (instead of env options), rejecting `--restore-with-checkpoint` and other conflicting options, and making row output opt-in matches what I asked for.
   - **Semantics**: the clarified `--sample-limit` wording ("limits rows forwarded into the sample pipeline; a source may finish its active poll/batch before stopping") is honest and useful — good to have that documented rather than implied.
   - **Schema logging**: logging only the table path and physical row schema, with the password-option regression test proving `CatalogTable` options are excluded, addresses the sensitive-data-in-logs concern.
   
   Remaining asks before I approve:
   
   1. **CI / merge gate**: I agree the Java 8 RocketMQ (20 vs 25 messages) and Doris (`UnknownHostException: doris_e2e`) failures look like unrelated E2E flakiness, especially with the Java 11 twins and both engine V2 integration jobs green. That said, the branch is still diverged from `dev` — please rebase/merge latest `dev` and rerun the failing jobs so we have a green (or clearly-flaky-only) signal on the merged base.
   2. **Minor cleanups from my earlier pass** — please confirm whether these landed in the recent commits, as I didn't see them called out explicitly:
      - the stale en `Usage` block relative to the new `--dry-run` description,
      - the duplicated `Common.setDeployMode` call,
      - `ParameterException` thrown from `execute()` outside parse time,
      - Javadoc on the new public members.
   
   If those four are done (or you point me to the commits), I'll do a final head-to-head diff read after the CI rerun. Thanks again for the disciplined, test-backed iterations!
   
   <!-- streview-comment:98 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5109087165

   @DanielLeens Addressed the review points in the latest commit.
   
   Sample mode now comes from the validated CLI path instead of user controlled env options. I also added validation for conflicting options and sample limits, made row output opt in, removed the shared static state test, and updated the English and Chinese docs.
   
   The focused CLI, parser, collector and transform tests pass locally. The related 55 module build also passes. Let me know. thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5316846835

   Thanks for the continued follow-through here, @goutamadwant. Before we can merge, could you confirm the current status of the required Build check on the latest head? If CI isn't green yet, syncing with dev and rerunning the checks would be the next step. There are no remaining source-level blockers from my side, so once CI passes this should be good to go.
   
   <!-- streview-comment:167 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5082320307

   thanks @DanielLeens 
   
   Sample mode already forces action parallelism to 1 in MultipleTableJobConfigParser so configured source parallelism cannot create multiple sample collectors
   
   I added a regression test with two sources configured at parallelism 3 and verified both source actions and the preview sink use parallelism 1
   
   I also documented this behavior so the sample limit is clearly source-wide and the preview output stays deterministic
   
   Rechecked the parser CLI collector and transform tests locally. Let me know. thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5322752860

   @davidzollo I see Daniel's approval and builds are green. Can you review too and could we merge this if all looks good? I dont see any outstanding fix comments. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5231608473

   @/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-nervous-swirles-887e0f/f978b90c-4e26-438b-9ad4-2fbf1f56ae20/scratchpad/pr-batch-20260809/post_bodies/reply_11538.txt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5234113970

   Addressed the remaining review points and pushed the update.\n\nInvalid sample option combinations now go through the standard JCommander error message usage and exit code path. The validation also runs before deploy mode is written to global state.\n\nAdded the missing DryRunSampleConfig getter Javadocs and regression coverage for the CLI validation path.\n\nVerified with the focused starter tests on JDK 11. ClientCommandArgsTest and SeaTunnelClientOOMTest pass with 24 tests. The branch is also synced with current dev and a fresh Build is queued.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5224721278

   @DanielLeens fixed the schema logging issue in the latest commit
   
   transform sample logs now include only the table path and physical row schema so CatalogTable options are not written to persistent logs and also added a regression test with a password option and verified that the schema remains visible while the option name and value are excluded
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5322429728

   Hi @SEZ9 — confirming: the required `Build` check on the latest head (`e88fdb90`) finished with **SUCCESS** (https://github.com/apache/seatunnel/runs/94934440297), and I already reflected that in a review update this morning, moving to **Approve**: https://github.com/apache/seatunnel/pull/11538#pullrequestreview-4948882257
   
   So CI is green and there are no outstanding source-level blockers from my side either — every prior-round item (parallelism/determinism, CLI-path validation, the schema-logging leak, the uncaught-`ParameterException` UX gap, Javadoc) has been independently re-verified as fixed on this head.
   
   The PR's `mergeStateStatus` is still showing `BLOCKED`, but that looks like the branch-protection review-count gate rather than a CI or code issue — should be ready for a maintainer to merge once that's satisfied.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5234151809

   @SEZ9 @DanielLeens I checked the four cleanup points against the latest head
   
   The English usage section already lists sample mode and both sample options and there is only one Common.setDeployMode call
   
   I fixed the two remaining areas. Sample option validation now runs through the CommandLineUtils parse callback so invalid combinations use the normal message usage and exit code path. It also runs before Common.setDeployMode so rejected commands do not modify global state
   
   Added Javadocs for isEnabled getLimit and isPrintData and added regression coverage for the CLI validation path
   
   The branch is synced with current dev. ClientCommandArgsTest and SeaTunnelClientOOMTest pass with 24 tests on JDK 11
   
   Could you please recheck the latest head? thanjks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5141522326

   Thanks @SEZ9 for the fresh pass. I read this as consistent with the current Daniel-side source conclusion: the earlier source blockers look closed on the unchanged head.
   
   The remaining blocker I still see is the CI / merge gate. This branch is diverged from `dev`, and the current failing `Build` signal still looks unrelated to the sample dry-run source diff. I would keep the next step focused: sync with the latest `dev` and rerun the failing check first, rather than adding unrelated fixes in this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5161669515

   Hi @DanielLeens , I checked the failed build on the latest commit. The failures are from the Java 8 RocketMQ and Doris E2E jobs. RocketMQ restore expected 20 messages but received 25, and Doris failed with UnknownHostException: doris_e2e / No route to host. The corresponding Java 11 jobs passed, along with both engine V2 integration jobs and the full unit test matrix. These connectors are not touched by this PR, so this looks like flaky E2E infrastructure. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5212355785

   @DanielLeens addressed the remaining sample mode contract points in the latest commit.
   
   Sample mode now rejects `--restore-with-checkpoint` and forces configured transform parallelism to `1`. I also clarified that `--sample-limit` limits rows forwarded into the sample pipeline, while a source may finish its active poll or batch before stopping.
   
   Added focused CLI and parser coverage. `ClientCommandArgsTest` passes 22 tests and `MultipleTableJobConfigParserTest` passes 15 tests locally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5249116609

   Synced the branch with latest dev without changing the sample dry-run implementation. ClientCommandArgsTest and SeaTunnelClientOOMTest pass locally with all 24 tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5340995858

   Hi @goutamadwant — confirming from my side: my review already landed as an approval yesterday (https://github.com/apache/seatunnel/pull/11538#pullrequestreview-4948882257) after I verified the required `Build` check passed on the current head (`e88fdb90`) and re-checked every item from the earlier rounds — parallelism/determinism, the CLI-path validation, the schema-logging leak, the uncaught-`ParameterException` UX gap, and the Javadoc gaps — as resolved.
   
   I don't hold write-level review permissions on this repository, so my approval alone doesn't satisfy the branch-protection review-count gate that's currently keeping `mergeStateStatus` at `BLOCKED`. That last step needs a maintainer with write access (cc @davidzollo) to complete the review/merge action. Nothing outstanding from my side — thanks again for the careful, test-backed iteration on this feature!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add local sample dry-run mode [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11538:
URL: https://github.com/apache/seatunnel/pull/11538#issuecomment-5366240739

   Thanks @goutamadwant — confirming your read of the current state is correct.
   
   - CI: the required `Build` check on the latest head (`e88fdb90`) finished with **SUCCESS** (run 94934440297), and @DanielLeens's review landed as **Approve** (review 4948882257).
   - Code: no outstanding source-level blockers. As noted in the review, every prior-round finding (env-option smuggling, `--restore-with-checkpoint` bypass, transform-parallelism escape, the credential-leaking schema log, the uncaught-`ParameterException` UX gap) was re-verified as fixed on this head.
   - Merge gate: the remaining `BLOCKED` status is the branch-protection review-count requirement, not anything on your side — since @DanielLeens's approval doesn't carry write-level permissions, a maintainer review is the correct next step.
   
   Two concrete asks so the non-blocking items from the review don't get lost after merge:
   
   1. Please open a follow-up issue for the documented idle/absolute timeout for sample mode, and link it back to this PR.
   2. Please open a second follow-up (or fold it into the same issue if you prefer) for restoring the end-to-end coverage.
   
   Once those trackers exist, nothing else is needed from you. Thanks for the disciplined follow-through across the review rounds — this is in good shape.
   
   <!-- streview-comment:414 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org