You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "chl-wxp (via GitHub)" <gi...@apache.org> on 2026/01/14 02:48:20 UTC
[PR] [Feature][seatunnel-api]Add enhanced configuration validator [seatunnel]
chl-wxp opened a new pull request, #10337:
URL: https://github.com/apache/seatunnel/pull/10337
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][seatunnel-api]Add enhanced configuration validator [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #10337:
URL: https://github.com/apache/seatunnel/pull/10337#issuecomment-3755074462
<!-- code-pr-reviewer -->
**JdbcSink.java:1-340** — The class is missing an override of `enhancedConfigurationValidator()`, so the `JdbcSinkEnhancedValidator` implemented in this PR will never be invoked.
The framework calls `sink.enhancedConfigurationValidator()` at `MultipleTableJobConfigParser.java:702`, and without the override it returns `Optional.empty()`, skipping all validation. This renders the new validator dead code and leaves config errors (e.g., enabling `use_copy_statement` on MySQL) undetected until runtime.
**ElasticSearchCatalog.java:108-110** — `getServiceVersion()` creates a new `EsRestClient` when `esRestClient == null`, while `DefaultEnhancedConfigurationValidator.detectCurrentServiceVersion()` calls `catalog.open()` afterward in a try-with-resources block.
This can create two client instances and leak the first one. Consider adding a state check:
```java
if (esRestClient == null) {
throw new IllegalStateException("Catalog must be opened before calling getServiceVersion()");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][seatunnel-api]Add enhanced configuration validator [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed pull request #10337: [Feature][seatunnel-api]Add enhanced configuration validator
URL: https://github.com/apache/seatunnel/pull/10337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][seatunnel-api]Add enhanced configuration validator [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #10337:
URL: https://github.com/apache/seatunnel/pull/10337#issuecomment-4278479727
Thanks for the work on this. I reviewed the latest PR locally from branch `seatunnel-review-10337` at commit `e5748a872a7a7982780d61d8c1032bd30a39a166`, against `upstream/dev` with merge base `81d873301d5d0bad16a3ef1680a3172d0968b8d9`.
The direction is good: a connector-level enhanced configuration validator is a better long-term model than putting every cross-option or version-specific rule into `OptionRule`. I also confirmed that the earlier concern about `JdbcSink` not exposing the validator has been addressed in the latest code: `JdbcSink.enhancedConfigurationValidator()` now returns `JdbcSinkEnhancedValidator`.
## What this PR solves
- User pain point: some invalid connector configurations can only fail at runtime today, such as `use_copy_statement=true` with a non-PostgreSQL JDBC URL, or Elasticsearch vector options against an unsupported service version.
- Fix approach: the PR adds `EnhancedConfigurationValidator`, structured validation issue types, default source/sink/catalog extension points, and connector-specific validators for JDBC and Elasticsearch.
- In one sentence: this PR is a good foundation for pre-flight connector configuration checks, but it currently only works on the Zeta parse path and has current `dev` conflicts.
Simple example: a JDBC sink with `url=jdbc:mysql://...` and `use_copy_statement=true` is rejected early when submitted through Zeta, but the same config still bypasses the enhanced validator when submitted through the Flink or Spark starters.
## Complete runtime path
```text
Zeta job submission
-> ClientJobExecutionEnvironment / RestJobExecutionEnvironment
-> new MultipleTableJobConfigParser(...).parse(...)
-> parseSource()
-> FactoryUtil.createAndPrepareSource(...) [FactoryUtil.java:80-164]
-> source.enhancedConfigurationValidator() [MultipleTableJobConfigParser.java:411]
-> validate(...)
-> EnhancedConfigurationValidator.validate(...)
-> log WARNING / ERROR issues
-> throw JobDefineCheckException if any ERROR exists [MultipleTableJobConfigParser.java:844-867]
-> createSinkAction()
-> FactoryUtil.createAndPrepareSink(...) [FactoryUtil.java:196-247]
-> sink.enhancedConfigurationValidator() [MultipleTableJobConfigParser.java:702]
-> validate(...)
-> handleSaveMode(...)
Flink job submission
-> SourceExecuteProcessor.execute()
-> FactoryUtil.createAndPrepareSource(...) [Flink SourceExecuteProcessor.java:146-155]
-> setJobContext / ensureJobModeMatch
-> no enhancedConfigurationValidator call
-> SinkExecuteProcessor / AbstractSinkExecuteProcessor
-> FactoryUtil.createAndPrepareSink(...) [Flink SinkExecuteProcessor.java:132-139]
-> setJobContext / handleSaveMode
-> no enhancedConfigurationValidator call
Spark job submission
-> SourceExecuteProcessor.execute()
-> FactoryUtil.createAndPrepareSource(...) [Spark SourceExecuteProcessor.java:134-140]
-> setJobContext / ensureJobModeMatch
-> no enhancedConfigurationValidator call
-> SinkExecuteProcessor.execute()
-> FactoryUtil.createAndPrepareSink(...) [Spark SinkExecuteProcessor.java:132-139]
-> setJobContext / handleSaveMode
-> no enhancedConfigurationValidator call
Elasticsearch version validation
-> ElasticsearchSink.enhancedConfigurationValidator() [ElasticsearchSink.java:104-105]
-> getCatalog() creates a new Catalog [ElasticsearchSink.java:108-118]
-> DefaultEnhancedConfigurationValidator.validateVersionCompatibilityRules()
-> if rules are non-empty, detectCurrentServiceVersion() runs first [DefaultEnhancedConfigurationValidator.java:124-130]
-> catalog.open() connects to ES and reads cluster info [DefaultEnhancedConfigurationValidator.java:166-173]
-> only after that does validateCompatibility() check whether the option is configured [DefaultEnhancedConfigurationValidator.java:138-145]
```
## Findings
Issue 1: the PR currently has real merge conflicts with `dev`
- Location: `seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java:84`; `seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:411`
- Severity: High
- Description: GitHub reports `mergeable=false` / `mergeable_state=dirty`. I reproduced this locally in a temporary worktree with `git merge --no-commit --no-ff seatunnel-review-10337`; the merge produced content conflicts in `CommonErrorCode.java` and `MultipleTableJobConfigParser.java`. Current `dev` already uses `COMMON-38` for `VALIDATION_FAILED`, while this PR also adds `SEATUNNEL_TASK_CONFIG_ERROR("COMMON-38", ...)`. The parser has also changed on `dev` around datasource/metalake/envOptions handling.
- Risk: this cannot be merged as-is, and the historical green Build from January 15 does not validate the current merged result.
- Suggested fix: rebase onto the latest `dev`, resolve the parser conflict while preserving the current datasource/metalake/envOptions logic, and avoid reusing `COMMON-38`. If the new error code is not actually used, removing it may be better than allocating a new one.
Issue 2: enhanced validation is wired only into Zeta, while Flink/Spark bypass it
- Location: `seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:411`; `seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:702`; `seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SourceExecuteProcessor.java:146`; `seatunnel-core/seatunnel-flink-starter/seatunnel-flink-13-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java:132`; `seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SourceExecuteProcessor.java:134`; `seatunnel-core/seatunnel-spark-starter/seatunnel-spark-2-starter/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java:132`
- Severity: High
- Description: this PR adds API-level default methods on `SeaTunnelSource` and `SeaTunnelSink`, but the framework invokes them only from Zeta’s `MultipleTableJobConfigParser`. Flink and Spark create the same connectors through `FactoryUtil.createAndPrepareSource/Sink`, then continue without calling `enhancedConfigurationValidator()`.
- Risk: the same SeaTunnel job config has different behavior depending on the engine. For example, JDBC `use_copy_statement=true` with a MySQL URL is rejected in Zeta but still reaches runtime in Flink/Spark. Connector authors may also assume implementing the new API makes validation engine-wide, which is not true in the current patch.
- Suggested fix: preferably move the validation call into the shared connector creation path, such as `FactoryUtil.createAndPrepareSource/Sink`, with an API/common-level exception for ERROR issues. If that boundary is too broad for this PR, extract a shared helper and call it from Zeta, Flink, and Spark source/sink processors. Please also add at least one integration-style test proving that the same invalid JDBC config is rejected across the relevant engine paths.
Issue 3: Elasticsearch version validation opens the catalog even when no version-gated option is configured
- Location: `seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/DefaultEnhancedConfigurationValidator.java:124`; `seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/DefaultEnhancedConfigurationValidator.java:130`; `seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/sink/ElasticsearchSinkEnhancedValidator.java:38`
- Severity: Medium
- Description: `validateVersionCompatibilityRules()` calls `detectCurrentServiceVersion()` as soon as the validator has any version rules. `ElasticsearchSinkEnhancedValidator` always returns rules for `vectorization_fields` and `vector_dimensions`, so even a normal ES sink that does not configure vector features will open an ES catalog and request cluster info during parsing.
- Risk: all ES sink jobs get an unnecessary network call during job definition, and multi-table jobs can repeat it. If ES is temporarily unreachable, users can see a warning about version detection even when they did not use any version-sensitive option.
- Suggested fix: first filter the version rules to options that are actually present in `ReadonlyConfig`; only call `detectCurrentServiceVersion()` when at least one configured option needs version validation. Caching the detected version per validator would also avoid repeated catalog opens.
## Compatibility
The new source/sink/catalog methods are default methods, so existing connector implementations remain source/binary compatible. Behavior-wise, Zeta will now reject some configurations at submission time, which is a reasonable improvement. The problem is that Flink/Spark do not get the same behavior yet, so the user-facing compatibility story is currently split by engine.
## Tests and docs
The unit tests for validator rules and the JDBC/Elasticsearch validators are useful. What is still missing is framework integration coverage:
- Zeta parser test proving ERROR issues stop job definition.
- Flink/Spark starter coverage or shared `FactoryUtil` coverage proving enhanced validation is not Zeta-only.
- Elasticsearch test proving service-version detection is not triggered when no version-gated option is configured.
Docs are not strictly required for the framework API itself, but if JDBC behavior changes from runtime failure to submission-time failure, the JDBC sink docs should mention that `use_copy_statement` is PostgreSQL/Greenplum-family only and keep docs/en and docs/zh aligned.
## Merge conclusion
### Conclusion: can be merged after fixes
1. Blocking items that must be fixed
- Issue 1: resolve the current `dev` merge conflicts and rerun CI on the rebased result.
- Issue 2: make enhanced validation execute consistently for Zeta, Flink, and Spark, or centralize it in a shared connector creation path.
2. Suggested non-blocking improvements
- Issue 3: avoid Elasticsearch service-version detection unless the user actually configured a version-gated option.
- Add concise user-facing docs for JDBC validation behavior if the PR changes when/where invalid configs fail.
Overall, this is a promising mechanism and the shape is close to what SeaTunnel needs. The key merge blocker is coverage: an API-level validator should not be Zeta-only. Once the PR is rebased, the validation entry point is made engine-consistent, and CI is green, this will be much safer to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][seatunnel-api]Add enhanced configuration validator [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #10337:
URL: https://github.com/apache/seatunnel/pull/10337#issuecomment-5083456851
Hi @chl-wxp, thank you for the thoughtful work on richer configuration validation. Aggregating multiple errors and expressing cross-option constraints are important usability improvements, and this proposal helped make those needs visible early.
The core goals have since been implemented on the current architecture through #10763 and #10977, including dry-run validation, shared `OptionRule` support, cross-field conditions, and multi-error reporting from the common factory path. Continuing this older Zeta-specific branch would duplicate those capabilities and conflict with the validation framework now in `dev`.
We are closing this PR as superseded, not as a rejection of the underlying idea. Your early design work contributed useful context to the direction the project ultimately adopted. Thank you for investing in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org