You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "ealeonraz (via GitHub)" <gi...@apache.org> on 2026/07/15 22:01:25 UTC

[PR] [Improve][Connector-V2] Migrate MQTT connector validation to declarative OptionRule [seatunnel]

ealeonraz opened a new pull request, #11461:
URL: https://github.com/apache/seatunnel/pull/11461

   ### Purpose of this pull request
   
   Part of #11007 (connector-mqtt slice). Migrates the MQTT connector's option validation from imperative `if/throw` checks to declarative `optionRule()` + `Conditions.*`, so invalid configs are caught at `--check`/submission time and constraints are exposed via option-rule metadata.
   
   Migrated rules:
   
   - **Source** (`MqttSourceFactory`): `qos` must be 0 or 1, `format` must be `json`/`text` (case-insensitive, via a shared `MqttFormatValidator` `ConditionExtension`), `reconnect_timeout` > 0, `max_queue_size` > 0, and `client_id` required **and** non-blank when `clean_session=false`. The client_id rule is expressed as a conditional requirement plus a conditional `notBlank` value constraint, because conditional value constraints alone are skipped when the option is absent — a single rule would silently accept `clean_session=false` without a `client_id`.
   - **Sink** (`MqttSinkFactory`): `qos` must be 0 or 1, `format` must be `json`/`text`, `batch_size` >= 1.
   
   Kept in runtime code (per the migration guide's "not suitable for declarative rules" section): broker connection error handling and the serialization-format dispatch `default` branch. No options were renamed and no defaults changed.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No config option changes. Invalid option values are now rejected during config validation (`OptionValidationException` at `--check`/submission time) instead of `IllegalArgumentException` at connector initialization.
   
   ### How was this patch tested?
   
   Existing imperative-validation unit tests were migrated to `ConfigValidator`-based tests against the factory option rules, plus new boundary cases (qos 0/1 accepted, batch_size 1 accepted, case-insensitive formats, and all three client_id scenarios: absent fails, blank fails, present passes).
   
   ```
   ./mvnw -pl seatunnel-connectors-v2/connector-mqtt test
   Tests run: 42, Failures: 0, Errors: 0, Skipped: 0
   ```
   
   `./mvnw spotless:apply` and `./mvnw -DskipTests verify` run clean on the module.
   
   ### Check list
   
   * [x] No new Jar binary package added
   * [x] No documentation update needed (option semantics unchanged; only the enforcement point moved)
   * [x] No incompatible change (`incompatible-changes.md` not affected)
   * [x] Not a new connector (plugin-mapping.properties / seatunnel-dist / labeler / e2e / plugin_config unchanged)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Migrate MQTT connector validation to declarative OptionRule [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11461:
URL: https://github.com/apache/seatunnel/pull/11461#issuecomment-5192171661

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Migrate MQTT connector validation to declarative OptionRule [seatunnel]

Posted by "ealeonraz (via GitHub)" <gi...@apache.org>.
ealeonraz commented on PR #11461:
URL: https://github.com/apache/seatunnel/pull/11461#issuecomment-4995705697

   @DanielLeens Done on both counts: Actions enabled in the fork, and the branch is rebased onto the latest `dev` (`3656ba8`) — the PR is now a single commit.
   
   Refreshed run: https://github.com/ealeonraz/seatunnel/actions/runs/29519036639
   
   Everything is green except the `unit-test` job, which fails in a test unrelated to this PR: `MetricsApiTest.metricsApiTest` in `seatunnel-engine-server` (metrics endpoint returns 500; the log shows `HazelcastInstanceNotActiveException: State: SHUT_DOWN` surfacing through `ExceptionHandlingFilter` — the request hits a stale instance on the hard-coded port 8080).
   
   Evidence it's independent of this change:
   - `seatunnel-engine-server` test scope doesn't include `connector-mqtt` (only connector-fake/console/file-local), so this diff isn't on that test's classpath
   - The upstream nightly Schedule Backend on clean `dev` at the same commit fails the identical test: https://github.com/apache/seatunnel/actions/runs/29515660951 (`unit-test (11, ubuntu-latest)`)
   - It failed consistently across JDK 8/11 ubuntu cells in three runs here (Windows cells skip it via `@DisabledOnOs`), including a failed-jobs rerun
   
   I didn't touch the engine test in this PR to keep the scope to the connector-mqtt slice, but I'm happy to file a separate issue for `MetricsApiTest` (and take a crack at a fix) if that's useful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Migrate MQTT connector validation to declarative OptionRule [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11461:
URL: https://github.com/apache/seatunnel/pull/11461#issuecomment-4990864837

   @ealeonraz Thank you for your contribution! This improvement further enhances the configuration verification system of SeaTunnel, allowing configuration issues to be exposed earlier and more clearly, which is very helpful in improving overall usability and maintainability.🚀
   
   pls enable CI: https://github.com/apache/seatunnel/pull/11461/checks?check_run_id=87482860873 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Migrate MQTT connector validation to declarative OptionRule [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11461:
URL: https://github.com/apache/seatunnel/pull/11461#issuecomment-6367058871

   I prepared a minimal patch for the serialization-test blocker on current head `8a9c8bf4f83e9af3f30b9f2573cc7cdd86d862c3`.
   
   The [current-head JDK 11 Ubuntu job](https://github.com/ealeonraz/seatunnel/actions/runs/38245466425/job/114800838331) reports `Expected java.lang.IllegalArgumentException to be thrown, but nothing was thrown` in `MqttSinkTest.testSinkConfigValuesSurviveJavaSerialization:93`.
   
   The test added by #12455 observes whether a non-default configuration value survives Java serialization. This PR moves QoS validation into the factory, so `qos=2` no longer triggers the writer-constructor exception used by that test. The patch uses `format=xml` instead: the writer still rejects that format before connecting to MQTT. It preserves the exception assertion and strengthens the message assertion to an exact match. No production code or test coverage is removed.
   
   Validation: module-scoped `./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-mqtt -nsu -Dmaven.gitcommitid.skip=true`, `git diff --check`, and patch application against the current-head index all passed. Compilation and tests have not been run for this patch; a new GitHub CI run is still required.
   
   I cannot push this directly: GitHub rejected a dry-run push to the source branch with `Permission to ealeonraz/seatunnel.git denied to DanielLeens` (HTTP 403). The branch owner or a maintainer with write access can apply this diff and push it normally:
   
   ```diff
   diff --git a/seatunnel-connectors-v2/connector-mqtt/src/test/java/org/apache/seatunnel/connectors/seatunnel/mqtt/sink/MqttSinkTest.java b/seatunnel-connectors-v2/connector-mqtt/src/test/java/org/apache/seatunnel/connectors/seatunnel/mqtt/sink/MqttSinkTest.java
   index f1443e4c79f..3ae7ddd773d 100644
   --- a/seatunnel-connectors-v2/connector-mqtt/src/test/java/org/apache/seatunnel/connectors/seatunnel/mqtt/sink/MqttSinkTest.java
   +++ b/seatunnel-connectors-v2/connector-mqtt/src/test/java/org/apache/seatunnel/connectors/seatunnel/mqtt/sink/MqttSinkTest.java
   @@ -77,13 +77,12 @@ class MqttSinkTest {
    
        @Test
        void testSinkConfigValuesSurviveJavaSerialization() {
   -        // qos 2 is not a legal MQTT QoS for this connector and MqttSinkWriter rejects it, but
   -        // MqttSink itself does not validate, so the sink builds fine and only a writer surfaces
   -        // it. That makes it an observable, non-default value: MqttSinkOptions.QOS defaults to 1,
   -        // so a pluginConfig that came back empty would yield a legal qos and the writer would
   -        // build without complaint. The rejection below only happens if the 2 travelled.
   +        // Unsupported formats are still rejected when the writer creates its serializer, before
   +        // connecting to MQTT. Unlike qos validation, this check remains in the writer after the
   +        // OptionRule migration. The default format is json, so this rejection proves that the
   +        // non-default xml value survived serialization.
            Map<String, Object> config = baseConfig();
   -        config.put("qos", 2);
   +        config.put("format", "xml");
            MqttSink sink = newSink(config);
    
            MqttSink restored = SerializationUtils.deserialize(SerializationUtils.serialize(sink));
   @@ -93,9 +92,7 @@ class MqttSinkTest {
                        Assertions.assertThrows(
                                IllegalArgumentException.class,
                                () -> restored.createWriter(Mockito.mock(SinkWriter.Context.class)));
   -            Assertions.assertTrue(
   -                    exception.getMessage().contains("got: 2"),
   -                    "expected the restored qos in the message, got: " + exception.getMessage());
   +            Assertions.assertEquals("Unsupported MQTT sink format: xml", exception.getMessage());
            }
        }
    
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org