You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "rucciva (via GitHub)" <gi...@apache.org> on 2026/07/15 15:04:40 UTC

[PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

rucciva opened a new pull request, #11460:
URL: https://github.com/apache/seatunnel/pull/11460

   [Feature] [Connector-V2] Add NATS JetStream sink connector
   
   ### Purpose of this pull request
   
   This pull request implements the sink portion of #10101 by adding a publish-only NATS JetStream Connector V2 sink.
   
   The connector supports:
   
   - JSON and native message formats
   - NATS username/password and token authentication
   - Configurable subjects
   - Configurable native field mappings for message ID, subject, headers, and payload
   - Optional SeaTunnel row-kind headers
   - Configuration and schema validation
   - At-least-once writer semantics without checkpoint state or transactional commit support
   - Pre-provisioned JetStream streams
   
   The NATS source connector and stream-management functionality are intentionally out of scope.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. This PR adds a new `NatsJetStream` sink connector and its documentation in English and Chinese.
   
   Example configuration:
   
   ```hocon
   sink {
     NatsJetStream {
       url = "nats://localhost:4222"
       subject = "events"
       format = "json"
     }
   }
   ```
   
   The connector also supports native messages with configurable field mappings:
   
   ```hocon
   sink {
     NatsJetStream {
       url = "nats://localhost:4222"
       format = "native"
       subject = "events"
       native_format_fields {
         id = "message_id"
         subject = "event_subject"
         headers = "attributes"
         data = "payload"
       }
     }
   }
   ```
   
   This is a new connector and does not change the behavior of existing connectors or configurations.
   
   ### How was this patch tested?
   
   Added unit tests covering:
   
   - Sink factory creation and option rules
   - Authentication validation
   - JSON and native format validation
   - Native field mapping validation
   - Subject fallback and resolution
   - Message ID and header serialization
   - Payload type validation
   - Row-kind header behavior
   - Invalid record handling
   
   Executed successfully with Java 11:
   
   ```bash
   ./mvnw -B \
     -pl seatunnel-connectors-v2/connector-nats-jetstream \
     -Dtest='*NatsJetStream*Test' test
   ```
   
   Result: 31 tests passed, 0 failures, 0 errors.
   
   Executed the NATS JetStream E2E test suite with Testcontainers and NATS `2.10.14-alpine`:
   
   ```bash
   ./mvnw -B \
     -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-nats-jetstream-e2e \
     -am \
     -DskipUT \
     -DskipIT=false \
     -Dit.test=NatsJetStreamIT \
     -DfailIfNoTests=false \
     verify
   ```
   
   Result: 21 tests passed, 0 failures, 0 errors.
   
   Also verified:
   
   - Java 11 Spotless formatting
   - Connector module compilation and verification
   - Distribution packaging
   - Connector registration checks
   - CI module detection for the connector and E2E module
   - No whitespace errors
   
   The repository-wide license check was attempted but could not complete because of the unrelated unresolved dependency `connector-http-airtable:3.0.0-SNAPSHOT`.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License Notice according to the [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [x] If you are contributing the connector code, please check that the following files are updated:
     1. Updated `plugin-mapping.properties` with the NATS JetStream connector.
     2. Updated `seatunnel-dist/pom.xml`.
     3. Added the NATS JetStream CI label scope in `label-scope-conf.yml`.
     4. Added the NATS JetStream E2E module and test cases.
     5. Updated `config/plugin_config`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5058232584

   Hei @DanielLeens , whats the next step for this to be accepted?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5199184265

   Hey @davidzollo does this mean a discussion is required on slack before this PR can be merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5225622186

   hey @davidzollo @DanielLeens , fixed the failing build


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5015144236

   hey @DanielLeens , addressed you review, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5015619513

   addressed the review, please recheck, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5192172914

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-4998777847

   hei @DanielLeens , thanks for the review, i've tried to address your comment, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5254695652

   > Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   > 
   > Please ignore the following message if you have joined in the Slack~
   > 
   > ```
   > To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   > Slack workspace link: https://s.apache.org/seatunnel-slack
   > 
   > This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   > ```
   
   Welcome to join the Slack for daily communication ^_^


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5200400167

   Hi @rucciva, thanks for the question — no, joining the Slack workspace is not a merge requirement. That part of the welcome message is an optional, general invitation to the #contributors channel (it even says "please ignore the following message if you have joined in Slack"), not a review gate.
   
   The current status on this PR is unchanged from my last review: the required `Build` check on the current head is still failing (https://github.com/apache/seatunnel/runs/88586024405), and the review decision is pending an approving review (`REVIEW_REQUIRED`, merge state `BLOCKED`). On the source side I do not have an outstanding blocker — once the Build check is green and a maintainer approval lands, this can proceed toward merge. Thanks for your patience and for the solid work on the connector!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "rucciva (via GitHub)" <gi...@apache.org>.
rucciva commented on PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460#issuecomment-5032097670

   hei @DanielLeens , #11502 is closed and i have resynced, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add NATS JetStream sink connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11460:
URL: https://github.com/apache/seatunnel/pull/11460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org