You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "srijan-singh (via GitHub)" <gi...@apache.org> on 2026/06/27 08:28:46 UTC

[PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

srijan-singh opened a new pull request, #11198:
URL: https://github.com/apache/seatunnel/pull/11198

   ### Purpose of this pull request
   https://github.com/apache/seatunnel/issues/10443#issuecomment-4796095891
   
   This pull request adds a new **Couchbase sink connector** for Apache SeaTunnel (V2 API).
   
   The connector writes `SeaTunnelRow` data as JSON documents into a Couchbase bucket/scope/collection
   using the Couchbase Java SDK 3.5.1. It supports:
   
   - **Upsert mode** — insert or replace existing documents via `upsert-enable`
   - **Composite document keys** — built from configurable `primary-key` field list (joined with `_`);
     falls back to a random UUID when not set
   - **Batch flushing** — buffers rows in memory and flushes on size threshold (`buffer-flush.max-rows`)
     or time threshold (`buffer-flush.interval`)
   - **Retry with backoff** — transient write failures are retried up to `retry.max` times with
     `retry.interval` ms delay (multiplied by attempt number)
   - **Full type mapping** — all SeaTunnel primitive types mapped to Couchbase JSON equivalents;
     `BYTES` stored as Base64 string
   
   ---
   
   ### Does this PR introduce _any_ user-facing change?
   
   **Yes.** This is a new connector. Users can now configure a `Couchbase` sink in their SeaTunnel jobs:
   
   ```hocon
   sink {
     Couchbase {
       connection.string = "couchbase://localhost"
       username          = "Administrator"
       password          = "password"
       bucket            = "my_bucket"
       collection        = "my_collection"
       primary-key       = ["id"]
       upsert-enable     = true
     }
   }
   ```
   
   There is no change to any existing connector or API — this is purely additive.
   
   ---
   
   ### How was this patch tested?
   
   **1. Live Docker integration test**
   
   A standalone integration test was run against a real `couchbase/server:community-7.1.1` Docker
   container, validating all core write behaviors:
   
   | Test | Scenario | Result |
   |------|----------|--------|
   | [1] | Plain document insert with UUID key | ✅ PASS |
   | [2] | Upsert — writing same key twice replaces the document | ✅ PASS |
   | [3] | Composite primary key: `user_id + order_id` joined as `u1_o42` | ✅ PASS |
   | [4] | N1QL `COUNT(*)` query returns expected document count after writes | ✅ PASS |
   
   **2. E2E test module**
   
   Added `CouchbaseIT` under
   `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-couchbase-e2e/`.
   
   The test uses Testcontainers to start a real Couchbase container, bootstraps the cluster fully
   via REST (node config → services → credentials → bucket → collection → index storage mode),
   executes a `FakeSource → Couchbase` sink job, then verifies document count via N1QL.
   
   ---
   
   ### Check list
   
   * [x] No new Jar binary packages added without license notice — Couchbase Java SDK
     (`com.couchbase.client:java-client:3.5.1`) is Apache 2.0 licensed
   * [x] Documentation updated: `docs/en/connectors/sink/Couchbase.md` added with full options
     table, data type mapping, and task examples
   * [ ] `incompatible-changes.md` — not applicable (new connector, no breaking changes)
   * [x] [`plugin-mapping.properties`](plugin-mapping.properties) updated:
     `seatunnel.sink.Couchbase = connector-couchbase`
   * [x] [`seatunnel-dist/pom.xml`](seatunnel-dist/pom.xml) updated:
     `connector-couchbase` dependency added
   * [x] CI label added in
     [`.github/workflows/labeler/label-scope-conf.yml`](.github/workflows/labeler/label-scope-conf.yml):
     `couchbase:` block
   * [x] E2E test added:
     [`seatunnel-e2e/seatunnel-connector-v2-e2e/connector-couchbase-e2e/`](seatunnel-e2e/seatunnel-connector-v2-e2e/connector-couchbase-e2e/)
   * [x] [`config/plugin_config`](config/plugin_config) updated: `connector-couchbase` listed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4863418454

   I tried fixing `Build website` stage by installing docusaurus but it's creating another issue, any advise?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4880231894

   > I tried fixing `Build website` stage by installing docusaurus but it's creating another issue, any advise?
   
   The issue with this website has been resolved in the website repository. Simply retrigger the failed task in the CI pipeline. I just manually synced the upstream branch, which will also trigger the full build process.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5174445191

   Thanks for splitting the CI change out into #11636 — that is exactly the right direction, and I appreciate the quick turnaround.
   
   Two things to make this PR ready for the next round:
   
   1. The blocker on this head is still that the shared `.github/workflows/backend.yml` edits are bundled inside this connector PR. Now that #11636 exists to carry the workflow discussion, could you push an update here that reverts the `backend.yml` changes from this branch, so the PR stays scoped to the Couchbase connector? Once that commit is up, I will re-review the full head from scratch.
   
   2. One honest heads-up on #11636 so it does not become a surprise later: in my review over there, the `-DargLine` heap setting as currently written is not consumed by the root backend test path, so as-is it will not deliver the intended memory increase — it needs to be wired through the root Surefire/Failsafe configuration first. So I would not treat #11636 as a guaranteed fix for the `all-connectors-it-3` cancellation yet; keeping that work out of this connector PR is still correct either way.
   
   The connector-side source points look addressed on the current head, so this is genuinely close. Happy to do the full re-review as soon as the descoped commit lands.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4883158707

   Test failure
   
   CI Job | kafka-connector-it | all-connectors-it-7
   -- | -- | --
   Failing test | KafkaIT.testKafkaToKafkaExactlyOnceOnStreaming | CouchbaseIT.startUp
   Error type | ConditionTimeoutException — assertion never true in 5 min | ContainerLaunchException — container crashed on startup
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4924192839

   Nice to see the current `Build` lane go green on this head. From Daniel's side I do not see a reopened source-level blocker on the current code. The only thing I would still keep an eye on is that this branch is now behind the latest `dev`, so if GitHub asks for an update branch step before merge, please sync the latest `dev` and rerun once on top of that updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5150590978

   Thanks for checking this. I looked at the fork-side Build run for the latest head `fbf38c13`, and the failing lane is `Run / unit-test (8, ubuntu-latest)`:
   
   https://github.com/srijan-singh/seatunnel/actions/runs/30686518877/job/91333572273
   
   The failure is in `seatunnel-ci-tools` during Surefire test execution:
   
   ```text
   There was an error in the forked process
   GC overhead limit exceeded
   ...
   Failed to execute goal ... maven-surefire-plugin ... on project seatunnel-ci-tools
   ```
   
   So this is not a Couchbase connector compile error. It is the test JVM fork running out of heap while executing the CI tools unit tests.
   
   A small CI-only fix would be reasonable here. Please try adding an explicit Surefire JVM heap for this job, because the current `MAVEN_OPTS=-Xmx4096m` mainly configures the Maven process, while this error happens in the forked Surefire process. For example, in `.github/workflows/backend.yml`, update the `unit-test` command to pass an explicit argLine:
   
   ```bash
   ./mvnw -B -T 1 clean verify \
     -DskipUT=false \
     -DskipIT=true \
     -D"license.skipAddThirdParty"=true \
     -D"skip.ui"=true \
     -DargLine="-Xmx2048m -Dfile.encoding=UTF-8" \
     --no-snapshot-updates
   ```
   
   Please keep this as a CI-only change. Do not change the Couchbase connector code for this OOM. After pushing the workflow update, the fork Build should rerun and we can check whether the `unit-test (8, ubuntu-latest)` lane passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4976869933

   The last build was successful, need approval with write access to merge it. Moreover we can cleanly merge it even the branch is out-dated from base.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5162368765

   **CI/CD issue:** `all-connectors-it-3 (8, ubuntu-latest)` and `all-connectors-it-3 (11, ubuntu-latest)` is getting auto cancelled as it exceeds the limit of 3h 30m to complete the test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4916110971

   Finally build was successful :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5200325262

   @DanielLeens the analysis on shard was correct moving couchbase module to the bottom resolved the timeout issue - we just need @davidzollo approval on the latest commit for merge.
   
   I've also updated the [PR for ci-cd](https://github.com/apache/seatunnel/pull/11636), please let me know if it would be a good addition.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5252600216

   @DanielLeens the build passed, we need maintainer approval to merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5151192132

   I checked the new run after `7dd25753`:
   
   https://github.com/srijan-singh/seatunnel/actions/runs/30693530415/job/91352609088
   
   The failure moved to `Run / unit-test (8, windows-latest)`, but it is still the same root cause:
   
   ```text
   project seatunnel-ci-tools
   There was an error in the forked process
   GC overhead limit exceeded
   ```
   
   The latest workflow change did not apply the fix in the best place. It currently sets:
   
   ```yaml
   MAVEN_OPTS: -Xmx2g -Dfile.encoding=UTF-8
   JAVA_TOOL_OPTIONS: -Xmx2g -XX:MaxMetaspaceSize=512m
   ```
   
   `JAVA_TOOL_OPTIONS` is too broad because it affects every Java process in the job, and `2g` is still not enough for this `seatunnel-ci-tools` Surefire fork. Also, lowering `MAVEN_OPTS` from the previous `4g` to `2g` is not helpful here.
   
   Please update the `unit-test` command itself and pass heap to Surefire via Maven `argLine`, then remove `JAVA_TOOL_OPTIONS`. For example:
   
   ```yaml
   - name: run all modules unit test
     run: |
       ./mvnw -B -T 1 clean verify \
         -DskipUT=false \
         -DskipIT=true \
         -D"license.skipAddThirdParty"=true \
         -D"skip.ui"=true \
         -DargLine="-Xmx3072m -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8" \
         --no-snapshot-updates
     env:
       MAVEN_OPTS: -Xmx4096m -Dfile.encoding=UTF-8
   ```
   
   This keeps the change CI-only and targets the Surefire test JVM directly. Please do not change the Couchbase connector code for this failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4845336326

   Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
   
   - base branch: `dev`
   - head SHA: `cbf6a9827f6d903d084e6d1f88400f950b1d7973`
   - compare status: `diverged`
   - ahead_by: `6`
   - behind_by: `23`
   - mergeable_state: `unknown`
   - CI / merge gate snapshot: `failing` (failing: `Build [failure]`)
   
   Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4899183765

   Thanks for the update. I checked the current head against the latest `dev` again.
   
   - base branch: `dev`
   - head SHA: `d814777f4056434b7a31f4de032e89fb1a82e6b4`
   - compare status: `diverged`
   - ahead_by: `23`
   - behind_by: `8`
   - current failing signal: `Build`
   
   On this round, the visible failing CI signal still does not look directly tied to the current PR-owned diff from Daniel's side, and the branch is still behind upstream. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If the same lane still fails on the updated head afterwards, please drop the new failing check link here and I can help narrow it down against the rebased revision.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4968425597

   Rechecked the same unchanged head `dfac80dfa4b9` after the CI / merge-gate fact changed.
   
   ## Runtime path rechecked
   ```text
   Job config with sink { Couchbase { ... } }
     -> CouchbaseSinkFactory.optionRule()
         -> requires connection.string / username / password / bucket / collection
         -> optional scope / flush / retry / upsert / primary-key
     -> CouchbaseSinkFactory.createSink(context)
         -> validates upsert-enable requires primary-key
         -> builds CouchbaseWriterOptions
         -> stamps CatalogTable identifier as Couchbase bucket/collection
     -> CouchbaseSink.createWriter(context)
         -> CouchbaseWriter(options, catalogTable, context)
             -> validates configured primary-key fields exist in SeaTunnelRowType
             -> Cluster.connect(...), waitUntilReady(bucket), scope(...).collection(...)
     -> CouchbaseWriter.write(row)
         -> skips UPDATE_BEFORE, rejects DELETE explicitly
         -> converts SeaTunnelRow fields to JsonObject
         -> builds stable document key from primary-key, or UUID when no key is configured
         -> buffers and flushes on size / timer / prepareCommit / close
     -> doFlush()
         -> materializes stable (docId, doc) units before retry
         -> insert/upsert with retry cursor and ambiguous-timeout handling
         -> clears buffer only after full success
   ```
   
   ## Merge conclusion
   ### Conclusion: source blocker cleared from Daniel's side; waiting for the current gate
   
   1. Blocking items
   - None from Daniel's source review on this unchanged head.
   
   2. Current gate
   - The current remaining gate is still pending: `PENDING gate: Build` (https://github.com/apache/seatunnel/runs/86998960454)
   - Daniel only has `READ` permission on this repository, so a write-capable maintainer may still need to handle the final approval / dismissal step once the gate is clear.
   
   I do not see a reopened code-side blocker from Daniel on the current head.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5192183084

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5242315156

   Thanks @davidzollo for the approval, and thanks for sticking with this one @srijan-singh.
   
   Re-checked the live state on the current head (`391b54ffe89d`) just now: `Build` is still **FAILURE**, so this isn't merge-ready yet — CI hasn't actually turned green.
   
   Tracing the outer Apache "Build" pointer to the real fork run, two jobs are failing:
   - `all-connectors-it-2 (8, ubuntu-latest)` — job step "run connector-v2 integration test (part-2)"; the check-run annotations point to a Spark-engine job execution failure ("SeaTunnel job executed failed" / "Run SeaTunnel on spark failed") inside that aggregated multi-connector partition, not anything Couchbase-specific.
   - `kafka-connector-it (11, ubuntu-latest)` — job step "run kafka connector integration test", exit code 1.
   
   On `kafka-connector-it`: this PR has hit this exact lane before (see the 2026-07-04 comment thread), and the failing test that round was `KafkaIT.testKafkaToKafkaExactlyOnceOnStreaming` with a `ConditionTimeoutException` — a known-flaky Kafka IT, unrelated to the Couchbase sink code. Given the pattern, my working assumption is this is the same pre-existing flake rather than a regression from the current head, but I haven't pulled today's raw job log far enough to confirm the specific failing test name on this run, so I'd treat that as likely rather than confirmed.
   
   One thing worth a deliberate double-check rather than assuming pure flake: this PR also modifies the shared `seatunnel-e2e-common` test container (`SeaTunnelContainer.java`'s thread-leak exemption logic, plus the new `SeaTunnelContainerThreadExemptionTest`), which is a transitive test dependency of essentially every connector-v2 E2E/IT module — including `kafka-connector-it`. I already reviewed that change on its own terms in my last pass and didn't flag it, but given it's a shared file and the failure is in exactly that class of module, it's worth a quick sanity check that the thread-check assertion isn't the actual cause on this run before writing it off as unrelated flakiness.
   
   To be clear on my own standing here: my last full review found no remaining source-side blockers (only the three non-blocking Low/Medium items already on record), so from my side this is purely a CI-gate question now, not a code question. I only have comment-level access, so I can't formally move this to approved myself — I'd support merging once `Build` is actually green on this head and davidzollo's approval still applies to that green run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5255178264

   Yes, but PR conflicts now, then we can merge it 
   
   <img width="1212" height="142" alt="image" src="https://github.com/user-attachments/assets/dee12f2e-f164-432d-a560-4c65735587b2" />
   
   
   > @DanielLeens the build passed, we need maintainer approval to merge.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5004931749

   @davidzollo could you please review it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5175174906

   > [!NOTE]
   Build is expected to fail as mentioned in https://github.com/apache/seatunnel/pull/11198#issuecomment-5162368765, once https://github.com/apache/seatunnel/pull/11636 is merged, it should fix the build issue.
   
   I've reverted ci-cd changes, current scope is limited to couchbase.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5167004342

   Raised PR with CI-CD fix
   - https://github.com/apache/seatunnel/pull/11636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "danielnadean (via GitHub)" <gi...@apache.org>.
danielnadean commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5154655767

   I tracked the current head `7a0a56183562eebe12b89277f416a31dac06d291` and the previous `target` exclusion is still present, so this is not the earlier fix being overwritten.
   
   The current failure is still `seatunnel-ci-tools` -> `ImportClassCheckTest` in `unit-test (8, ubuntu-latest)` from fork run `srijan-singh/seatunnel/actions/runs/30726060311`, failing with `java.lang.OutOfMemoryError: GC overhead limit exceeded`.
   
   The root cause is that the test still parses checked-in generated protobuf sources and also retains all import AST nodes for the whole repository, which is enough to trip the JDK 8 unit-test lane even after excluding `target` outputs.
   
   I prepared a one-commit patch PR against the contributor branch here: https://github.com/srijan-singh/seatunnel/pull/1
   
   It keeps the existing `target` exclusion, skips protobuf-generated sources detected by `@@protoc_insertion_point`, and stores only lightweight import metadata instead of full JavaParser import nodes. Merging that patch into `couchbase-sink-v1` (or cherry-picking `56c1eed88d`) should unblock the OOM in this lane.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on code in PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#discussion_r3522380655


##########
.github/workflows/backend.yml:
##########
@@ -384,6 +384,7 @@ jobs:
           cd seatunnel-website
           npm set strict-ssl false
           npm install
+          npm install @docusaurus/plugin-client-redirects@2.4.3

Review Comment:
   ```suggestion
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5230722222

   Thanks @srijan-singh — the descoped commit is exactly what I asked for. With the `backend.yml` changes reverted and the branch limited to the Couchbase connector, this is back in a reviewable state, and I'll do the full re-review of the latest head from scratch as promised.
   
   A few things on the current status:
   
   1. **Shard ordering fix** — glad the analysis held up and moving the Couchbase module in the shard ordering resolved the `all-connectors-it-3` timeout on this branch. That's a scoped, connector-adjacent change, so it's fine to keep here.
   2. **#11636** — I'll leave detailed feedback on that PR directly. My earlier note still stands: the `-DargLine` heap setting needs to be wired through the root Surefire/Failsafe config to actually take effect, so please don't treat it as a guaranteed fix until that's addressed there.
   3. **Merge approval** — correct that final approval/merge needs a committer with write access; @davidzollo has already chimed in, so once my re-review passes and CI is green on this head, we should be in good shape.
   
   Concrete asks before merge:
   - Please sync with the latest `dev` and rerun CI so we have a clean, current run on the final head (and confirm there's no residual `backend.yml` diff in the branch).
   - If the shared CI flakiness (`ryuk` pull failure / `DatabendCDCSinkIT`) reappears, just flag it — those are tracked separately in #11624 and won't be held against this PR.
   
   Also, your last comment ("@DanielLeens there's a…") looks like it got cut off — could you repost the full message so I don't miss anything?
   
   Thanks for the persistence on this one — it's genuinely close.
   
   <!-- streview-comment:111 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5157375790

   I checked the latest failing fork run (`30734891079`). The remaining failures are not caused by the Couchbase sink changes in this PR:
   
   1. `connector-assert-e2e` failed before the tests started because Testcontainers could not pull `testcontainers/ryuk:0.3.4` from Docker Hub on the hosted runner (`job 91461994549`).
   2. `DatabendCDCSinkIT` is still flaky on `Flink 1.15.3`: the CDC job exited with code `0`, but `sink_table` stayed empty until the Awaitility timeout (`job 91461994456`).
   
   I opened a separate upstream fix PR for these shared CI failures: #11624. Once that lands, please sync with the latest `dev` and rerun the checks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5264960196

   Conflict resolved and build passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4949102494

   Quick CI follow-up on the `Build` check I previously described as queued.
   
   That check has now completed as failed. I traced the outer Apache check to the fork workflow run and the failing lane is:
   
   ```text
   Build -> all-connectors-it-1 (Java 8) -> connector-http-e2e
   HttpIT.testSourceToAssertSink:371 expected <0> but was <1>
   ```
   
   The failing module is `connector-http-e2e`, not the Couchbase sink module touched by this PR. I also checked the current base/head relation: this branch is not behind `dev` (`compare status: ahead`, `behind_by=0`), so syncing latest `dev` would not bring in a newer upstream fix for this particular run.
   
   From Daniel's source-review side, I still do not see this CI failure as a new Couchbase code blocker. The lowest-cost next step is to rerun the `Build` check, or specifically rerun the failed `all-connectors-it-1` lane. If the same HTTP E2E assertion fails again on the rerun, we should treat that as a separate CI / HTTP E2E stability issue rather than patching unrelated Couchbase code in this PR.
   
   So the current state is: source-level blocker remains cleared from my side; merge is still waiting on the required CI / review gate to go green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4944509845

   Thanks @SEZ9. I rechecked the same head `c15514f59f49` against your follow-up review and re-walked the current flush path in `CouchbaseWriter`.
   
   On the specific data-loss point, I do not see a reopened source blocker from that path on the current revision:
   ```text
   write()
     -> buffer rows under synchronized guard
   prepareCommit()
     -> doFlush()
   close()
     -> cancel timer
     -> shutdown scheduler
     -> doFlush()
   doFlush()
     -> synchronized
     -> clear buffer only after the write loop completes successfully
   ```
   So the current head already does tie the buffered path to both `prepareCommit()` and `close()`, and the timer path does not drop buffered rows on a failed periodic flush because `buffer.clear()` only happens on success.
   
   I do think your docs and backoff follow-ups are reasonable, but on this unchanged head I would still keep Daniel's source-level conclusion as cleared.
   
   The CI fact did change since my last comment, though: the `Build` run I referenced earlier is now cancelled rather than queued:
   https://github.com/apache/seatunnel/runs/86400709410
   
   So the current Daniel-side conclusion for this same head is:
   - no reopened source-side blocker from Daniel on the current revision
   - your documentation and contract notes are useful follow-ups, but I am not reopening them as blockers on this unchanged head
   - the remaining gate is getting a fresh Build result on top of the current head
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-4885777494

   Rechecked the current head again.
   
   I still do not see a reopened Couchbase source-side blocker from Daniel's side on the current revision, but the branch is now `ahead_by=22`, `behind_by=12`, compare status `diverged`, and the current red `Build` is failing in broad unrelated lanes such as `kafka-connector-it` and `all-connectors-it-7`:
   https://github.com/srijan-singh/seatunnel/actions/runs/28714428477
   
   Those failures are outside the Couchbase sink path I rechecked here, so the lowest-cost next step is to sync the latest `dev` and rerun CI first. I would avoid trying to patch those broad CI signals by adding unrelated changes in this PR. If the updated head is still red after the sync, please share the new failing check link and I can help narrow it down further.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5154735697

   正在修复这个问题


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "srijan-singh (via GitHub)" <gi...@apache.org>.
srijan-singh commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5217421654

   @DanielLeens there's a recurring issue in CI [unit-test (11, windows-latest)](https://github.com/srijan-singh/seatunnel/actions/runs/31151661801/job/92867849175), it's not related to couchbase.
   
   ```shell
   java.lang.RuntimeException: java.io.IOException: boom-before-schema-entry
       at ...SchemaChangeBroadcastTest.java:412  ← coordinator.close()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Add Couchbase Sink Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11198:
URL: https://github.com/apache/seatunnel/pull/11198#issuecomment-5242358773

   Thanks @davidzollo for the approval, and thanks for sticking with this one @srijan-singh.
   
   Re-checked the live state on the current head (`391b54ffe89d`) just now: `Build` is still **FAILURE**, so this isn't merge-ready yet — CI hasn't actually turned green.
   
   Tracing the outer Apache "Build" pointer to the real fork run, two jobs are failing:
   - `all-connectors-it-2 (8, ubuntu-latest)` — job step "run connector-v2 integration test (part-2)"; the check-run annotations point to a Spark-engine job execution failure ("SeaTunnel job executed failed" / "Run SeaTunnel on spark failed") inside that aggregated multi-connector partition, not anything Couchbase-specific.
   - `kafka-connector-it (11, ubuntu-latest)` — job step "run kafka connector integration test", exit code 1.
   
   On `kafka-connector-it`: this PR has hit this exact lane before (see the 2026-07-04 comment thread), and the failing test that round was `KafkaIT.testKafkaToKafkaExactlyOnceOnStreaming` with a `ConditionTimeoutException` — a known-flaky Kafka IT, unrelated to the Couchbase sink code. Given the pattern, my working assumption is this is the same pre-existing flake rather than a regression from the current head, but I haven't pulled today's raw job log far enough to confirm the specific failing test name on this run, so I'd treat that as likely rather than confirmed.
   
   One thing worth a deliberate double-check rather than assuming pure flake: this PR also modifies the shared `seatunnel-e2e-common` test container (`SeaTunnelContainer.java`'s thread-leak exemption logic, plus the new `SeaTunnelContainerThreadExemptionTest`), which is a transitive test dependency of essentially every connector-v2 E2E/IT module — including `kafka-connector-it`. I already reviewed that change on its own terms in my last pass and didn't flag it, but given it's a shared file and the failure is in exactly that class of module, it's worth a quick sanity check that the thread-check assertion isn't the actual cause on this run before writing it off as unrelated flakiness.
   
   To be clear on my own standing here: my last full review found no remaining source-side blockers (only the three non-blocking Low/Medium items already on record), so from my side this is purely a CI-gate question now, not a code question. I only have comment-level access, so I can't formally move this to approved myself — I'd support merging once `Build` is actually green on this head and davidzollo's approval still applies to that green run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org