You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/06/09 12:54:42 UTC
[PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
davidzollo opened a new pull request, #11029:
URL: https://github.com/apache/seatunnel/pull/11029
## What's changed
- add a new `connector-cdc-pg-base` module to centralize the shared PostgreSQL-compatible Debezium property assembly and row deserialization logic
- migrate the Postgres CDC source config and incremental source to the shared PG base implementation without changing the existing connector identifier, slot/plugin options, or replica-identity behavior
- wire OpenGauss CDC through its own source entry point so it can reuse the PG base behavior while keeping the current OpenGauss plugin identity
- add a regression test to keep Postgres table identifier normalization (`schema.table` and `db.schema.table`) backward compatible
## Why
- Postgres and OpenGauss currently duplicate the same PG-compatible CDC assembly logic, which makes follow-up maintenance and future connector additions much harder
- this change keeps the refactor in a phase-1 shape: common config/deserialization logic moves to the PG base module first, while dialect-specific runtime behavior stays in the existing connector modules to reduce migration risk
## Validation
- `./mvnw -f seatunnel-connectors-v2/connector-cdc/pom.xml -pl connector-cdc-pg-base,connector-cdc-postgres,connector-cdc-opengauss -am spotless:apply`
- `./mvnw -f seatunnel-connectors-v2/connector-cdc/pom.xml -pl connector-cdc-pg-base,connector-cdc-postgres,connector-cdc-opengauss -am -Dtest=PostgresSourceConfigFactoryTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test`
- `./mvnw -f seatunnel-e2e/seatunnel-connector-v2-e2e/pom.xml -pl connector-cdc-postgres-e2e -am -Dtest=PostgresCDCIT#testDialectCheckDisabledCDCTable -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test`
## Notes
- a broader Postgres/OpenGauss CDC e2e matrix was also attempted locally, but it still hits existing environment-level noise outside this refactor path (for example Flink container startup instability in the PR matrix). This PR is intentionally opened as draft so we can keep tightening the wider validation path in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5409366423
Thanks for the thorough verification, @DanielLeens — that's exactly the confirmation I was hoping for.
On Issue 1: your merge-base diff matches my read. Since `SNAPSHOT_ONLY` is handled entirely by the shared incremental framework and not by anything Postgres-dialect-specific, my preference is to simply add it back to `OpengaussSourceOptions.STARTUP_MODE` rather than document it away — dropping a working mode with no replacement isn't something we should ship, and restoring it also resolves the compat break cleanly. Agreed this remains a blocker, and agreed your earlier approval is superseded until it's addressed.
On Issue 3: agreed on both the diagnosis and the severity. It's safe today only by accident (stateless static constant), so hardening via a constructor argument or a `requireNonNull` plus a Javadoc warning on the hook is the right shape — non-blocking, but worth doing while this code is being touched.
To be clear on status: nothing has changed on head `075001b0b5a9`, so all points from my review remain open. Concrete asks for the next push:
1. **Blocking:** restore `SNAPSHOT_ONLY` in the OpenGauss `startup.mode` choice set (Issue 1), and update the en/zh Opengauss-CDC docs to state the valid values (Issue 6).
2. Add an actionable error message and an explicit upgrade note for the restored `require-replica-identity-full` enforcement (Issues 2 and 5).
3. Confirm or address the serialized-DAG compatibility question around the new `PgBaseIncrementalSource` hierarchy (Issue 4).
4. Minor: fix the raw `(SingleChoiceOption)` cast (Issue 7) and the "Apache OpenGauss" Javadoc wording (Issue 8).
Happy to re-review together with you once a new revision is pushed.
<!-- streview-comment:530 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-4968424072
Rechecked the same unchanged head `cd09a2938e32` after the CI / merge-gate fact changed.
## Runtime path rechecked
```text
PG-base source initialization
-> PgBaseSourceConfigFactory.create(...)
-> build common Debezium properties
-> leave connector-specific identifiers to subclasses
-> PgBaseIncrementalSource.createDebeziumDeserializationSchema(...)
-> loadTableChanges()
-> discoverDataCollections(...)
-> serialize table changes for the row deserializer
concrete connectors
-> PostgresSourceConfigFactory / PostgresIncrementalSource
-> OpengaussIncrementalSourceFactory / OpengaussIncrementalSource
-> keep connector-specific plugin names and dialect behavior
```
## Merge conclusion
### Conclusion: source blocker cleared from Daniel's side
1. Blocking items
- None from Daniel's source review on this unchanged head.
2. Current gate
- The current CI signal is now green: https://github.com/apache/seatunnel/actions/runs/28321236971/job/83903431386
- Daniel only has `READ` permission on this repository, so GitHub may still show earlier review state and a write-capable maintainer may still need to handle the final approval / dismissal step.
I do not see a reopened code-side blocker from Daniel on the current head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5190579127
Thanks @SEZ9 for taking a look as well.
The head is still `f3d46a0718e8`, the same commit I did a full re-review of yesterday, so there's no new code to re-review today. On your Issue 1 (possible public API change in `PostgresSourceConfigFactory`): my own pass on this exact head went through that file line by line against the deleted code and the one real behavioral delta I found is the `require-replica-identity-full` wiring move into `createDataSourceDialect()` (write-up in my "Issue 1" and section 1.2 above) — that's a genuine, disclosed compatibility note, not a silent signature break, and I didn't find another removed/changed public or protected method in that class on this head. If you're seeing a different signature change than the one I traced, a `path:line` pointer would help me re-check it specifically.
Either way, my own conclusion from yesterday stands unchanged: this is "ready to merge after fixes," blocked on (1) adding the small regression test for the replica-identity wiring plus a one-paragraph disclosure in the PR description, and (2) getting the currently-red `Build` check green (single failing Windows unit-test lane that looks unrelated to this diff based on timing and the fact that 69 other jobs across both JDKs are green). Once those two are closed I don't have a further source-side blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-4667940167
Addressed the OpenGauss follow-up in 8491dfce21 by adding a dedicated regression test for the new entry path. The new test covers both the factory source class contract and the Opengauss-CDC plugin name contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx merged PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5412546290
Thanks @SEZ9 - quick confirmation pass on top of your reply, since nothing changed on my end from my Aug 24 review.
**Head check:** confirmed still `075001b0b5a9366da74bcc37f1b16b08e08d21f7`, matches what you noted.
**Issue 1 (restore `SNAPSHOT_ONLY`):** I re-traced this independently again before replying, since your ask ("restore" vs "document away") is a real fork in the fix. `OpengaussSourceOptions.STARTUP_MODE` (`OpengaussSourceOptions.java:43-55`, current head) narrows the choice set to `INITIAL, EARLIEST, LATEST`, while `PostgresSourceOptions.STARTUP_MODE` (`PostgresSourceOptions.java:29-43`) keeps `INITIAL, SNAPSHOT_ONLY, COMMITTED_OFFSET, EARLIEST, LATEST`. I also checked whether `SNAPSHOT_ONLY` genuinely depends on Postgres-only behavior, since the class's own Javadoc claims both dropped modes are "backed by PostgreSQL-specific behavior." That claim doesn't hold up under tracing: `SnapshotOnlySplitAssigner` lives entirely in the dialect-agnostic `connector-cdc-base` module, and `IncrementalSource.java:324-331` gates it purely on `StartupMode`, with no call into any Postgres-specific dialect method anywhere in that path. `COMMITTED_OFFSET` is the one that's genuinely Postgres-specifi
c (it reads `pg_replication_slots`); `SNAPSHOT_ONLY` is not. So I agree with your preference: restoring `SNAPSHOT_ONLY` in the OpenGauss choice set is the correct fix, not documenting the drop away. This remains a blocker.
**Issue 3:** agreed, still non-blocking, same assessment as before.
**One item not yet called out explicitly in this thread:** I checked the live merge status directly (`gh pr view --json mergeable,mergeStateStatus`), and this branch currently has a real merge conflict against `dev` (`mergeable: CONFLICTING`, `mergeStateStatus: DIRTY`), separate from the ordinary "behind dev" staleness we've both flagged before as a CI-noise issue. That will need a rebase/conflict resolution alongside the code fixes before this can go green again.
Status from my side is unchanged: waiting on a revision that (a) restores `SNAPSHOT_ONLY` for OpenGauss and updates the en/zh docs, and (b) resolves the merge conflict against `dev`. I'll do a full re-review once that lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5579562787
Resolved the merge conflict against `dev` and pushed `f561cad5e13`. This one needed real reconciliation, not a mechanical merge, so here's the full trail:
**What collided**: `dev` had independently merged apache/seatunnel#11922 ("Support PostgreSQL CDC ADD COLUMN schema evolution") directly onto the pre-refactor `PostgresIncrementalSource`/`PostgresSourceConfigFactory` — adding `include.schema.changes` wiring, a `PostgresRelationSchemaChangeResolver`, `createDebeziumDeserializationSchema` overrides for RELATION-message schema tracking, and `SupportSchemaEvolution` + `ADD_COLUMN` support — while this branch had already refactored those same two classes onto the new `PgBaseIncrementalSource`/`PgBaseSourceConfigFactory` template-method base this PR introduces. Both sides touched the same methods for different reasons.
**How I reconciled it, method by method:**
- `PgBaseIncrementalSource.createDebeziumDeserializationSchema` already covers everything dev's version did *except* the two PostgreSQL-specific calls, `setSchemaChangeResolver(new PostgresRelationSchemaChangeResolver())` and `setSchemaChangeEventFilter(...)`. Rather than duplicate the whole method, `PostgresIncrementalSource` now overrides it to call the inherited `loadTableChanges()` (which reuses the already-constructed dialect, unlike dev's version which built a redundant second one) and adds just those two Postgres-specific builder calls on top — kept out of the shared base since OpenGauss isn't guaranteed to share PostgreSQL's RELATION-message wire format.
- `PostgresIncrementalSource` now also `implements SupportSchemaEvolution` (in addition to `SupportParallelism`, inherited from the base) and carries dev's `supports()` (`ADD_COLUMN`) and `validateSchemaEvolutionOptions` (schema evolution requires `pgoutput`) unchanged.
- For `PostgresSourceConfigFactory`, I checked `PgBaseSourceConfigFactory.create()` line by line against dev's inline version: `table.include.list` parsing, `dbzProperties` passthrough, `snapshot.mode` handling and `enableConcurrentRead` propagation were all already identically present in the base (the comment on the concurrent-read line literally reads "Keep the concurrent-read flag wired after moving concrete connectors onto PG-base" — this was anticipated). The one real gap: the base hardcoded `include.schema.changes` to `false` instead of the actual `schema-changes.enabled` value, which would have silently broken the whole ADD_COLUMN feature for every PG-base connector, not just Postgres. Fixed at the base-class level (so OpenGauss gets it too) and kept dev's explicit ordering — applied after the `dbzProperties` merge, not before, so SeaTunnel's own option stays authoritative over a raw passthrough that happens to set the same key.
- `PostgresSourceConfigFactory`'s own conflict resolved to keeping this branch's template-method overrides (`driverClassName`, `configureConnectorProperties`, `createSourceConfig`) as-is, since the base now fully subsumes what dev's inline version did.
Everything dev's `PostgresDialect.java`, `PostgresSourceFetchTaskContext.java`, `PostgresRelationSchemaChangeResolver.java`, `RelationAwarePostgresSchema.java`, `PostgresIncrementalSourceFactory.java` and `PostgresTypeUtils.java` changes added merged cleanly with no conflicts, so they're untouched by me.
**Tests**: added `shouldEnableSchemaChangesWhenSchemaEvolutionIsEnabled`, `shouldDisableSchemaChangesByDefault` and `shouldKeepSchemaChangesEnabledAuthoritativeOverUserDebeziumProperties` to `PostgresSourceConfigFactoryTest`, since nothing in the existing suite asserted on `include.schema.changes` at all — that gap is exactly what let the base-class hardcoding go unnoticed by tests.
Separately, on the SNAPSHOT_ONLY blocker from your 2026-08-25 review: it's already fixed on this head (`getStartupModeOption` / `OpengaussSourceOptions.STARTUP_MODE` restoring `SNAPSHOT_ONLY`, matching the docs) — that predates my involvement here, so no action needed from me there; flagging only that it's still present after the merge and unaffected by the conflict resolution.
`./mvnw spotless:apply` run on `connector-cdc-pg-base`, `connector-cdc-postgres` and `connector-cdc-opengauss` before pushing. Per the SeaTunnel local-verification rule I'm not running the modules' tests locally — GitHub CI on this head is the verification source of truth, and given the depth of this reconciliation I'd treat that CI result as load-bearing before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5167173961
Rechecked the latest head on August 3, 2026: the earlier failing CI signal I referenced is no longer the current fact. The latest head is now pending in CI again, and GitHub is also reporting merge conflicts against `dev`.
From Daniel's side I do not see a new source-level blocker on this head; the remaining gates at this point are the merge conflicts plus the current pending checks. Could you please sync with the latest `dev` and resolve the conflicts first? Once that is done and the checks are rerun, I'm happy to take another look.
I only have read-only review controls in this repository, so a maintainer with write access may still need to handle the final approval or dismissal step after the branch is mergeable again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-4845342788
Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
- base branch: `dev`
- head SHA: `cd09a2938e32d69dcf889887647b0ea26b48d367`
- compare status: `diverged`
- ahead_by: `4`
- behind_by: `96`
- mergeable_state: `unknown`
- CI / merge gate snapshot: `pending`
Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5555551636
Thanks @DanielLeens and @SEZ9 — pushed `f48adc80a76` addressing the open items from the 2026-08-23 and 2026-08-31 reviews. I re-traced the `snapshot-only` question against source before changing anything: `SnapshotOnlySplitAssigner` lives in `connector-cdc-base` and `IncrementalSource` gates it purely on `StartupMode` (`getBoundedness()` and `createEnumerator()`), with no dialect call in that path — so you were both right that only `committed-offset` is PostgreSQL-specific.
**Blocking**
- Issue 2 (2026-08-31) / SEZ9 Issue 1 — `SNAPSHOT_ONLY` restored in `OpengaussSourceOptions.STARTUP_MODE`; the choice set is now `initial`, `snapshot-only`, `earliest`, `latest`, with only `committed-offset` excluded. `exactly_once` is offered under `snapshot-only` as well, mirroring the PostgreSQL rule. The Javadoc that claimed both modes were PostgreSQL-specific is corrected. `testOptionRuleExposesOnlyOpengaussStartupModes` pins the four values and explicitly asserts `COMMITTED_OFFSET` is absent; new `testOptionRuleOffersExactlyOnceForBothSnapshotModes` pins the `exactly_once` conditional for both `initial` and `snapshot-only`.
- Issue 1 (2026-08-31) / SEZ9 Issue 6 — the en/zh Opengauss-CDC `startup.mode` rows (which the dev sync had taken from #11707's five-value wording) now list exactly the four accepted values and state that `committed-offset` is PostgreSQL-only and rejected at config validation. The `exactly_once` row now says `initial` or `snapshot-only`.
**Non-blocking**
- Issue 3 / SEZ9 Issue 3 — `OpengaussIncrementalSource.getStartupModeOption()` now documents that it is invoked from inside `IncrementalSource`'s constructor and must remain a stateless constant. I kept it as a Javadoc contract rather than a constructor argument, since the latter would change `IncrementalSource` in `connector-cdc-base`, which this PR deliberately leaves untouched.
- Issue 4 / SEZ9 Issue 7 — the raw `(SingleChoiceOption)` cast is removed entirely; the single-choice builder already returns `SingleChoiceOption<StartupMode>`.
- Issue 5 / SEZ9 Issue 8 — "Apache OpenGauss" wording fixed.
- SEZ9 Issues 2 and 5 — `PostgresDialect`'s replica-identity failure now names both remediations (`ALTER TABLE ... REPLICA IDENTITY FULL` or `require-replica-identity-full = false`), since on enumerator restore the config is baked into the persisted DAG and can only be changed by cancel and resubmit. `PostgresCDCIT` asserts the new exact wording and `PostgresIncrementalSourceTest` asserts both remediations are present. The en/zh Opengauss-CDC prerequisite step now names the opt-out like PostgreSQL-CDC.md does, and the PR description has a "Restore caveat" paragraph covering the cancel-and-resubmit case.
- SEZ9 Issue 4 — agree with Daniel's trace: the pre-refactor `PostgresIncrementalSource` had exactly one instance field, which this PR deletes rather than relocates, and `PgBaseIncrementalSource` declares none, so there is no moved field for per-declaring-class resolution to mis-assign. Added a Compatibility bullet in the description stating this.
CI on the fork for `f48adc80a76` is in flight now. The only remaining human step is the write-capable maintainer approval and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5200417939
Quick sync-status note on top of my earlier review of this exact head.
This branch is currently behind `dev` again (diverged, `behind_by=6`, `ahead_by=7`), and the `Build` check is still red on the same run I already analyzed: a single failing lane, `Run / unit-test (11, windows-latest)`, which fails about 8.5 minutes into a ~60-minute step, well before the reactor reaches the `connector-cdc` modules this PR touches. All 69 other jobs (both JDK 8 and 11, including `all-connectors-it-1..8`, which exercise the Postgres-CDC/OpenGauss-CDC E2E suites end-to-end) are green, and the same lane passed the same day on a sibling branch built off the same `dev`. So I still do not believe this failure is caused by this PR's diff.
Since then, `dev` has also picked up a few CI-stability fixes (including Hazelcast test port isolation and hosted-connector IT stabilization), so the lowest-cost next step now is to sync this branch onto the latest `dev` and rerun CI, rather than just re-running the same lane in isolation.
To be clear, this is on top of the one real blocker from my last review, which still needs to be addressed regardless of CI: adding the small regression test for the `require-replica-identity-full` dialect wiring and disclosing the enforcement restoration in the PR description.
If the `Build` check still fails after the sync, please share the log and I'll dig into that specific lane.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5611668194
Thanks @DanielLeens for the third pass and for re-checking against the head that includes `f561cad5e13`.
On the `snapshot-only` item (PR11029-F1): noted that your read of `OpengaussSourceOptions` is that `SNAPSHOT_ONLY` is back alongside `INITIAL`/`EARLIEST`/`LATEST` with only `COMMITTED_OFFSET` excluded, and that the two option-rule tests you named cover it. I'll confirm that directly against the diff before marking it resolved.
Your review body appears to have been cut off right after the test-name list, so I can't see the rest of your re-verification or your notes on the merge reconciliation. Could you re-post the remainder? In particular, I'd like your read on the other items from the previous round, which I still have open until confirmed:
- **PR11029-F2 / F5** — the restored fail-fast `require-replica-identity-full` enforcement: does the error surfaced at enumerator restore tell the user what to change, and is there now an upgrade/migration note in the Opengauss-CDC docs?
- **PR11029-F3** — `getStartupModeOption()` being an overridable hook consumed during base-class construction in `OpengaussIncrementalSource`; is the initialization-order trap avoided now?
- **PR11029-F4** — switching the Opengauss-CDC runtime source class and inserting `PgBaseIncrementalSource` into the Postgres hierarchy vs. Java-serialized job DAGs from a pre-upgrade Zeta cluster; has this been tested or documented?
- **PR11029-F6** — Opengauss-CDC docs (en/zh) stating the connector-owned `startup.mode` value set.
- **PR11029-F7 / F8** — the raw `(SingleChoiceOption)` cast in `OpengaussSourceOptions.STARTUP_MODE` and the "Apache OpenGauss" wording in the `OpengaussIncrementalSource` Javadoc.
Once the full pass is visible and each item is either confirmed fixed or has a concrete follow-up, I'm happy to move this forward.
<!-- streview-comment:923 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-4667938864
Addressed the OpenGauss follow-up in 8491dfce21 by adding a dedicated regression test for the new entry path. The new test covers both the factory source class contract and the plugin name contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5611982347
Thanks @SEZ9. Checked the raw API body of review 5148548584 again — it is not cut off, it runs the full five-section structure through "Overall assessment" at the end of the Merge Recommendation. Same rendering/collapse pattern we've seen elsewhere on long bodies.
Going through your open items against that full text, plus a direct source check on this head (`f561cad5e13`) for the two your list flags that the review text doesn't restate in prose:
- **F1 (snapshot-only)** — confirmed in section 1.1: `OpengaussSourceOptions.java:50-56` includes `SNAPSHOT_ONLY` alongside `INITIAL`/`EARLIEST`/`LATEST`, only `COMMITTED_OFFSET` excluded, pinned by `testOptionRuleExposesOnlyOpengaussStartupModes`/`testOptionRuleOffersExactlyOnceForBothSnapshotModes`.
- **F2/F5 (replica-identity fail-fast + docs)** — covered in section 1.1 ("Replica-identity error message" bullet: `PostgresDialect.java:140-141` now names both remediations, `ALTER TABLE ... REPLICA IDENTITY FULL` or `require-replica-identity-full=false`) and section 2.3 (`Opengauss-CDC.md` en/zh updated to match the four valid `startup.mode` values). Both confirmed fixed.
- **F3 (`getStartupModeOption()` init-order trap)** — covered in section 1.1: now documented in place at `OpengaussIncrementalSource.java:49-58` as a hard constraint ("must never read instance state"). I independently re-fetched that file at the current head and confirmed the javadoc is there verbatim, right above the override.
- **F4 (serialized job DAG vs. hierarchy change)** — covered in section 1.2 (Compatibility Impact): `serialVersionUID` on `PostgresIncrementalSource`/`PgBaseIncrementalSource`/`PgBaseSourceConfigFactory` is pinned and cross-checked against real `serialver` output from published 2.3.12/2.3.13 artifacts, with an explicit note that a computed UID would drift and break rolling upgrades. That's the right way to prove this is safe for pre-upgrade serialized DAGs.
- **F6 (docs stating connector-owned `startup.mode` set)** — same as F2/F5's doc half, confirmed in section 2.3.
- **F7 (raw `(SingleChoiceOption)` cast)** — not restated in the review prose, so I went and re-pulled `OpengaussSourceOptions.java` at `f561cad5e13` directly. It's fixed: `STARTUP_MODE` is declared straight from `.singleChoice(StartupMode.class, ...)` with no cast, and there's now a javadoc on the field explaining why none is needed ("the single-choice builder already returns a `SingleChoiceOption` of the chosen enum type").
- **F8 ("Apache OpenGauss" wording)** — also re-checked directly: `OpengaussIncrementalSource.java`'s class javadoc now reads "OpenGauss source entry point that reuses the shared PG-base PostgreSQL implementation..." — no "Apache OpenGauss" anywhere in the file. Fixed.
So all eight items (F1-F8) are confirmed fixed on `f561cad5e13`, F1-F6 from the review text itself and F7/F8 from a direct re-check of source I just did. On the one open item from my own review — `Build` was still `in_progress` on this head when I posted — I just re-checked `gh pr checks`: it has since completed `SUCCESS`. With CI green and all eight items confirmed on the same head, I have no remaining findings here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Introduce PG base CDC abstraction for Postgres and OpenGauss [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11029:
URL: https://github.com/apache/seatunnel/pull/11029#issuecomment-5390001346
Thanks for the deep re-review, @SEZ9 — this is on the same head (`075001b0b5a9`) I approved on Aug 14, so I went back and checked the two points that matter most before saying anything.
**Issue 1 (startup.mode narrowing) — I agree, this reopens a real blocker.** I diffed this file against the merge-base: before this PR, `OpengaussIncrementalSourceFactory` wired the shared `PostgresSourceOptions.STARTUP_MODE`, whose choice set is `INITIAL, SNAPSHOT_ONLY, COMMITTED_OFFSET, EARLIEST, LATEST`. The new `OpengaussSourceOptions.STARTUP_MODE` narrows that to `INITIAL, EARLIEST, LATEST` only. `SNAPSHOT_ONLY` is driven by the shared incremental framework (`SnapshotOnlySplitAssigner`, gated in `IncrementalSource.createEnumerator`/`getBoundedness`) rather than by anything Postgres-dialect-specific, so it isn't in the same category as `COMMITTED_OFFSET` (which genuinely does read Postgres-only `pg_replication_slots` columns). That means any existing `Opengauss-CDC` job with `startup.mode = snapshot-only` will now fail `OptionRule` validation at submit with no replacement. That's a straightforward backward-compat break and I missed it in my approval — good catch.
**Issue 3 (getStartupModeOption() as a constructor-time virtual call) — confirmed, and I agree it's the same footgun class.** `IncrementalSource`'s constructor calls `getStartupConfig(readonlyConfig)`, which calls `config.get(getStartupModeOption())`, before the subclass constructor body runs. `OpengaussIncrementalSource.getStartupModeOption()` only happens to be safe today because it returns a stateless static constant — but that's exactly the same shape of bug as the `require-replica-identity-full` init-order issue this PR fixes elsewhere. Agreed this is worth hardening (constructor arg or a `requireNonNull`/Javadoc guard), non-blocking given it's currently harmless.
I haven't re-traced issues 2/4/5/6/7/8 line-by-line in this pass, but they read as consistent with the same compatibility/documentation theme and I don't have reason to push back on any of them.
Given Issue 1 is a genuine blocker, I'm treating my Aug 14 approval as superseded — this needs another pass once the startup-mode compatibility gap is closed (either add `SNAPSHOT_ONLY` back or document/migrate it explicitly), and I'll do a full re-review at that point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org