You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "david-streamlio (via GitHub)" <gi...@apache.org> on 2026/07/09 19:53:14 UTC
[I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
david-streamlio opened a new issue, #63:
URL: https://github.com/apache/pulsar-connectors/issues/63
## Motivation
The repository pins `debezium = "3.4.2.Final"` (released 2026-03-02). The current Debezium release is **3.6.0.Final** (2026-07-01), so we are two minor versions behind.
Debezium has no LTS line — fixes land on the newest minor only. Once #NN bumps us to `3.4.3.Final` (2026-03-30, the last patch the 3.4 line will ever receive), the 3.4 branch is end-of-line and we stop receiving bug and security fixes entirely. Each additional minor release makes the eventual migration more expensive.
## Scope
Bump `debezium` in `gradle/libs.versions.toml` from `3.4.3.Final` to `3.6.0.Final` and fix any resulting breakage.
**The connector set is unchanged.** Comparing the `debezium-bom` contents for 3.4.2 and 3.6.0, the only addition is `debezium-connector-common` (an internal refactor). No wrapper module gains or loses a target.
## Risks
The real risk is our coupling to Debezium **internals**, which are not API-stable across minor versions:
- `debezium/core` implements the schema-history SPI (`PulsarSchemaHistory`) against `io.debezium.storage` / `SchemaHistory` interfaces.
- `kafka-connect-adaptor` (`AbstractKafkaConnectSource`) starts Debezium tasks directly and depends on connector/task config-key conventions.
- `DebeziumSource` subclasses assert on Debezium's `connector.class` / `task.class` values.
Issue #61 is a concrete example of this coupling already breaking against 3.4.x (multi-partition SQL Server connectors never receive `task.id` because we bypass `taskConfigs()`). Expect similar sharp edges on a minor bump; #61 should ideally be fixed first, or at least understood, since it touches the same task-startup path.
## Prerequisites
The integration tests added in #58 (MongoDB), #60 (SQL Server), and #62 (Oracle) should be **merged before this migration**. Prior to those, a Debezium bump was validated only by the MySQL and Postgres tests. With all five source connectors exercised against real databases via Testcontainers, this upgrade either passes CI across the board or points directly at the connector that broke.
## Note: this will not fix the Testcontainers conflict
`debezium-bom:3.4.2.Final` pins Testcontainers core to `2.0.2`; `3.6.0.Final` (via `debezium-build-parent`) pins `2.0.3`. Both are 2.x, while the specialized Testcontainers modules (e.g. `org.testcontainers:oracle-free`) still target 1.21.x and reference shaded classes removed in 2.x. Debezium-module integration tests must keep using the `GenericContainer` pattern (as `debezium/mysql` and `debezium/oracle` do) regardless of this upgrade — see the discussion in #62.
## Suggested order
1. Merge #58, #60, #62 (integration-test safety net)
2. Land the `3.4.3.Final` patch bump (low risk, free fixes)
3. Address or characterize #61
4. Migrate to `3.6.0.Final` and let the five connector integration tests validate it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
Posted by "david-streamlio (via GitHub) via commits" <co...@pulsar.apache.org>.
david-streamlio commented on issue #63:
URL: https://github.com/apache/pulsar-connectors/issues/63#issuecomment-6747352166
Since that is only a minor version difference, e.g they are both 3.x, I would say go with the latest version 3.7.1.Final
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
Posted by "Ayush0612005 (via GitHub) via commits" <co...@pulsar.apache.org>.
Ayush0612005 commented on issue #63:
URL: https://github.com/apache/pulsar-connectors/issues/63#issuecomment-6735189499
@david-streamlio I'd like to take this migration. The prerequisites you listed are in place: #58, #60 and #62 are merged, and #64 brought us to `3.4.3.Final`.
Plan:
- Bump `debezium` to `3.6.0.Final` and fix compile-level breakage, focusing on the internal coupling points you listed: `PulsarSchemaHistory` against the `io.debezium.storage` schema-history SPI, the task startup path in `AbstractKafkaConnectSource`, and the `connector.class` / `task.class` assertions in the `DebeziumSource` subclasses.
- Run the Debezium integration tests (MySQL, PostgreSQL, MongoDB, SQL Server, Oracle) and report each result in the PR, separating anything already failing on `3.4.3.Final` from anything the bump introduces.
- Keep the change to the version bump and the breakage it causes, without touching the connector set.
On #61: since the design choice there is still open, my plan is to keep this PR independent of it. The SQL Server test would be compared against its current `3.4.3.Final` behaviour, and #61 stays a separate fix. If you'd rather have #61 resolved first, I'm happy to wait or to help with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
Posted by "david-streamlio (via GitHub) via commits" <co...@pulsar.apache.org>.
david-streamlio commented on issue #63:
URL: https://github.com/apache/pulsar-connectors/issues/63#issuecomment-6744618823
Happy to let you handle this one @Ayush0612005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
Posted by "Ayush0612005 (via GitHub) via commits" <co...@pulsar.apache.org>.
Ayush0612005 commented on issue #63:
URL: https://github.com/apache/pulsar-connectors/issues/63#issuecomment-6746308808
Thanks @david-streamlio, and thanks for merging #156. I'll start on this, keeping #61 separate as described above, and I'll report the result of each Debezium integration test in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final [pulsar-connectors]
Posted by "Ayush0612005 (via GitHub) via commits" <co...@pulsar.apache.org>.
Ayush0612005 commented on issue #63:
URL: https://github.com/apache/pulsar-connectors/issues/63#issuecomment-6746865568
One question before I start: since this issue was written, Debezium has released 3.6.3.Final (the latest patch on the 3.6 line) and 3.7.1.Final (the current release). Following the reasoning above that fixes only land on the newest minor, would you prefer that I target 3.7.1.Final directly, or stay with 3.6.x (3.6.3.Final) as scoped here and leave 3.7 for a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org