You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tkaymak (via GitHub)" <gi...@apache.org> on 2026/07/31 10:29:55 UTC
[I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
tkaymak opened a new issue, #39568:
URL: https://github.com/apache/beam/issues/39568
### What needs to happen?
`DebeziumIO` currently pins Debezium `3.1.3.Final` (bumped in #37571). Upstream has moved on considerably and the connector should be brought up to the current stable patch line, `3.5.2.Final`.
What the upgrade absorbs:
* **Kafka Connect 4.x alignment.** Debezium 3.5 builds against Kafka Connect 4.1.2, so the `connect-api` pin has to move from 3.9.0. Connect 4.1 also adds an abstract `pluginMetrics()` to `SourceTaskContext`, which `BeamSourceTaskContext` must implement.
* **`debezium-core` module split.** Debezium 3.5 split `debezium-core` into smaller modules (`debezium-config`, `debezium-connector-common`, and friends). The Gradle dependencies and the expansion-service shadow jar need auditing so the split classes are still bundled.
* **Config and behaviour changes.** The `schema_only` snapshot mode was removed in 3.3, and connection failures now surface wrapped in a `RetriableException`.
* **Postgres JDBC driver.** The 3.5 Postgres connector requires driver 42.7.x, while Beam globally pins 42.6.2 for CVE reasons (24f562d84f6).
### Notes
`3.6.0` is intentionally skipped for now: it shipped a rewritten MySQL DDL parser with several open regressions and there is no `3.6.1` yet. A follow-up bump to the 3.6 line should be trivial once those settle.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Java SDK
- [x] Component: IO connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6316512017
Completed by #39569; follow-up metadata work tracked in #40701.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak closed issue #39568: Upgrade DebeziumIO to Debezium 3.5.2.Final
URL: https://github.com/apache/beam/issues/39568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6316511395
Hi @ilyasahsan123, sorry for the slow reply, and thanks a lot for the blog post and the example, really great to see!
You're right about the metadata. `SourceRecordJson` only kept six fields of the Debezium `source` block and dropped everything else, including `pos`, `row` and `ts_ms`. I opened #40701 for it, and #40702 adds the complete `source` block under `metadata.source` (additive, existing fields unchanged). That works for MySQL (`file`, `pos`, `row`, `gtid`, `ts_ms`) as well as the other connectors. Would be great if you could take a look and maybe try it with your pipeline.
Since the 3.5.2 upgrade itself landed with #39569 (shipped in 2.76.0), I'll close this issue and we can continue on #40701.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-5970509144
@ilyasahsan123 please go ahead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "ilyasahsan123 (via GitHub)" <gi...@apache.org>.
ilyasahsan123 commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6309191119
hi @tkaymak
I am writing to share that I have published a blog post on `DebeziumIO` within the Debezium Blog page. The link is below:
- https://debezium.io/blog/2026/10/21/native-support-for-debezium-in-apache-beam
- https://github.com/debezium/debezium-examples/tree/main/apache-beam
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "ilyasahsan123 (via GitHub)" <gi...@apache.org>.
ilyasahsan123 commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-5778148228
hi @tkaymak
I noticed a typo [here](https://github.com/apache/beam/blob/master/sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/DebeziumIO.java#L96C20-L96C89). May I submit a pull request for the correction?
```text
* <p>See <a href="https://debezium.io/documentation/reference/3.5/connectors/index.html">Debezium
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "ilyasahsan123 (via GitHub)" <gi...@apache.org>.
ilyasahsan123 commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6105370007
Hi @tkaymak
We should enable the `pos` and `row` attributes within the `metadata` to support data duplication (MySQL) in the data warehouse.
Additionally, We also need source.ts_ms to identify the database commit time, which is useful for debugging.
Currently, Apache Beam supports only these following attributes, and we could not deduplicate the data correctly.
Apache Beam:
```
metadata={
"connector": "mysql",
"version": "3.5.2.Final",
"name": "beam-debezium-connector",
"database": "inventory",
"schema": "mysql-bin.000003",
"table": "products_on_hand"
}
```
However, here is the complete metadata generated by Debezium:
Debezium
```
"source": {
"version": "3.5.2.Final",
"connector": "mysql",
"name": "tutorial",
"ts_ms": 1791689147000,
"snapshot": "false",
"db": "inventory",
"sequence": null,
"ts_us": 1791689147000000,
"ts_ns": 1791689147000000000,
"table": "customers",
"server_id": 223344,
"gtid": null,
"file": "mysql-bin.000003",
"pos": 796,
"row": 0,
"thread": 13,
"query": null
}
```
Is there any explanation about this?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6315426098
Hi @ilyasahsan123
Thank you for the blog post and the issue description, I will pick this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade DebeziumIO to Debezium 3.5.2.Final [beam]
Posted by "ilyasahsan123 (via GitHub)" <gi...@apache.org>.
ilyasahsan123 commented on issue #39568:
URL: https://github.com/apache/beam/issues/39568#issuecomment-6282917265
hi @tkaymak @Abacn,
How are you? I hope you are doing well.
Do you have answer about my question above?
Please tell me if you need assistance to enable that; I am glad to contribute to this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org