You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "CC-Hsu (via GitHub)" <gi...@apache.org> on 2023/06/27 09:33:13 UTC
[GitHub] [pulsar] CC-Hsu opened a new issue, #20656: Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features
CC-Hsu opened a new issue, #20656:
URL: https://github.com/apache/pulsar/issues/20656
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Currently the Pulsar I/O DBZ connector is v1.9.7.
https://github.com/apache/pulsar/blob/v3.0.0/pom.xml#L194
https://github.com/apache/pulsar/blob/master/pom.xml#L195
Previously there was an update from DBZ v1.0. https://github.com/apache/pulsar/issues/12418
Recently there are many notable new features for the recent releases, and I list several big features of them.
* [DBZ v2.3](https://debezium.io/releases/2.3/release-notes)
- Support exactly-once semantic for streaming phase from Postgres connector [DBZ-6547](https://issues.redhat.com/browse/DBZ-6547)
- PostgreSQL: Set Replica Identity when the connector starts [DBZ-6112](https://issues.redhat.com/browse/DBZ-6112)
- Support multiple tasks when streaming shard list [DBZ-6365](https://issues.redhat.com/browse/DBZ-6365)
* [DBZ v2.2](https://debezium.io/releases/2.2/release-notes)
- Capture evenents in order across mongodb shards [DBZ-5590](https://issues.redhat.com/browse/DBZ-5590)
- Debezium JDBC Sink Connector [DBZ-3647](https://issues.redhat.com/browse/DBZ-3647)
* [DBZ v2.1](https://debezium.io/releases/2.1/release-notes)
- Implement support for JSON_TABLE in MySQL parser [DBZ-3575](https://issues.redhat.com/browse/DBZ-3575)
- Support for Postgres 15 [DBZ-5370](https://issues.redhat.com/browse/DBZ-5370)
- MySQL Connector capture TRUNCATE command as message in table topic [DBZ-5610](https://issues.redhat.com/browse/DBZ-5610)
* [DBZ v2.0](https://debezium.io/releases/2.0/release-notes)
- Improve LogMiner query performance by reducing REGEXP_LIKE disjunctions [DBZ-5648](https://issues.redhat.com/browse/DBZ-5648)
- Added support for Mongo pre-image in change stream [DBZ-5628](https://issues.redhat.com/browse/DBZ-5628)
- Support binlog compression for MySQL [DBZ-2663](https://issues.redhat.com/browse/DBZ-2663)
- support mongodb connection string as configuration option [DBZ-4733](https://issues.redhat.com/browse/DBZ-4733)
- Mysql Commit Timestamp [DBZ-5170](https://issues.redhat.com/browse/DBZ-5170)
And there is [a whole new DBZ JDBC sink connector available](https://debezium.io/documentation/reference/2.3/connectors/jdbc.html).
It would be great if the Pulsar I/O DBZ connector can also be updated and including these new exciting features if possible.
### Solution
1. Update Pulsar IO DBZ connector to at lease v2.2 (update to the latest one is more appreciated), which contains the DBZ JDBC sink connector, and MongoDB sharding support.
2. Adding [the DBZ JDBC sink connector](https://debezium.io/documentation/reference/2.3/connectors/jdbc.html) to Pulsar IO CDC connector
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #20656: Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20656:
URL: https://github.com/apache/pulsar/issues/20656#issuecomment-1692637252
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #20656: Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #20656:
URL: https://github.com/apache/pulsar/issues/20656#issuecomment-1650819914
@CC-Hsu can you submit a patch to see if version is compatible? Version bump especially major version bump may not be quite easy to implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed issue #20656: Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features
URL: https://github.com/apache/pulsar/issues/20656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Updating Debezium from v1.9 to the newer one (currently v2.3) and include new features [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20656:
URL: https://github.com/apache/pulsar/issues/20656#issuecomment-4404617457
Addressed in #24712 and #25335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org