You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Joao Boto <bo...@boto.pro> on 2025/07/25 12:00:16 UTC
[Discuss][Proposal] Decouple the Debezium Connector from the Flink CDC Project
Hi,
I would like to open a discussion on the possibility of separating the
Debezium connector from the Flink CDC project. If this topic has been
previously discussed, I would be grateful for a link to the relevant
information.
The Flink CDC project is a spectacular initiative. However, embedding the
Debezium connector within it imposes certain constraints that limit the
connector's own development pace. A primary example of this is the minimum
Java version requirement.
Currently, Flink CDC uses Debezium 1.9, which is the last version
compatible with Java 8. While an upgrade to Flink 2.0 will likely enable a
move to a newer version, we would still be constrained to Debezium 2.7 (the
last version supporting Java 11). With the main Debezium project now at
version 3.2 (requiring Java 17), the Flink community is unable to leverage
many valuable evolutions. These features may not be critical to the core
Flink CDC framework but are highly valuable for users who rely on the
connector for data extraction.
My proposal is to extract the connector into a separate
*flink-connector-debezium
*project. This would enable the connector to have its own release cycle,
aligning with Debezium's major versions. This approach would support Flink
CDC's roadmap while also allowing the connector to evolve independently,
benefiting a wider range of users.
I look forward to hearing your thoughts on this.
Re: [Discuss][Proposal] Decouple the Debezium Connector from the Flink CDC Project
Posted by João Boto <es...@apache.org>.
Hi Leonard,
Thanks for your comments.
I will try not to focus on the other connectors and the apparent widespread abandonment they seem to be facing (with the exception of the most commonly used ones). I believe that could give us a separated discussion.
Regarding the matter at hand, I find your point about wanting to have both in the same project completely valid. However, that also becomes the connector's greatest limitation. It is perfectly feasible for a project like this to wait for all of its connectors to evolve before upgrading, but this also limits the evolution of the embedded connector itself. In this case, we are not moving to Flink 2.0 to wait for the other connectors, which in turn also limits the Debezium connector.
My main concern is that the evolution of the FlinkCDC application shouldn't block the evolution of its primary connector.
Perhaps we can likely reduce the problem to the Java version "limitation" imposed by Flink:
- For a standalone connector, is it necessary to adhere to that limitation? I remember a discussion about a connector where the minimum Java version was raised to match the connector's (as long as Flink supported it). This implies your job would have to run on that Java version, but if you're already using that connector, it shouldn't be a problem.
- For FlinkCDC, I understand we want to keep the version as close as possible to Flink's minimum, which makes a lot of sense.
As for use cases, there are others, as example we use the connector standalone in various scenarios, which I would be happy to discuss with you offline if you are interested.
On 2025/07/27 05:50:13 Leonard Xu wrote:
> Hey Joao,
>
> Thanks for kicking off this thread, I have some comments as follows:
>
> > Currently, Flink CDC uses Debezium 1.9, which is the last version
> > compatible with Java 8. While an upgrade to Flink 2.0 will likely enable a
> > move to a newer version, we would still be constrained to Debezium 2.7 (the
> > last version supporting Java 11).
>
> This is already part of Flink CDC's roadmap — we've discussed upgrading Flink to version 2.0 [1]. However, like other independent connector repositories, we prefer to wait until Flink 2.0 becomes more stable before bumping both the Flink version and the Debezium version to 2.7.
>
> > With the main Debezium project now at
> > version 3.2 (requiring Java 17), the Flink community is unable to leverage
> > many valuable evolutions. These features may not be critical to the core
>
> In this case, we will still encounter the Java 17 issue regardless of whether we choose to decouple the Debezium connector from Flink or upgrade the Debezium version within Flink CDC, since both approaches depend on Flink, which currently does not support Java 17, right?
>
> So why don't we directly accelerate the upgrade to Flink 2.0 and Debezium 2.7 in the Flink CDC project? To be honest, I'm not particularly in favor of splitting the connectors into external projects and releasing them independently.
>
> (1) Frankly, Flink's external connectors currently lack sufficient community support for maintenance. For instance, HBase is a sufficiently influential project within the big data ecosystem. However, the latest supported Flink version is 1.19.x, which was released last year [2]. I've seen contributors in the community pushing for adaptation to Flink 2.0. Without enough dedicated committers and PMCs willing to invest time and effort into developing and maintaining these connectors, their future development cannot be guaranteed.
>
> (2) The Flink CDC project has been steadily releasing versions and fixing various issues [3]. This is because contributors to the Flink CDC project, including myself, come from multiple companies and use Flink CDC extensively in our respective organizations. We have the motivation and resources to invest in this project, enabling its healthy growth.
>
> (3) Currently, the primary use case for flink-connector-debezium is still within the various CDC connectors in the Flink CDC project. I haven't seen many other usage scenarios. Even if there are, as mentioned in my previous response, the motivation to drive solutions would likely be stronger within the Flink CDC project itself.
>
> Overall, I still prefer concentrating the community's limited resources on jointly maintaining a single repository. I don't want to see situations where Flink is already planning for version 2.2, while certain community-maintained connectors remain stuck on version 1.16.
>
>
> Best,
> Leonard
>
> [1]https://lists.apache.org/thread/n7zq6yt31s9xy77wrqtv2wxdn6gv5ytm
> [2] https://github.com/apache/flink-connector-hbase
> [3] https://github.com/apache/flink-cdc/releases
>
>
>
>
>
>
>
Re: [Discuss][Proposal] Decouple the Debezium Connector from the Flink CDC Project
Posted by Leonard Xu <xb...@gmail.com>.
Hey Joao,
Thanks for kicking off this thread, I have some comments as follows:
> Currently, Flink CDC uses Debezium 1.9, which is the last version
> compatible with Java 8. While an upgrade to Flink 2.0 will likely enable a
> move to a newer version, we would still be constrained to Debezium 2.7 (the
> last version supporting Java 11).
This is already part of Flink CDC's roadmap — we've discussed upgrading Flink to version 2.0 [1]. However, like other independent connector repositories, we prefer to wait until Flink 2.0 becomes more stable before bumping both the Flink version and the Debezium version to 2.7.
> With the main Debezium project now at
> version 3.2 (requiring Java 17), the Flink community is unable to leverage
> many valuable evolutions. These features may not be critical to the core
In this case, we will still encounter the Java 17 issue regardless of whether we choose to decouple the Debezium connector from Flink or upgrade the Debezium version within Flink CDC, since both approaches depend on Flink, which currently does not support Java 17, right?
So why don't we directly accelerate the upgrade to Flink 2.0 and Debezium 2.7 in the Flink CDC project? To be honest, I'm not particularly in favor of splitting the connectors into external projects and releasing them independently.
(1) Frankly, Flink's external connectors currently lack sufficient community support for maintenance. For instance, HBase is a sufficiently influential project within the big data ecosystem. However, the latest supported Flink version is 1.19.x, which was released last year [2]. I've seen contributors in the community pushing for adaptation to Flink 2.0. Without enough dedicated committers and PMCs willing to invest time and effort into developing and maintaining these connectors, their future development cannot be guaranteed.
(2) The Flink CDC project has been steadily releasing versions and fixing various issues [3]. This is because contributors to the Flink CDC project, including myself, come from multiple companies and use Flink CDC extensively in our respective organizations. We have the motivation and resources to invest in this project, enabling its healthy growth.
(3) Currently, the primary use case for flink-connector-debezium is still within the various CDC connectors in the Flink CDC project. I haven't seen many other usage scenarios. Even if there are, as mentioned in my previous response, the motivation to drive solutions would likely be stronger within the Flink CDC project itself.
Overall, I still prefer concentrating the community's limited resources on jointly maintaining a single repository. I don't want to see situations where Flink is already planning for version 2.2, while certain community-maintained connectors remain stuck on version 1.16.
Best,
Leonard
[1]https://lists.apache.org/thread/n7zq6yt31s9xy77wrqtv2wxdn6gv5ytm
[2] https://github.com/apache/flink-connector-hbase
[3] https://github.com/apache/flink-cdc/releases
Re: [Discuss][Proposal] Decouple the Debezium Connector from the Flink CDC Project
Posted by João Boto <es...@apache.org>.
Hi,
Following Leonard Xu's suggestion, if we try to update to Flink 2.0, we encounter a significant limitation with FlinkCDC. While not a major issue for FlinkCDC itself, it becomes more pronounced for the connector: some of the connectors that FlinkCDC use, such as Doris and Paimon, do not yet have a version compatible with Flink 2.0.
Individually, these limitations (and the previous mentioned) aren't severe. However, from a broader perspective, the evolution of the Debezium connector is tied to several external factors that arguably shouldn't influence its development and update cycle.
I still believe that externalizing the connector would be the best path forward. I agree with Leonard Xu's point about some connectors lacking dedicated maintainers. However, in this particular case, I think the connector would actually attract more contributors if it weren't part of an ecosystem so dependent on external factors.
Best regards
On 2025/07/25 12:00:16 Joao Boto wrote:
> Hi,
>
> I would like to open a discussion on the possibility of separating the
> Debezium connector from the Flink CDC project. If this topic has been
> previously discussed, I would be grateful for a link to the relevant
> information.
> The Flink CDC project is a spectacular initiative. However, embedding the
> Debezium connector within it imposes certain constraints that limit the
> connector's own development pace. A primary example of this is the minimum
> Java version requirement.
> Currently, Flink CDC uses Debezium 1.9, which is the last version
> compatible with Java 8. While an upgrade to Flink 2.0 will likely enable a
> move to a newer version, we would still be constrained to Debezium 2.7 (the
> last version supporting Java 11). With the main Debezium project now at
> version 3.2 (requiring Java 17), the Flink community is unable to leverage
> many valuable evolutions. These features may not be critical to the core
> Flink CDC framework but are highly valuable for users who rely on the
> connector for data extraction.
>
> My proposal is to extract the connector into a separate
> *flink-connector-debezium
> *project. This would enable the connector to have its own release cycle,
> aligning with Debezium's major versions. This approach would support Flink
> CDC's roadmap while also allowing the connector to evolve independently,
> benefiting a wider range of users.
>
> I look forward to hearing your thoughts on this.
>
Re: [Discuss][Proposal] Decouple the Debezium Connector from the Flink CDC Project
Posted by Tom Cooper <co...@tomcooper.dev>.
Hi Joao,
+1 from me on this proposal.
I think having a more dynamic (and up-to-date) connector eco-system is vital for Flink's success.
Being able to take advantage of new features (and patch old bugs and vulnerabilities) as soon as possible is a big plus point!
Tom Cooper
@tomcooper.dev | https://tomcooper.dev
On Friday, 25 July 2025 at 13:00, Joao Boto <bo...@boto.pro> wrote:
> Hi,
>
> I would like to open a discussion on the possibility of separating the
> Debezium connector from the Flink CDC project. If this topic has been
> previously discussed, I would be grateful for a link to the relevant
> information.
> The Flink CDC project is a spectacular initiative. However, embedding the
> Debezium connector within it imposes certain constraints that limit the
> connector's own development pace. A primary example of this is the minimum
> Java version requirement.
> Currently, Flink CDC uses Debezium 1.9, which is the last version
> compatible with Java 8. While an upgrade to Flink 2.0 will likely enable a
> move to a newer version, we would still be constrained to Debezium 2.7 (the
> last version supporting Java 11). With the main Debezium project now at
> version 3.2 (requiring Java 17), the Flink community is unable to leverage
> many valuable evolutions. These features may not be critical to the core
> Flink CDC framework but are highly valuable for users who rely on the
> connector for data extraction.
>
> My proposal is to extract the connector into a separate
> *flink-connector-debezium
> *project. This would enable the connector to have its own release cycle,
> aligning with Debezium's major versions. This approach would support Flink
> CDC's roadmap while also allowing the connector to evolve independently,
> benefiting a wider range of users.
>
> I look forward to hearing your thoughts on this.