You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Tejansh Rana <te...@autodesk.com.INVALID> on 2025/10/17 14:58:14 UTC
[PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Hi team,
Following my discussion with Leonard Xu at Flink Forward, I am writing to propose a feature enhancement for the Flink MySQL CDC connector related to how it handles transaction metadata from the MySQL binary log.
Problem Statement:
In data streaming pipelines that require transactional guarantees or need to group atomic changes together, it is essential to identify the boundaries of the original database transaction (i.e., the BEGIN and COMMIT or END events). Currently, the Flink MySQL CDC connector appears to skip these transaction lifecycle events - https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77 .
I have also attached a screenshot of the logs from this behaviour.
This omission makes it challenging to reconstruct the original transaction scope. Without explicit transaction markers, downstream Flink jobs cannot easily guarantee atomicity across sinks.
Proposed Solution:
The underlying CDC mechanism, Debezium, supports emitting transaction boundary events (BEGIN and END/COMMIT) through its configuration.
We propose enhancing the Flink MySQL CDC connector to expose this transaction metadata to the Flink pipeline. The connector should emit specialised records or metadata fields that indicate the start and end of a transaction as emitted. We would be happy to create a PR with this feature if this proposal goes ahead.
Thank you,
Tejansh
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Tejansh Rana <te...@autodesk.com>.
Hi Hang, Leonard,
Based on your feedback, I have updated the Jira ticket and incorporated the changes in PR. I have now marked the PR ready for review. Could you please share your feedback?
Link to PR - https://github.com/apache/flink-cdc/pull/4170
Thank you,
Tejansh
From: Tejansh Rana <te...@autodesk.com>
Date: Monday, 24 November 2025 at 18:23
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
Thank you for the feedback, Hang and Leonard. I agree to your points about this being supported only in the DataStream API. Also, I will add a new configuration for these transaction metadata events to only be emitted when they are set to true (default to false) so this is a non-breaking change.
I will update the ticket and the PR and come back to this thread for further reviews.
Thank you,
Tejansh
From: Leonard Xu <xb...@gmail.com>
Date: Thursday, 20 November 2025 at 07:57
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
+1 fro Hang’s point to introduce a default false option to avoid possible performance regression and potential compatibility breaking.
Best,
Leonard
> 2025 11月 20 15:08，Hang Ruan <ru...@apache.org> 写道：
>
> Hi, Tejansh
>
> Thanks for your work.
>
> I have some thoughts regarding this proposed change, and there are a
> few points that need to be emphasized:
> First, since Flink SQL/Table API relies on RowData, it cannot handle
> transactional metadata. Therefore, this feature can only be supported
> in the DataStream API.
> Second, because parsing this transactional information may introduce
> performance overhead, it would be best to add an option to control
> whether this parsing is enabled or not—and by default, it should be
> disabled.
> As I understand it, the current proposal only includes changes to the
> CDC connector and does not cover the MySQL pipeline connector. If we
> decide to support this feature in the MySQL pipeline connector in the
> future, we can discuss that separately in another JIRA ticket.
>
> Best,
> Hang
>
> On Tue, Nov 18, 2025 at 5:13 PM Tejansh Rana
> <te...@autodesk.com.invalid> wrote:
>>
>> Hello,
>>
>> Bumping this thread.
>> I have created this Jira ticket describing the proposal and based on Gunnar’s feedback, I have also included the base Source Emitter which would cover this feature for connectors like Postgres - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-38691&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119391291%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MGaFGgLuKllx4tHAJrqfEsnfZNPcUMJ1y4OLPbFbU3A%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-38691&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119410781%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Uih6j5YeHLCrE19bUPpXM7kxaDia7L6kNSQmjUkY6VY%3D&reserved=0><https://issues.apache.org/jira/browse/FLINK-38691>
>> This is the draft PR with proposed changes for MySQL connector - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119426665%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gnVLH9%2Bc1pPx7U7d4Vu3%2BDu2ovvRvqfjixWf6h9UIII%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119443324%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XPKf9DXBQqMIHfzL%2FGwqpACHR6rJcNqSBzH10KFI9aY%3D&reserved=0><https://github.com/apache/flink-cdc/pull/4170>
>>
>> I would appreciate some feedback on this proposal and I would be happy to contribute the feature.
>>
>> Thank you,
>> Tejansh
>>
>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>> Date: Tuesday, 4 November 2025 at 16:50
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Thank you, Gunnar!
>> I have created a draft PR for the proposed feature - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119459289%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fEbDmmSrAw%2Fu8EerwbTRQsKAOwIAhiMwQdgOxQYoNPM%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119476793%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Z31evGEqDB%2FcYxLhYEm2faME03Jmiqm3W3ODye6xcz4%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119493224%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sg%2FAo0oWLwUaEQt41jtShSVbO1BQwLzRe2HSJ9n7jh0%3D&reserved=0><https://github.com/apache/flink-cdc/pull/4170>
>> Looking forward to hearing more feedback.
>>
>> Thank you,
>> Tejansh
>>
>> From: Gunnar Morling <gu...@googlemail.com.INVALID>
>> Date: Monday, 3 November 2025 at 17:19
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Hey all,
>>
>> I'd love to see support for this! Coincidentally, I am just working on a
>> PoC right now which uses the custom watermarks in the DataStream v2 API to
>> represent transaction boundaries. It seems this is a great fit
>> conceptually. In any case, it would be nice to not only support this for
>> MySQL but also other DBs. Debezium provides that transaction metadata for a
>> range of connectors, including Postgres.
>>
>> --Gunnar
>>
>>
>> On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
>> wrote:
>>
>>> Hello,
>>>
>>> Following up on the below proposal. Would appreciate your thoughts and if
>>> we could move forward with this feature.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>>> Date: Friday, 17 October 2025 at 15:58
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
>>> CDC Connector
>>> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
>>> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>>>
>>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
>>> trust the sender and know the content is safe.
>>>
>>> Hi team,
>>>
>>> Following my discussion with Leonard Xu at Flink Forward, I am writing to
>>> propose a feature enhancement for the Flink MySQL CDC connector related to
>>> how it handles transaction metadata from the MySQL binary log.
>>>
>>> Problem Statement:
>>> In data streaming pipelines that require transactional guarantees or need
>>> to group atomic changes together, it is essential to identify the
>>> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
>>> or END events). Currently, the Flink MySQL CDC connector appears to skip
>>> these transaction lifecycle events -
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119509502%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=chQYG5urnsLhP4kZ46lb1S4EAfQB%2FlF%2F7z6fzTF24zc%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119525790%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=4YG27CxP1vYe5RMGm7qSqUWwYBQu4qhA6OULJbBK8ZM%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119542508%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=af3uToPd%2F3wxBVaV5YLNetWN7nUU8s4b8quUUj8jmuY%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119558995%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZeB4NjnNgXATSg5jYtxD0yKnHiro21XvQ6U2Q7cse%2BM%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
>>> .
>>> I have also attached a screenshot of the logs from this behaviour.
>>>
>>> This omission makes it challenging to reconstruct the original transaction
>>> scope. Without explicit transaction markers, downstream Flink jobs cannot
>>> easily guarantee atomicity across sinks.
>>>
>>> Proposed Solution:
>>> The underlying CDC mechanism, Debezium, supports emitting transaction
>>> boundary events (BEGIN and END/COMMIT) through its configuration.
>>>
>>> We propose enhancing the Flink MySQL CDC connector to expose this
>>> transaction metadata to the Flink pipeline. The connector should emit
>>> specialised records or metadata fields that indicate the start and end of a
>>> transaction as emitted. We would be happy to create a PR with this feature
>>> if this proposal goes ahead.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>>
>>>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Tejansh Rana <te...@autodesk.com>.
Thank you for the feedback, Hang and Leonard. I agree to your points about this being supported only in the DataStream API. Also, I will add a new configuration for these transaction metadata events to only be emitted when they are set to true (default to false) so this is a non-breaking change.
I will update the ticket and the PR and come back to this thread for further reviews.
Thank you,
Tejansh
From: Leonard Xu <xb...@gmail.com>
Date: Thursday, 20 November 2025 at 07:57
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
+1 fro Hang’s point to introduce a default false option to avoid possible performance regression and potential compatibility breaking.
Best,
Leonard
> 2025 11月 20 15:08，Hang Ruan <ru...@apache.org> 写道：
>
> Hi, Tejansh
>
> Thanks for your work.
>
> I have some thoughts regarding this proposed change, and there are a
> few points that need to be emphasized:
> First, since Flink SQL/Table API relies on RowData, it cannot handle
> transactional metadata. Therefore, this feature can only be supported
> in the DataStream API.
> Second, because parsing this transactional information may introduce
> performance overhead, it would be best to add an option to control
> whether this parsing is enabled or not—and by default, it should be
> disabled.
> As I understand it, the current proposal only includes changes to the
> CDC connector and does not cover the MySQL pipeline connector. If we
> decide to support this feature in the MySQL pipeline connector in the
> future, we can discuss that separately in another JIRA ticket.
>
> Best,
> Hang
>
> On Tue, Nov 18, 2025 at 5:13 PM Tejansh Rana
> <te...@autodesk.com.invalid> wrote:
>>
>> Hello,
>>
>> Bumping this thread.
>> I have created this Jira ticket describing the proposal and based on Gunnar’s feedback, I have also included the base Source Emitter which would cover this feature for connectors like Postgres - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-38691&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540781159%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=l1orm7NmpQ4%2Fspt5nd0W9oJLpzyvwa1RZmHdG%2BcQwS4%3D&reserved=0<https://issues.apache.org/jira/browse/FLINK-38691>
>> This is the draft PR with proposed changes for MySQL connector - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540802139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=C7ZCE88tS%2FxQJOgubMGiiVVCKyVv9wvYo4cc2yvCupg%3D&reserved=0<https://github.com/apache/flink-cdc/pull/4170>
>>
>> I would appreciate some feedback on this proposal and I would be happy to contribute the feature.
>>
>> Thank you,
>> Tejansh
>>
>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>> Date: Tuesday, 4 November 2025 at 16:50
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Thank you, Gunnar!
>> I have created a draft PR for the proposed feature - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540818073%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=z2rhfyuufkfcwjWWeTKA6rN44Vg8Y0c8KWxZGO0CWCk%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540834316%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Xg%2FRwXFWnAKtoWNtJg2OXzccousXWqBxC1gzJ9Ahi2w%3D&reserved=0><https://github.com/apache/flink-cdc/pull/4170>
>> Looking forward to hearing more feedback.
>>
>> Thank you,
>> Tejansh
>>
>> From: Gunnar Morling <gu...@googlemail.com.INVALID>
>> Date: Monday, 3 November 2025 at 17:19
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Hey all,
>>
>> I'd love to see support for this! Coincidentally, I am just working on a
>> PoC right now which uses the custom watermarks in the DataStream v2 API to
>> represent transaction boundaries. It seems this is a great fit
>> conceptually. In any case, it would be nice to not only support this for
>> MySQL but also other DBs. Debezium provides that transaction metadata for a
>> range of connectors, including Postgres.
>>
>> --Gunnar
>>
>>
>> On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
>> wrote:
>>
>>> Hello,
>>>
>>> Following up on the below proposal. Would appreciate your thoughts and if
>>> we could move forward with this feature.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>>> Date: Friday, 17 October 2025 at 15:58
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
>>> CDC Connector
>>> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
>>> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>>>
>>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
>>> trust the sender and know the content is safe.
>>>
>>> Hi team,
>>>
>>> Following my discussion with Leonard Xu at Flink Forward, I am writing to
>>> propose a feature enhancement for the Flink MySQL CDC connector related to
>>> how it handles transaction metadata from the MySQL binary log.
>>>
>>> Problem Statement:
>>> In data streaming pipelines that require transactional guarantees or need
>>> to group atomic changes together, it is essential to identify the
>>> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
>>> or END events). Currently, the Flink MySQL CDC connector appears to skip
>>> these transaction lifecycle events -
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540850486%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=hIByDrBV%2FQ%2F4JG1zWP9SBAi7tFKfK0yArTjbbxO%2Bd3k%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540868562%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Z0qfLfUPisAJvW878SPB5dDkN4RmtauXRwZMsYUWJ84%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ce167be9af5ef460b0aab08de280a7490%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638992222540884867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bLfqRjE3FcCB0LwI4Dsn4k9igF6OGE6WZd0BZaMCN7g%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
>>> .
>>> I have also attached a screenshot of the logs from this behaviour.
>>>
>>> This omission makes it challenging to reconstruct the original transaction
>>> scope. Without explicit transaction markers, downstream Flink jobs cannot
>>> easily guarantee atomicity across sinks.
>>>
>>> Proposed Solution:
>>> The underlying CDC mechanism, Debezium, supports emitting transaction
>>> boundary events (BEGIN and END/COMMIT) through its configuration.
>>>
>>> We propose enhancing the Flink MySQL CDC connector to expose this
>>> transaction metadata to the Flink pipeline. The connector should emit
>>> specialised records or metadata fields that indicate the start and end of a
>>> transaction as emitted. We would be happy to create a PR with this feature
>>> if this proposal goes ahead.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>>
>>>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Leonard Xu <xb...@gmail.com>.
+1 fro Hang’s point to introduce a default false option to avoid possible performance regression and potential compatibility breaking.
Best,
Leonard
> 2025 11月 20 15:08，Hang Ruan <ru...@apache.org> 写道：
>
> Hi, Tejansh
>
> Thanks for your work.
>
> I have some thoughts regarding this proposed change, and there are a
> few points that need to be emphasized:
> First, since Flink SQL/Table API relies on RowData, it cannot handle
> transactional metadata. Therefore, this feature can only be supported
> in the DataStream API.
> Second, because parsing this transactional information may introduce
> performance overhead, it would be best to add an option to control
> whether this parsing is enabled or not—and by default, it should be
> disabled.
> As I understand it, the current proposal only includes changes to the
> CDC connector and does not cover the MySQL pipeline connector. If we
> decide to support this feature in the MySQL pipeline connector in the
> future, we can discuss that separately in another JIRA ticket.
>
> Best,
> Hang
>
> On Tue, Nov 18, 2025 at 5:13 PM Tejansh Rana
> <te...@autodesk.com.invalid> wrote:
>>
>> Hello,
>>
>> Bumping this thread.
>> I have created this Jira ticket describing the proposal and based on Gunnar’s feedback, I have also included the base Source Emitter which would cover this feature for connectors like Postgres - https://issues.apache.org/jira/browse/FLINK-38691
>> This is the draft PR with proposed changes for MySQL connector - https://github.com/apache/flink-cdc/pull/4170
>>
>> I would appreciate some feedback on this proposal and I would be happy to contribute the feature.
>>
>> Thank you,
>> Tejansh
>>
>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>> Date: Tuesday, 4 November 2025 at 16:50
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Thank you, Gunnar!
>> I have created a draft PR for the proposed feature - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322279639%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJAvbS0TVL8FWX62mqD69pJyyb6PDzd3osZpPy9LawI%3D&reserved=0<https://github.com/apache/flink-cdc/pull/4170>
>> Looking forward to hearing more feedback.
>>
>> Thank you,
>> Tejansh
>>
>> From: Gunnar Morling <gu...@googlemail.com.INVALID>
>> Date: Monday, 3 November 2025 at 17:19
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
>> [You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>>
>>
>> Hey all,
>>
>> I'd love to see support for this! Coincidentally, I am just working on a
>> PoC right now which uses the custom watermarks in the DataStream v2 API to
>> represent transaction boundaries. It seems this is a great fit
>> conceptually. In any case, it would be nice to not only support this for
>> MySQL but also other DBs. Debezium provides that transaction metadata for a
>> range of connectors, including Postgres.
>>
>> --Gunnar
>>
>>
>> On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
>> wrote:
>>
>>> Hello,
>>>
>>> Following up on the below proposal. Would appreciate your thoughts and if
>>> we could move forward with this feature.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>> From: Tejansh Rana <te...@autodesk.com.INVALID>
>>> Date: Friday, 17 October 2025 at 15:58
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
>>> CDC Connector
>>> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
>>> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>>>
>>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
>>> trust the sender and know the content is safe.
>>>
>>> Hi team,
>>>
>>> Following my discussion with Leonard Xu at Flink Forward, I am writing to
>>> propose a feature enhancement for the Flink MySQL CDC connector related to
>>> how it handles transaction metadata from the MySQL binary log.
>>>
>>> Problem Statement:
>>> In data streaming pipelines that require transactional guarantees or need
>>> to group atomic changes together, it is essential to identify the
>>> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
>>> or END events). Currently, the Flink MySQL CDC connector appears to skip
>>> these transaction lifecycle events -
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322297418%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6szuQvjDu7qdG8SVG7xKoZAgQo5zIQhBR25vsE%2F50UU%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322412677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=APMT7%2FgNDJZFza8K9JnG1OvAW1z2qyYzEyoiTI8%2Btx0%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
>>> .
>>> I have also attached a screenshot of the logs from this behaviour.
>>>
>>> This omission makes it challenging to reconstruct the original transaction
>>> scope. Without explicit transaction markers, downstream Flink jobs cannot
>>> easily guarantee atomicity across sinks.
>>>
>>> Proposed Solution:
>>> The underlying CDC mechanism, Debezium, supports emitting transaction
>>> boundary events (BEGIN and END/COMMIT) through its configuration.
>>>
>>> We propose enhancing the Flink MySQL CDC connector to expose this
>>> transaction metadata to the Flink pipeline. The connector should emit
>>> specialised records or metadata fields that indicate the start and end of a
>>> transaction as emitted. We would be happy to create a PR with this feature
>>> if this proposal goes ahead.
>>>
>>> Thank you,
>>> Tejansh
>>>
>>>
>>>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Hang Ruan <ru...@apache.org>.
Hi, Tejansh
Thanks for your work.
I have some thoughts regarding this proposed change, and there are a
few points that need to be emphasized:
First, since Flink SQL/Table API relies on RowData, it cannot handle
transactional metadata. Therefore, this feature can only be supported
in the DataStream API.
Second, because parsing this transactional information may introduce
performance overhead, it would be best to add an option to control
whether this parsing is enabled or not—and by default, it should be
disabled.
As I understand it, the current proposal only includes changes to the
CDC connector and does not cover the MySQL pipeline connector. If we
decide to support this feature in the MySQL pipeline connector in the
future, we can discuss that separately in another JIRA ticket.
Best,
Hang
On Tue, Nov 18, 2025 at 5:13 PM Tejansh Rana
<te...@autodesk.com.invalid> wrote:
>
> Hello,
>
> Bumping this thread.
> I have created this Jira ticket describing the proposal and based on Gunnar’s feedback, I have also included the base Source Emitter which would cover this feature for connectors like Postgres - https://issues.apache.org/jira/browse/FLINK-38691
> This is the draft PR with proposed changes for MySQL connector - https://github.com/apache/flink-cdc/pull/4170
>
> I would appreciate some feedback on this proposal and I would be happy to contribute the feature.
>
> Thank you,
> Tejansh
>
> From: Tejansh Rana <te...@autodesk.com.INVALID>
> Date: Tuesday, 4 November 2025 at 16:50
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
> [You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>
>
> Thank you, Gunnar!
> I have created a draft PR for the proposed feature - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322279639%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJAvbS0TVL8FWX62mqD69pJyyb6PDzd3osZpPy9LawI%3D&reserved=0<https://github.com/apache/flink-cdc/pull/4170>
> Looking forward to hearing more feedback.
>
> Thank you,
> Tejansh
>
> From: Gunnar Morling <gu...@googlemail.com.INVALID>
> Date: Monday, 3 November 2025 at 17:19
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
> [You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>
>
> Hey all,
>
> I'd love to see support for this! Coincidentally, I am just working on a
> PoC right now which uses the custom watermarks in the DataStream v2 API to
> represent transaction boundaries. It seems this is a great fit
> conceptually. In any case, it would be nice to not only support this for
> MySQL but also other DBs. Debezium provides that transaction metadata for a
> range of connectors, including Postgres.
>
> --Gunnar
>
>
> On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
> wrote:
>
> > Hello,
> >
> > Following up on the below proposal. Would appreciate your thoughts and if
> > we could move forward with this feature.
> >
> > Thank you,
> > Tejansh
> >
> > From: Tejansh Rana <te...@autodesk.com.INVALID>
> > Date: Friday, 17 October 2025 at 15:58
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
> > CDC Connector
> > You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
> > why this is important<https://aka.ms/LearnAboutSenderIdentification>
> >
> > EXTERNAL EMAIL : Do not click any links or open any attachments unless you
> > trust the sender and know the content is safe.
> >
> > Hi team,
> >
> > Following my discussion with Leonard Xu at Flink Forward, I am writing to
> > propose a feature enhancement for the Flink MySQL CDC connector related to
> > how it handles transaction metadata from the MySQL binary log.
> >
> > Problem Statement:
> > In data streaming pipelines that require transactional guarantees or need
> > to group atomic changes together, it is essential to identify the
> > boundaries of the original database transaction (i.e., the BEGIN and COMMIT
> > or END events). Currently, the Flink MySQL CDC connector appears to skip
> > these transaction lifecycle events -
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322297418%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6szuQvjDu7qdG8SVG7xKoZAgQo5zIQhBR25vsE%2F50UU%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322412677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=APMT7%2FgNDJZFza8K9JnG1OvAW1z2qyYzEyoiTI8%2Btx0%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
> > .
> > I have also attached a screenshot of the logs from this behaviour.
> >
> > This omission makes it challenging to reconstruct the original transaction
> > scope. Without explicit transaction markers, downstream Flink jobs cannot
> > easily guarantee atomicity across sinks.
> >
> > Proposed Solution:
> > The underlying CDC mechanism, Debezium, supports emitting transaction
> > boundary events (BEGIN and END/COMMIT) through its configuration.
> >
> > We propose enhancing the Flink MySQL CDC connector to expose this
> > transaction metadata to the Flink pipeline. The connector should emit
> > specialised records or metadata fields that indicate the start and end of a
> > transaction as emitted. We would be happy to create a PR with this feature
> > if this proposal goes ahead.
> >
> > Thank you,
> > Tejansh
> >
> >
> >
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Tejansh Rana <te...@autodesk.com.INVALID>.
Hello,
Bumping this thread.
I have created this Jira ticket describing the proposal and based on Gunnar’s feedback, I have also included the base Source Emitter which would cover this feature for connectors like Postgres - https://issues.apache.org/jira/browse/FLINK-38691
This is the draft PR with proposed changes for MySQL connector - https://github.com/apache/flink-cdc/pull/4170
I would appreciate some feedback on this proposal and I would be happy to contribute the feature.
Thank you,
Tejansh
From: Tejansh Rana <te...@autodesk.com.INVALID>
Date: Tuesday, 4 November 2025 at 16:50
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
[You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
Thank you, Gunnar!
I have created a draft PR for the proposed feature - https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322279639%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJAvbS0TVL8FWX62mqD69pJyyb6PDzd3osZpPy9LawI%3D&reserved=0<https://github.com/apache/flink-cdc/pull/4170>
Looking forward to hearing more feedback.
Thank you,
Tejansh
From: Gunnar Morling <gu...@googlemail.com.INVALID>
Date: Monday, 3 November 2025 at 17:19
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
[You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
Hey all,
I'd love to see support for this! Coincidentally, I am just working on a
PoC right now which uses the custom watermarks in the DataStream v2 API to
represent transaction boundaries. It seems this is a great fit
conceptually. In any case, it would be nice to not only support this for
MySQL but also other DBs. Debezium provides that transaction metadata for a
range of connectors, including Postgres.
--Gunnar
On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
wrote:
> Hello,
>
> Following up on the below proposal. Would appreciate your thoughts and if
> we could move forward with this feature.
>
> Thank you,
> Tejansh
>
> From: Tejansh Rana <te...@autodesk.com.INVALID>
> Date: Friday, 17 October 2025 at 15:58
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
> CDC Connector
> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>
> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
> trust the sender and know the content is safe.
>
> Hi team,
>
> Following my discussion with Leonard Xu at Flink Forward, I am writing to
> propose a feature enhancement for the Flink MySQL CDC connector related to
> how it handles transaction metadata from the MySQL binary log.
>
> Problem Statement:
> In data streaming pipelines that require transactional guarantees or need
> to group atomic changes together, it is essential to identify the
> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
> or END events). Currently, the Flink MySQL CDC connector appears to skip
> these transaction lifecycle events -
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322297418%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6szuQvjDu7qdG8SVG7xKoZAgQo5zIQhBR25vsE%2F50UU%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Ccc84f50fbd34414f286408de1bc24231%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638978718322412677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=APMT7%2FgNDJZFza8K9JnG1OvAW1z2qyYzEyoiTI8%2Btx0%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
> .
> I have also attached a screenshot of the logs from this behaviour.
>
> This omission makes it challenging to reconstruct the original transaction
> scope. Without explicit transaction markers, downstream Flink jobs cannot
> easily guarantee atomicity across sinks.
>
> Proposed Solution:
> The underlying CDC mechanism, Debezium, supports emitting transaction
> boundary events (BEGIN and END/COMMIT) through its configuration.
>
> We propose enhancing the Flink MySQL CDC connector to expose this
> transaction metadata to the Flink pipeline. The connector should emit
> specialised records or metadata fields that indicate the start and end of a
> transaction as emitted. We would be happy to create a PR with this feature
> if this proposal goes ahead.
>
> Thank you,
> Tejansh
>
>
>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Tejansh Rana <te...@autodesk.com.INVALID>.
Thank you, Gunnar!
I have created a draft PR for the proposed feature - https://github.com/apache/flink-cdc/pull/4170
Looking forward to hearing more feedback.
Thank you,
Tejansh
From: Gunnar Morling <gu...@googlemail.com.INVALID>
Date: Monday, 3 November 2025 at 17:19
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
[You don't often get email from gunnar.morling@googlemail.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
Hey all,
I'd love to see support for this! Coincidentally, I am just working on a
PoC right now which uses the custom watermarks in the DataStream v2 API to
represent transaction boundaries. It seems this is a great fit
conceptually. In any case, it would be nice to not only support this for
MySQL but also other DBs. Debezium provides that transaction metadata for a
range of connectors, including Postgres.
--Gunnar
On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
wrote:
> Hello,
>
> Following up on the below proposal. Would appreciate your thoughts and if
> we could move forward with this feature.
>
> Thank you,
> Tejansh
>
> From: Tejansh Rana <te...@autodesk.com.INVALID>
> Date: Friday, 17 October 2025 at 15:58
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
> CDC Connector
> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>
> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
> trust the sender and know the content is safe.
>
> Hi team,
>
> Following my discussion with Leonard Xu at Flink Forward, I am writing to
> propose a feature enhancement for the Flink MySQL CDC connector related to
> how it handles transaction metadata from the MySQL binary log.
>
> Problem Statement:
> In data streaming pipelines that require transactional guarantees or need
> to group atomic changes together, it is essential to identify the
> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
> or END events). Currently, the Flink MySQL CDC connector appears to skip
> these transaction lifecycle events -
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7C24d0f6303e9f49a16dc908de1afd3461%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638977871962762959%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9KT7qKicpHoHobqCjYrjsQ%2FmEDh0WAuO7ahY1X%2FhV7o%3D&reserved=0<https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77>
> .
> I have also attached a screenshot of the logs from this behaviour.
>
> This omission makes it challenging to reconstruct the original transaction
> scope. Without explicit transaction markers, downstream Flink jobs cannot
> easily guarantee atomicity across sinks.
>
> Proposed Solution:
> The underlying CDC mechanism, Debezium, supports emitting transaction
> boundary events (BEGIN and END/COMMIT) through its configuration.
>
> We propose enhancing the Flink MySQL CDC connector to expose this
> transaction metadata to the Flink pipeline. The connector should emit
> specialised records or metadata fields that indicate the start and end of a
> transaction as emitted. We would be happy to create a PR with this feature
> if this proposal goes ahead.
>
> Thank you,
> Tejansh
>
>
>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Gunnar Morling <gu...@googlemail.com.INVALID>.
Hey all,
I'd love to see support for this! Coincidentally, I am just working on a
PoC right now which uses the custom watermarks in the DataStream v2 API to
represent transaction boundaries. It seems this is a great fit
conceptually. In any case, it would be nice to not only support this for
MySQL but also other DBs. Debezium provides that transaction metadata for a
range of connectors, including Postgres.
--Gunnar
On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <te...@autodesk.com.invalid>
wrote:
> Hello,
>
> Following up on the below proposal. Would appreciate your thoughts and if
> we could move forward with this feature.
>
> Thank you,
> Tejansh
>
> From: Tejansh Rana <te...@autodesk.com.INVALID>
> Date: Friday, 17 October 2025 at 15:58
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink
> CDC Connector
> You don't often get email from tejansh.rana@autodesk.com.invalid. Learn
> why this is important<https://aka.ms/LearnAboutSenderIdentification>
>
> EXTERNAL EMAIL : Do not click any links or open any attachments unless you
> trust the sender and know the content is safe.
>
> Hi team,
>
> Following my discussion with Leonard Xu at Flink Forward, I am writing to
> propose a feature enhancement for the Flink MySQL CDC connector related to
> how it handles transaction metadata from the MySQL binary log.
>
> Problem Statement:
> In data streaming pipelines that require transactional guarantees or need
> to group atomic changes together, it is essential to identify the
> boundaries of the original database transaction (i.e., the BEGIN and COMMIT
> or END events). Currently, the Flink MySQL CDC connector appears to skip
> these transaction lifecycle events -
> https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77
> .
> I have also attached a screenshot of the logs from this behaviour.
>
> This omission makes it challenging to reconstruct the original transaction
> scope. Without explicit transaction markers, downstream Flink jobs cannot
> easily guarantee atomicity across sinks.
>
> Proposed Solution:
> The underlying CDC mechanism, Debezium, supports emitting transaction
> boundary events (BEGIN and END/COMMIT) through its configuration.
>
> We propose enhancing the Flink MySQL CDC connector to expose this
> transaction metadata to the Flink pipeline. The connector should emit
> specialised records or metadata fields that indicate the start and end of a
> transaction as emitted. We would be happy to create a PR with this feature
> if this proposal goes ahead.
>
> Thank you,
> Tejansh
>
>
>
Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
Posted by Tejansh Rana <te...@autodesk.com.INVALID>.
Hello,
Following up on the below proposal. Would appreciate your thoughts and if we could move forward with this feature.
Thank you,
Tejansh
From: Tejansh Rana <te...@autodesk.com.INVALID>
Date: Friday, 17 October 2025 at 15:58
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector
You don't often get email from tejansh.rana@autodesk.com.invalid. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe.
Hi team,
Following my discussion with Leonard Xu at Flink Forward, I am writing to propose a feature enhancement for the Flink MySQL CDC connector related to how it handles transaction metadata from the MySQL binary log.
Problem Statement:
In data streaming pipelines that require transactional guarantees or need to group atomic changes together, it is essential to identify the boundaries of the original database transaction (i.e., the BEGIN and COMMIT or END events). Currently, the Flink MySQL CDC connector appears to skip these transaction lifecycle events - https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77 .
I have also attached a screenshot of the logs from this behaviour.
This omission makes it challenging to reconstruct the original transaction scope. Without explicit transaction markers, downstream Flink jobs cannot easily guarantee atomicity across sinks.
Proposed Solution:
The underlying CDC mechanism, Debezium, supports emitting transaction boundary events (BEGIN and END/COMMIT) through its configuration.
We propose enhancing the Flink MySQL CDC connector to expose this transaction metadata to the Flink pipeline. The connector should emit specialised records or metadata fields that indicate the start and end of a transaction as emitted. We would be happy to create a PR with this feature if this proposal goes ahead.
Thank you,
Tejansh