You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jaskaran Singh Kukreja (Jira)" <ji...@apache.org> on 2026/05/26 13:47:00 UTC
[jira] [Updated] (FLINK-39765) Add FLIP-314 lineage support (LineageVertexProvider) to MySQL CDC source connector
[ https://issues.apache.org/jira/browse/FLINK-39765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaskaran Singh Kukreja updated FLINK-39765:
-------------------------------------------
Description:
*Summary*
Implement Flink's {{LineageVertexProvider}} interface (FLIP-314) on {{MySqlSource}} to enable native lineage reporting.
*Motivation*
Flink 1.20 introduced FLIP-314 which provides a native lineage API via {{{}LineageVertexProvider{}}}. OpenLineage's Flink integration uses this to extract source/sink datasets and emit lineage events. Currently, none of the CDC source connectors implement {{{}LineageVertexProvider{}}}, so OpenLineage events emitted from CDC pipelines contain empty inputs.
*Proposed Changes*
Implement {{LineageVertexProvider}} on {{MySqlSource}} to report captured tables as input datasets with their schemas. The scope of this ticket targets MySQL only, but shared lineage utilities will be added to {{flink-cdc-common}} (e.g., {{{}LineageUtils{}}}, {{{}CdcSourceLineageVertex{}}}, {{{}CdcLineageDataset{}}}) so that other source connectors (Postgres, Oracle, etc.) can easily adopt lineage support in follow-up PRs.
*Result*
Each captured MySQL table is reported as an input dataset using Flink native APIs with openlineage standard:
* {*}Namespace{*}: {{mysql://hostname:port}}
* {*}Name{*}: resolved table name (e.g., {{{}mydb.users{}}})
* {*}Config facet{*}: source type ({{{}mysql-cdc{}}})
* {*}Schema facet{*}: column names and MySQL types
was:
*Summary*
Implement Flink's {{LineageVertexProvider}} interface (FLIP-314) on {{MySqlSource}} to enable native lineage reporting.
*Motivation*
Flink 1.20 introduced FLIP-314 which provides a native lineage API via {{{}LineageVertexProvider{}}}. OpenLineage's Flink integration uses this to extract source/sink datasets and emit lineage events. Currently, none of the CDC source connectors implement {{{}LineageVertexProvider{}}}, so OpenLineage events emitted from CDC pipelines contain empty inputs.
*Proposed Changes*
Implement {{LineageVertexProvider}} on {{MySqlSource}} to report captured tables as input datasets with their schemas. The scope of this ticket targets MySQL only, but shared lineage utilities will be added to {{flink-cdc-common}} (e.g., {{{}LineageUtils{}}}, {{{}CdcSourceLineageVertex{}}}, {{{}CdcLineageDataset{}}}) so that other source connectors (Postgres, Oracle, etc.) can easily adopt lineage support in follow-up PRs.
*Result*
**
Each captured MySQL table is reported as an input dataset using Flink native APIs with openlineage standard:
* {*}Namespace{*}: {{mysql://hostname:port}}
* {*}Name{*}: resolved table name (e.g., {{{}mydb.users{}}})
* {*}Config facet{*}: source type ({{{}mysql-cdc{}}})
* {*}Schema facet{*}: column names and MySQL types
> Add FLIP-314 lineage support (LineageVertexProvider) to MySQL CDC source connector
> ----------------------------------------------------------------------------------
>
> Key: FLINK-39765
> URL: https://issues.apache.org/jira/browse/FLINK-39765
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: Jaskaran Singh Kukreja
> Priority: Minor
> Labels: Flink-CDC
>
> *Summary*
> Implement Flink's {{LineageVertexProvider}} interface (FLIP-314) on {{MySqlSource}} to enable native lineage reporting.
> *Motivation*
> Flink 1.20 introduced FLIP-314 which provides a native lineage API via {{{}LineageVertexProvider{}}}. OpenLineage's Flink integration uses this to extract source/sink datasets and emit lineage events. Currently, none of the CDC source connectors implement {{{}LineageVertexProvider{}}}, so OpenLineage events emitted from CDC pipelines contain empty inputs.
> *Proposed Changes*
> Implement {{LineageVertexProvider}} on {{MySqlSource}} to report captured tables as input datasets with their schemas. The scope of this ticket targets MySQL only, but shared lineage utilities will be added to {{flink-cdc-common}} (e.g., {{{}LineageUtils{}}}, {{{}CdcSourceLineageVertex{}}}, {{{}CdcLineageDataset{}}}) so that other source connectors (Postgres, Oracle, etc.) can easily adopt lineage support in follow-up PRs.
> *Result*
> Each captured MySQL table is reported as an input dataset using Flink native APIs with openlineage standard:
> * {*}Namespace{*}: {{mysql://hostname:port}}
> * {*}Name{*}: resolved table name (e.g., {{{}mydb.users{}}})
> * {*}Config facet{*}: source type ({{{}mysql-cdc{}}})
> * {*}Schema facet{*}: column names and MySQL types
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)