You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MartijnVisser (via GitHub)" <gi...@apache.org> on 2026/09/22 10:26:09 UTC
[PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
MartijnVisser opened a new pull request, #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245
## What is the purpose of the change
- The JDBC Table sink reports an empty lineage dataset name. `TableLineageUtils` asks the object the sink provider returns for lineage at plan time, and `JdbcOutputFormat.getLineageVertex()` read the statement off the executor that only exists after `open()`.
## Brief change log
- `JdbcOutputFormatBuilder` hands the dialect's INSERT statement to `JdbcOutputFormat` as the lineage query, on the upsert and the append branch
- `JdbcOutputFormat.getLineageVertex()` uses that query and keeps the executor's statement as the fallback for the three-argument constructor
## Verifying this change
This change added tests and can be verified as follows:
- `JdbcOutputFormatTest#testGetLineageVertex` asserts before and after `open()`, and `JdbcDynamicTableSinkITCase#testLineageDatasetNameIsKnownAtPlanTime` goes through the provider the way the planner does, for a keyed and a keyless table. Both fail on 99b1666 with `""` and pass with this change.
- `JdbcDialectTest#testInsertStatementNamesTheLineageTable` checks per dialect that the INSERT parses to the table name, quoting included.
- Green on Derby, MySQL and Postgres.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
Release note: the lineage dataset name reported by the JDBC Table sink is now the physical table name instead of an empty string. OpenLineage consumers will see a new dataset node rather than a renamed one.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Fable 5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#issuecomment-5832456118
I can check on Monday. Will try to do before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "eskabetxe (via GitHub)" <gi...@apache.org>.
eskabetxe commented on PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#issuecomment-6081943058
LFTM
Will you also fix on source? as it still reposts OpenLineage's qualified name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#discussion_r4154103641
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/internal/JdbcOutputFormat.java:
##########
@@ -256,10 +271,12 @@ public Connection getConnection() {
@Override
public LineageVertex getLineageVertex() {
+ String query = lineageQuery;
Review Comment:
Intentional, and it's now the table name itself. Flink only reads lineage before `open()`, and the planner still uses the catalog name (FLINK-39935), so I dropped the release note.
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/internal/JdbcOutputFormat.java:
##########
@@ -90,9 +92,22 @@ public JdbcOutputFormat(
@Nonnull JdbcConnectionProvider connectionProvider,
@Nonnull JdbcExecutionOptions executionOptions,
@Nonnull StatementExecutorFactory<JdbcExec> statementExecutorFactory) {
+ this(connectionProvider, executionOptions, statementExecutorFactory, null);
Review Comment:
The DataStream `JdbcSink` has its own `getLineageVertex()`, which parses the query statement directly. `JdbcWriter` only creates this format on the task manager, where nobody reads lineage.
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/sink/JdbcOutputFormatBuilder.java:
##########
@@ -86,19 +88,24 @@ public JdbcOutputFormatBuilder setFieldDataTypes(DataType[] fieldDataTypes) {
Arrays.stream(fieldDataTypes)
.map(DataType::getLogicalType)
.toArray(LogicalType[]::new);
+ final String sql =
+ dmlOptions
+ .getDialect()
+ .getInsertIntoStatement(
+ dmlOptions.getTableName(), dmlOptions.getFieldNames());
+ // Lineage is read at plan time, before open(). The plain INSERT names the target table in
+ // every dialect; the upsert forms do not all parse.
+ final String lineageQuery =
+ FieldNamedPreparedStatementImpl.parseNamedStatement(sql, new HashMap<>());
Review Comment:
Turns out no parsing is needed here: the builder already has the table name, so it passes that directly and no longer touches the Impl class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#issuecomment-6080043344
@eskabetxe Can you take one more look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "eskabetxe (via GitHub)" <gi...@apache.org>.
eskabetxe commented on code in PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#discussion_r4130868167
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/internal/JdbcOutputFormat.java:
##########
@@ -256,10 +271,12 @@ public Connection getConnection() {
@Override
public LineageVertex getLineageVertex() {
+ String query = lineageQuery;
Review Comment:
Since lineageQuery takes precedence over the executor's statement, post-open() lineage for upsert sinks on dialects with native upsert (e.g. Postgres ON CONFLICT) now reports the plain INSERT instead of the upsert statement. Same table name and more consistent plan-time vs runtime, just confirming it's intentional and worth a line in the PR description/release note.
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/internal/JdbcOutputFormat.java:
##########
@@ -90,9 +92,22 @@ public JdbcOutputFormat(
@Nonnull JdbcConnectionProvider connectionProvider,
@Nonnull JdbcExecutionOptions executionOptions,
@Nonnull StatementExecutorFactory<JdbcExec> statementExecutorFactory) {
+ this(connectionProvider, executionOptions, statementExecutorFactory, null);
Review Comment:
Nit/out-of-scope note: the DataStream JdbcSink path still goes through this constructor, so its getLineageVertex() remains empty before open() — same plan-time bug for LineageVertexProvider consumers of DataStream jobs. Not a blocker here; suggest a follow-up JIRA to thread a lineage query through JdbcSink/JdbcSinkBuilder too.
##########
flink-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/core/table/sink/JdbcOutputFormatBuilder.java:
##########
@@ -86,19 +88,24 @@ public JdbcOutputFormatBuilder setFieldDataTypes(DataType[] fieldDataTypes) {
Arrays.stream(fieldDataTypes)
.map(DataType::getLogicalType)
.toArray(LogicalType[]::new);
+ final String sql =
+ dmlOptions
+ .getDialect()
+ .getInsertIntoStatement(
+ dmlOptions.getTableName(), dmlOptions.getFieldNames());
+ // Lineage is read at plan time, before open(). The plain INSERT names the target table in
+ // every dialect; the upsert forms do not all parse.
+ final String lineageQuery =
+ FieldNamedPreparedStatementImpl.parseNamedStatement(sql, new HashMap<>());
Review Comment:
Pulling the parsed statement via a static on FieldNamedPreparedStatementImpl works, but a tiny helper like LineageUtils.normalizedQuery(sql) (or moving parseNamedStatement to the non-Impl utility) would keep the builder from depending on an Impl class. Either way fine to leave as is since the static already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#issuecomment-5929025477
Rebased, and the format now reports the table name the builder passes instead of parsing any SQL, which also keeps OpenLineage's native parser out of the sink's planning. The dialect test went with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40743][jdbc] Fix the Table sink's empty lineage dataset name [flink-connector-jdbc]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #245:
URL: https://github.com/apache/flink-connector-jdbc/pull/245#issuecomment-5831411969
@eskabetxe @spuru9 Want to take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org