You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gustavodemorais (via GitHub)" <gi...@apache.org> on 2026/08/28 13:53:32 UTC
[PR] [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR [flink]
gustavodemorais opened a new pull request, #29040:
URL: https://github.com/apache/flink/pull/29040
## What is the purpose of the change
A materialized table has no syntax for an explicit `ON CONFLICT` clause, so it always falls back to the implicit `DO DEDUPLICATE` strategy when its primary key differs from its query's upsert key. This adds an opt-in option to default it to `DO ERROR` instead, avoiding the state-heavier deduplicating materializer, applied only when every source already declares a watermark so table creation never fails because of it.
## Brief change log
- Add TABLE_EXEC_SINK_MATERIALIZED_TABLE_FORCES_ON_CONFLICT_ERROR, default false.
- convertMaterializedTableAsToRel picks ON CONFLICT DO ERROR when the option is enabled and every source has a watermark, otherwise keeps the previous behavior.
- Add allSourcesHaveWatermarks, walking the logical input tree to check each source's watermark spec.
- Thread getTableConfig through PlannerBase's call to convertMaterializedTableAsToRel.
- Tailor the require-on-conflict validation message with a materialized-table-specific hint when the new option is enabled but a source is missing a watermark.
- Document the new option and its interaction with require-on-conflict in the INSERT reference docs.
## Verifying this change
- ExplainTest
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - new option added to ExecutionConfigOptions (`@PublicEvolving`)
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? (docs/content/docs/sql/reference/dml/insert.md, docs/content.zh/docs/sql/reference/dml/insert.md, generated config option docs)
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes (please specify the tool below)
2.1.235 (Claude Code) with Sonnet 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29040:
URL: https://github.com/apache/flink/pull/29040#discussion_r3881284559
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSinkUtils.java:
##########
@@ -285,7 +299,28 @@ public static RelNode convertMaterializedTableAsToRel(
null,
isOverwrite,
sink,
- null);
+ conflictStrategy);
+ }
+
+ /** Whether every table scanned by {@code rel} declares a watermark. */
+ private static boolean allSourcesHaveWatermarks(RelNode rel) {
+ if (rel instanceof TableScan) {
+ final TableSourceTable table =
+ ((TableScan) rel).getTable().unwrap(TableSourceTable.class);
+ // An unresolvable scan can't be proven to lack a watermark, so it isn't treated as a
+ // reason to fall back - matching the same convention used for the physical-tree check.
+ return table == null
+ || !table.contextResolvedTable()
+ .getResolvedSchema()
+ .getWatermarkSpecs()
+ .isEmpty();
+ }
+ for (RelNode input : rel.getInputs()) {
+ if (!allSourcesHaveWatermarks(input)) {
+ return false;
+ }
+ }
+ return true;
}
Review Comment:
Also noticed that `collectSourcesWithoutWatermarks` does the same check
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSinkUtils.java:
##########
@@ -285,7 +299,28 @@ public static RelNode convertMaterializedTableAsToRel(
null,
isOverwrite,
sink,
- null);
+ conflictStrategy);
+ }
+
+ /** Whether every table scanned by {@code rel} declares a watermark. */
+ private static boolean allSourcesHaveWatermarks(RelNode rel) {
+ if (rel instanceof TableScan) {
+ final TableSourceTable table =
+ ((TableScan) rel).getTable().unwrap(TableSourceTable.class);
+ // An unresolvable scan can't be proven to lack a watermark, so it isn't treated as a
+ // reason to fall back - matching the same convention used for the physical-tree check.
+ return table == null
+ || !table.contextResolvedTable()
+ .getResolvedSchema()
+ .getWatermarkSpecs()
+ .isEmpty();
+ }
+ for (RelNode input : rel.getInputs()) {
+ if (!allSourcesHaveWatermarks(input)) {
+ return false;
+ }
+ }
+ return true;
}
Review Comment:
We can simplify this
```java
private static boolean allSourcesHaveWatermarks(RelNode rel) {
if (rel instanceof TableScan) {
final TableSourceTable table =
((TableScan) rel).getTable().unwrap(TableSourceTable.class);
return table == null
|| !table.contextResolvedTable().getResolvedSchema().getWatermarkSpecs().isEmpty();
}
return rel.getInputs().stream().allMatch(DynamicSinkUtils::allSourcesHaveWatermarks);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais closed pull request #29040: [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR
URL: https://github.com/apache/flink/pull/29040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on PR #29040:
URL: https://github.com/apache/flink/pull/29040#issuecomment-5512477861
Won't do for now. So closing this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29040:
URL: https://github.com/apache/flink/pull/29040#issuecomment-5453490156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "49a67f39db69b749819aed11aa1ede5a80d98c54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49a67f39db69b749819aed11aa1ede5a80d98c54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49a67f39db69b749819aed11aa1ede5a80d98c54 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org