You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/05/15 06:41:19 UTC
[PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
raminqaf opened a new pull request, #28164:
URL: https://github.com/apache/flink/pull/28164
## What is the purpose of the change
`FROM_CHANGELOG` now emits an upsert changelog (`INSERT`, `UPDATE_AFTER`, full `DELETE`) when the input table is partitioned (set semantics via `PARTITION BY`) and the active `op_mapping` maps to `UPDATE_AFTER` without `UPDATE_BEFORE`. The partition key acts as the upsert key. In all other cases the output remains a retract changelog.
Submitting an `op_mapping` with `UPDATE_AFTER` but no` UPDATE_BEFORE` without `PARTITION BY` is rejected at validation time, because upsert mode requires a key.
To enable the strategy to inspect the resolved op_mapping and the input table's partition keys, ChangelogFunction.ChangelogContext is extended with two default methods: getArgumentValue(int, Class) and getTableSemantics(int). Defaults return Optional.empty() to preserve source compatibility for existing implementations.
The planner-side wrapper in FlinkChangelogModeInferenceProgram delegates the two new methods to the underlying CallContext.
Upsert mode uses full deletes (ChangelogMode.upsert(false)) because the runtime forwards each input delete row with all fields populated; only the RowKind is rewritten. This matches the runtime's behavior and avoids forcing downstream operators to reconstruct rows from state.
The upsert key derivation in FlinkRelMdUniqueKeys.getPtfUniqueKeys already returns the partition columns when a PTF emits upsert, so no metadata changes are needed.
## Verifying this change
Changes are backed by semantic tests and plan verification tests
- Added semantics tests in `FromChangelogTestProgram`
- Added tests in `FromChangelogTest` to verify generated plan and upsertKeys
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs / JavaDocs )
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [X] Yes (please specify the tool below)
Generated-by: Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3258993550
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
Review Comment:
```suggestion
#### Upsert table
```
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
+
+```sql
+-- Upsert input: INSERT / UPDATE_AFTER / DELETE only
+-- +I[id:1, op:'INSERT', name:'Alice']
+-- +I[id:2, op:'INSERT', name:'Bob']
+-- +I[id:1, op:'UPDATE_AFTER', name:'Alice2']
+-- +I[id:2, op:'DELETE', name:'Bob']
+
+SELECT * FROM FROM_CHANGELOG(
+ input => TABLE cdc_stream PARTITION BY id,
+ op_mapping => MAP[
+ 'INSERT', 'INSERT',
+ 'UPDATE_AFTER', 'UPDATE_AFTER',
+ 'DELETE', 'DELETE']
+)
+
+-- Output (upsert changelog, upsert key = id):
+-- +I[id:1, name:'Alice']
+-- +I[id:2, name:'Bob']
+-- +U[id:1, name:'Alice2']
+-- -D[id:2, name:'Bob']
+```
+
+Without `PARTITION BY`, or when the active `op_mapping` includes `UPDATE_BEFORE`, the output remains a retract changelog.
Review Comment:
```suggestion
By default, without `PARTITION BY`, or when the active `op_mapping` includes `UPDATE_BEFORE`, the output remains a retract changelog.
```
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
+ * UPDATE_AFTER}, and {@code DELETE}.
+ * <li><b>Upsert</b>: the {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code
+ * UPDATE_BEFORE}. The output emits {@code INSERT}, {@code UPDATE_AFTER}, and full {@code
Review Comment:
```suggestion
* UPDATE_BEFORE}. The output emits {@code INSERT}, {@code UPDATE_AFTER}, and {@code
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
Review Comment:
```suggestion
* <p>Upsert mode uses full deletes by default ({@link ChangelogMode#upsert(boolean) upsert(false)}). We currently only support consuming from a changelog stream with full deletes (where not only the primary keys, but all fields fields are populated).
```
nit: we could maybe even delete this and add only more info when we add "consume_full_updates"
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java:
##########
@@ -182,10 +183,26 @@ public RelWriter explainTerms(RelWriter pw) {
for (Ord<RelNode> ord : Ord.zip(inputs)) {
pw.input("input#" + ord.i, ord.e);
}
- return pw.item("invocation", scan.getCall())
+ pw.item("invocation", scan.getCall())
.item("uid", uid)
.item("select", String.join(",", getRowType().getFieldNames()))
.item("rowType", getRowType());
+ final Set<ImmutableBitSet> upsertKeys =
Review Comment:
I don't think you want to do this. This would change the plan output for all PTFs and you'd theoretically have to regenerate all PTF tests in the code base.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java:
##########
@@ -182,10 +183,26 @@ public RelWriter explainTerms(RelWriter pw) {
for (Ord<RelNode> ord : Ord.zip(inputs)) {
pw.input("input#" + ord.i, ord.e);
}
- return pw.item("invocation", scan.getCall())
+ pw.item("invocation", scan.getCall())
.item("uid", uid)
.item("select", String.join(",", getRowType().getFieldNames()))
.item("rowType", getRowType());
+ final Set<ImmutableBitSet> upsertKeys =
Review Comment:
You just want to output the upsertKeys in the plan, right? Then you want to use/thinker your tests to use the testing utility that allows you do so instead of changing the default explain for ptf nodes
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
Review Comment:
```suggestion
* or only {@code INSERT} and {@code DELETE} pairs. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
```
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
Review Comment:
```suggestion
To generate an `Upsert table`, the following requirements must be met:
* **Key Partitioning:** You must use `PARTITION BY <key>`, where the partition key corresponds to the unique/primary key of the dataset.
* **Op Mapping Configuration:** The `op_mapping` must include `UPDATE_AFTER` and must NOT include `UPDATE_BEFORE`.
**How it works:**
The engine assumes that the keys provided in the `PARTITION BY` clause function as the unique upsert keys. The resulting output changelog becomes an upsert table keyed on these partition columns. Each incoming row is evaluated and produces `INSERT`, `UPDATE_AFTER`, or `DELETE` events, using the partition key as the explicit upsert key. Therefore, if the incoming changelog contains unique keys (such as a primary key), they **must** be used in the `PARTITION BY` clause.
The FROM_CHANGELOG PTF assumes events arrived ordered. If the source itself does not guarantee ordering for events for the same PARTITION BY keys, consider using using ORDER BY <link>.
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
Review Comment:
```suggestion
* or no updates at all. The output possibly emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
```
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
+ * UPDATE_AFTER}, and {@code DELETE}.
+ * <li><b>Upsert</b>: the {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code
Review Comment:
```suggestion
* <li><b>Upsert</b>: the {@code op_mapping} maps {@code UPDATE_AFTER} without {@code
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal, since the default mapping includes both (retract).
Review Comment:
```suggestion
* cannot be resolved as a literal. The default mapping maps to retract table.
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
Review Comment:
The whole change here in this file is very nice and easy to read 🙂
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
Review Comment:
This is a fair point to keep this simple. I thought a bit and still think it'd be good to have the section. We have retract as the default mode. The changelog mode of a table is an user facing option which users are aware which users are often dealing with. So a section telling them what they have to pay attention to when they want to fabricate an upsert table makes sense imo
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/FromChangelogTestPrograms.java:
##########
@@ -200,6 +200,69 @@ public class FromChangelogTestPrograms {
+ "input => TABLE cdc_stream PARTITION BY id)")
.build();
+ public static final TableTestProgram UPSERT_PARTITION_BY =
+ TableTestProgram.of(
+ "from-changelog-upsert-partition-by",
+ "PARTITION BY + op_mapping without UPDATE_BEFORE produces an "
+ + "upsert changelog keyed on the partition columns")
+ .setupTableSource(
+ SourceTestStep.newBuilder("cdc_stream")
+ .addSchema("name STRING", "id INT", "op STRING")
+ .producedValues(
+ Row.of("Alice", 1, "INSERT"),
+ Row.of("Bob", 2, "INSERT"),
+ Row.of("Alice2", 1, "UPDATE_AFTER"),
+ Row.of("Bob", 2, "DELETE"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .consumedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.UPDATE_AFTER, 1, "Alice2"),
+ Row.ofKind(RowKind.DELETE, 2, "Bob"))
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM FROM_CHANGELOG("
+ + "input => TABLE cdc_stream PARTITION BY id, "
+ + "op_mapping => MAP["
+ + "'INSERT', 'INSERT', "
+ + "'UPDATE_AFTER', 'UPDATE_AFTER', "
+ + "'DELETE', 'DELETE'])")
+ .build();
+
+ public static final TableTestProgram UPSERT_PARTITION_BY_CUSTOM_MAPPING =
Review Comment:
Agree with @fhueske
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3264453714
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +100,56 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal. The default mapping maps to a retract table.
+ */
+ private static boolean isUpsertConfig(final ChangelogContext ctx) {
+ if (!isPartitioned(ctx::getTableSemantics)) {
+ return false;
+ }
+ return ctx.getArgumentValue(ARG_OP_MAPPING, Map.class).stream()
+ .anyMatch(FromChangelogTypeStrategy::describesUpsert);
Review Comment:
Thanks! Addressed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260546159
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +100,56 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal. The default mapping maps to a retract table.
+ */
+ private static boolean isUpsertConfig(final ChangelogContext ctx) {
+ if (!isPartitioned(ctx::getTableSemantics)) {
+ return false;
+ }
+ return ctx.getArgumentValue(ARG_OP_MAPPING, Map.class).stream()
+ .anyMatch(FromChangelogTypeStrategy::describesUpsert);
Review Comment:
nit
```suggestion
return ctx.getArgumentValue(ARG_OP_MAPPING, Map.class).map(FromChangelogTypeStrategy::describesUpsert).orElse(false);
```
`stream()` of an `Optional` surely works, but kinda hides that this is just a single element.
`map()` is more clear, IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260032413
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
Review Comment:
Added
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260059081
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java:
##########
@@ -182,10 +183,26 @@ public RelWriter explainTerms(RelWriter pw) {
for (Ord<RelNode> ord : Ord.zip(inputs)) {
pw.input("input#" + ord.i, ord.e);
}
- return pw.item("invocation", scan.getCall())
+ pw.item("invocation", scan.getCall())
.item("uid", uid)
.item("select", String.join(",", getRowType().getFieldNames()))
.item("rowType", getRowType());
+ final Set<ImmutableBitSet> upsertKeys =
Review Comment:
Reverted this and now stringify the plan through `TableTestBase`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28164:
URL: https://github.com/apache/flink/pull/28164#issuecomment-4496432982
Merging this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3267581005
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -1057,7 +1073,8 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
withRowType,
expectedPlans,
() => assertEqualsOrExpand("sql", query),
- withQueryBlockAlias)
+ withQueryBlockAlias,
+ withUpsertKey = withUpsertKey)
Review Comment:
Addressed
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -614,7 +614,32 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
/** Java-friendly overload that accepts a list of [[ExplainDetail]]s. */
def verifyRelPlan(query: String, extraDetails: java.util.List[ExplainDetail]): Unit = {
- verifyRelPlan(query, extraDetails.asScala.toSeq: _*)
+ verifyRelPlan(query, extraDetails.asScala: _*)
+ }
+
+ /**
+ * Verify the AST and the optimized rel plan for the given SELECT query. The rendered optimized
+ * rel plan includes the `upsertKeys=[...]` term for rel nodes that derive upsert keys.
+ */
+ def verifyRelPlanWithUpsertKey(query: String, extraDetails: ExplainDetail*): Unit = {
Review Comment:
Improved the API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260027353
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3265110228
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -614,7 +614,32 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
/** Java-friendly overload that accepts a list of [[ExplainDetail]]s. */
def verifyRelPlan(query: String, extraDetails: java.util.List[ExplainDetail]): Unit = {
- verifyRelPlan(query, extraDetails.asScala.toSeq: _*)
+ verifyRelPlan(query, extraDetails.asScala: _*)
+ }
+
+ /**
+ * Verify the AST and the optimized rel plan for the given SELECT query. The rendered optimized
+ * rel plan includes the `upsertKeys=[...]` term for rel nodes that derive upsert keys.
+ */
+ def verifyRelPlanWithUpsertKey(query: String, extraDetails: ExplainDetail*): Unit = {
Review Comment:
This does not seem to fit the style of the utilities we have here. We don have unique verifyRelPlan for each param we can configure. Take a look at how withDuplicateChanges and withQueryBlockAlias were implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske merged PR #28164:
URL: https://github.com/apache/flink/pull/28164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3271568200
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -1057,7 +1073,8 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
withRowType,
expectedPlans,
() => assertEqualsOrExpand("sql", query),
- withQueryBlockAlias)
+ withQueryBlockAlias,
+ withUpsertKey = withUpsertKey)
Review Comment:
@gustavodemorais assertPlanEquals param order is (..., withQueryBlockAlias, withDuplicateChanges, withUpsertKey). Passing withUpsertKey positionally at slot 7 bound it to withDuplicateChanges.
This produces a new plan and breaks the test. I will revert to the named arg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3259736698
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal, since the default mapping includes both (retract).
+ */
+ @SuppressWarnings("unchecked")
+ public static boolean isUpsertConfig(final ChangelogContext ctx) {
Review Comment:
This can be (package-)private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260029971
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
+
+```sql
+-- Upsert input: INSERT / UPDATE_AFTER / DELETE only
+-- +I[id:1, op:'INSERT', name:'Alice']
+-- +I[id:2, op:'INSERT', name:'Bob']
+-- +I[id:1, op:'UPDATE_AFTER', name:'Alice2']
+-- +I[id:2, op:'DELETE', name:'Bob']
+
+SELECT * FROM FROM_CHANGELOG(
+ input => TABLE cdc_stream PARTITION BY id,
+ op_mapping => MAP[
+ 'INSERT', 'INSERT',
+ 'UPDATE_AFTER', 'UPDATE_AFTER',
+ 'DELETE', 'DELETE']
+)
+
+-- Output (upsert changelog, upsert key = id):
+-- +I[id:1, name:'Alice']
+-- +I[id:2, name:'Bob']
+-- +U[id:1, name:'Alice2']
+-- -D[id:2, name:'Bob']
+```
+
+Without `PARTITION BY`, or when the active `op_mapping` includes `UPDATE_BEFORE`, the output remains a retract changelog.
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260039142
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
+ * UPDATE_AFTER}, and {@code DELETE}.
+ * <li><b>Upsert</b>: the {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28164:
URL: https://github.com/apache/flink/pull/28164#issuecomment-4457632553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "80a2c1bf5b63e1e4b3b75551bf06ac1e9b3ae51a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "80a2c1bf5b63e1e4b3b75551bf06ac1e9b3ae51a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80a2c1bf5b63e1e4b3b75551bf06ac1e9b3ae51a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3259867576
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal, since the default mapping includes both (retract).
+ */
+ @SuppressWarnings("unchecked")
+ public static boolean isUpsertConfig(final ChangelogContext ctx) {
+ final boolean partitioned =
+ ctx.getTableSemantics(ARG_TABLE)
+ .map(ts -> ts.partitionByColumns().length > 0)
+ .orElse(false);
+ if (!partitioned) {
+ return false;
+ }
+ final Optional<Map> opMapping = ctx.getArgumentValue(ARG_OP_MAPPING, Map.class);
+ if (opMapping.isEmpty()) {
+ return false;
+ }
+ final Map<String, String> mapping = opMapping.get();
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
Review Comment:
good catch! I filter the nulls before motching
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
+
+```sql
+-- Upsert input: INSERT / UPDATE_AFTER / DELETE only
+-- +I[id:1, op:'INSERT', name:'Alice']
+-- +I[id:2, op:'INSERT', name:'Bob']
+-- +I[id:1, op:'UPDATE_AFTER', name:'Alice2']
+-- +I[id:2, op:'DELETE', name:'Bob']
+
+SELECT * FROM FROM_CHANGELOG(
+ input => TABLE cdc_stream PARTITION BY id,
+ op_mapping => MAP[
+ 'INSERT', 'INSERT',
+ 'UPDATE_AFTER', 'UPDATE_AFTER',
+ 'DELETE', 'DELETE']
+)
+
+-- Output (upsert changelog, upsert key = id):
+-- +I[id:1, name:'Alice']
+-- +I[id:2, name:'Bob']
+-- +U[id:1, name:'Alice2']
+-- -D[id:2, name:'Bob']
+```
+
+Without `PARTITION BY`, or when the active `op_mapping` includes `UPDATE_BEFORE`, the output remains a retract changelog.
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3257326993
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
Review Comment:
I'm wondering to what extend we need to document this behavior.
`FROM_CHANGELOG` converts and explicit changelog into a Flink dynamic table that is internally handled. From a user's point of view, does it really matter how Flink encodes changes? It shouldn't change the semantics of the following SQL operations. Is using `upsert` changelog mode not "just" an internal optimization?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal, since the default mapping includes both (retract).
+ */
+ @SuppressWarnings("unchecked")
+ public static boolean isUpsertConfig(final ChangelogContext ctx) {
+ final boolean partitioned =
+ ctx.getTableSemantics(ARG_TABLE)
+ .map(ts -> ts.partitionByColumns().length > 0)
+ .orElse(false);
+ if (!partitioned) {
+ return false;
+ }
+ final Optional<Map> opMapping = ctx.getArgumentValue(ARG_OP_MAPPING, Map.class);
+ if (opMapping.isEmpty()) {
+ return false;
+ }
+ final Map<String, String> mapping = opMapping.get();
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
Review Comment:
was already checked that `v` isn't null or might `v.trim()` throw a NPE?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
Review Comment:
The return type of this method looks strange.
It never returns an actual `List`, only `Optional.empty()`.
I'd rather return a boolean and move the error message into `validateOpMapping()`.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/FromChangelogTestPrograms.java:
##########
@@ -200,6 +200,69 @@ public class FromChangelogTestPrograms {
+ "input => TABLE cdc_stream PARTITION BY id)")
.build();
+ public static final TableTestProgram UPSERT_PARTITION_BY =
+ TableTestProgram.of(
+ "from-changelog-upsert-partition-by",
+ "PARTITION BY + op_mapping without UPDATE_BEFORE produces an "
+ + "upsert changelog keyed on the partition columns")
+ .setupTableSource(
+ SourceTestStep.newBuilder("cdc_stream")
+ .addSchema("name STRING", "id INT", "op STRING")
+ .producedValues(
+ Row.of("Alice", 1, "INSERT"),
+ Row.of("Bob", 2, "INSERT"),
+ Row.of("Alice2", 1, "UPDATE_AFTER"),
+ Row.of("Bob", 2, "DELETE"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .consumedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.UPDATE_AFTER, 1, "Alice2"),
+ Row.ofKind(RowKind.DELETE, 2, "Bob"))
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM FROM_CHANGELOG("
+ + "input => TABLE cdc_stream PARTITION BY id, "
+ + "op_mapping => MAP["
+ + "'INSERT', 'INSERT', "
+ + "'UPDATE_AFTER', 'UPDATE_AFTER', "
+ + "'DELETE', 'DELETE'])")
+ .build();
+
+ public static final TableTestProgram UPSERT_PARTITION_BY_CUSTOM_MAPPING =
Review Comment:
Isn't the only difference to the other added test that the mapping uses "c" instead of "INSERT"?
Isn't that essentially the same test?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
+ final boolean hasUpdateBefore =
+ mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
+ if (!hasUpdateAfter || hasUpdateBefore) {
Review Comment:
Add a comment describing the rational of this condition.
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
+ */
+ public static final ChangelogModeStrategy CHANGELOG_MODE_STRATEGY =
+ ctx -> isUpsertConfig(ctx) ? ChangelogMode.upsert(false) : ChangelogMode.all();
+
+ /**
+ * Returns {@code true} when the FROM_CHANGELOG call should emit an upsert changelog: the input
+ * table is partitioned AND the resolved {@code op_mapping} contains {@code UPDATE_AFTER}
+ * without {@code UPDATE_BEFORE}. Falls back to {@code false} when the mapping is absent or
+ * cannot be resolved as a literal, since the default mapping includes both (retract).
+ */
+ @SuppressWarnings("unchecked")
+ public static boolean isUpsertConfig(final ChangelogContext ctx) {
Review Comment:
does this function need to be `public`?
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,34 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert output
+
+When `PARTITION BY` is combined with an `op_mapping` that does NOT include `UPDATE_BEFORE`, the output changelog is an upsert table keyed on the partition columns. Each input row produces an `INSERT`, `UPDATE_AFTER`, or `DELETE` event with the partition key acting as the upsert key.
+
+```sql
+-- Upsert input: INSERT / UPDATE_AFTER / DELETE only
+-- +I[id:1, op:'INSERT', name:'Alice']
+-- +I[id:2, op:'INSERT', name:'Bob']
+-- +I[id:1, op:'UPDATE_AFTER', name:'Alice2']
+-- +I[id:2, op:'DELETE', name:'Bob']
+
+SELECT * FROM FROM_CHANGELOG(
+ input => TABLE cdc_stream PARTITION BY id,
+ op_mapping => MAP[
+ 'INSERT', 'INSERT',
+ 'UPDATE_AFTER', 'UPDATE_AFTER',
+ 'DELETE', 'DELETE']
+)
+
+-- Output (upsert changelog, upsert key = id):
+-- +I[id:1, name:'Alice']
+-- +I[id:2, name:'Bob']
+-- +U[id:1, name:'Alice2']
+-- -D[id:2, name:'Bob']
+```
+
+Without `PARTITION BY`, or when the active `op_mapping` includes `UPDATE_BEFORE`, the output remains a retract changelog.
Review Comment:
Document that validation fails if there is no `PARTITION BY` and no `UPDATE_BEFORE`?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
+ final boolean hasUpdateBefore =
+ mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
+ if (!hasUpdateAfter || hasUpdateBefore) {
+ return Optional.empty();
+ }
+ final boolean partitioned =
+ callContext
+ .getTableSemantics(ARG_TABLE)
+ .map(ts -> ts.partitionByColumns().length > 0)
+ .orElse(false);
Review Comment:
Could be extracted into a helper method (is also used in `isUpsertConfig()`
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
Review Comment:
same question about NPE as above
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/FromChangelogTestPrograms.java:
##########
@@ -200,6 +200,69 @@ public class FromChangelogTestPrograms {
+ "input => TABLE cdc_stream PARTITION BY id)")
.build();
+ public static final TableTestProgram UPSERT_PARTITION_BY =
+ TableTestProgram.of(
+ "from-changelog-upsert-partition-by",
+ "PARTITION BY + op_mapping without UPDATE_BEFORE produces an "
+ + "upsert changelog keyed on the partition columns")
+ .setupTableSource(
+ SourceTestStep.newBuilder("cdc_stream")
+ .addSchema("name STRING", "id INT", "op STRING")
+ .producedValues(
+ Row.of("Alice", 1, "INSERT"),
+ Row.of("Bob", 2, "INSERT"),
+ Row.of("Alice2", 1, "UPDATE_AFTER"),
+ Row.of("Bob", 2, "DELETE"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .consumedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.UPDATE_AFTER, 1, "Alice2"),
+ Row.ofKind(RowKind.DELETE, 2, "Bob"))
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM FROM_CHANGELOG("
+ + "input => TABLE cdc_stream PARTITION BY id, "
+ + "op_mapping => MAP["
+ + "'INSERT', 'INSERT', "
+ + "'UPDATE_AFTER', 'UPDATE_AFTER', "
+ + "'DELETE', 'DELETE'])")
+ .build();
+
+ public static final TableTestProgram UPSERT_PARTITION_BY_CUSTOM_MAPPING =
Review Comment:
Rather add a validation test for the case of `UPDATE_AFTER` without `UPDATE_BEFORE` and `PARTITION BY`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3259873217
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
+ final boolean hasUpdateBefore =
+ mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
+ if (!hasUpdateAfter || hasUpdateBefore) {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3267487617
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -614,7 +614,7 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
/** Java-friendly overload that accepts a list of [[ExplainDetail]]s. */
def verifyRelPlan(query: String, extraDetails: java.util.List[ExplainDetail]): Unit = {
- verifyRelPlan(query, extraDetails.asScala.toSeq: _*)
+ verifyRelPlan(query, extraDetails.asScala: _*)
Review Comment:
Yes `toSeq` was not needed there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260037106
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
Review Comment:
Added
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
+ * UPDATE_AFTER}, and {@code DELETE}.
+ * <li><b>Upsert</b>: the {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code
+ * UPDATE_BEFORE}. The output emits {@code INSERT}, {@code UPDATE_AFTER}, and full {@code
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3264978825
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,44 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert table
+
+To generate an upsert table, two requirements must be met:
+
+* **Key partitioning**: use `PARTITION BY <key>`, where the partition key corresponds to the unique/primary key of the dataset.
+* **Op mapping configuration**: the `op_mapping` must include `UPDATE_AFTER` and must NOT include `UPDATE_BEFORE`.
+
+The engine assumes that the keys provided in the `PARTITION BY` clause function as the unique upsert keys. The resulting output changelog becomes an upsert table keyed on these partition columns. Each incoming row is evaluated and produces `INSERT`, `UPDATE_AFTER`, or `DELETE` events, using the partition key as the explicit upsert key. Therefore, if the incoming changelog contains unique keys (such as a primary key), they **must** be used in the `PARTITION BY` clause.
+
+<span class="label label-danger">Note</span>
+- An `op_mapping` that produces `UPDATE_AFTER` without `UPDATE_BEFORE` describes an upsert changelog and requires a key. `PARTITION BY` must be present on the table argument; otherwise the call would produce key-less updates and is rejected at validation time with a `ValidationException`.
Review Comment:
The note is a bit repetitive, we already stated that it's an upsert table and we have explicit above that the partition by is required
```suggestion
An `op_mapping` that produces `UPDATE_AFTER` without `UPDATE_BEFORE` describes an upsert changelog and requires a key. The engine assumes that the keys provided in the `PARTITION BY` clause function as the unique upsert keys. The resulting output changelog becomes an upsert table keyed on these partition columns. Each incoming row is evaluated and produces `INSERT`, `UPDATE_AFTER`, or `DELETE` events, using the partition key as the explicit upsert key. Therefore, if the incoming changelog contains unique keys (such as a primary key), they **must** be used in the `PARTITION BY` clause.
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java:
##########
@@ -226,6 +226,18 @@ public interface PartitionedTable {
* <p>For row semantics (each row processed independently), use {@link Table#fromChangelog} on
* the unpartitioned table.
*
+ * <p>Output changelog mode:
+ *
+ * <ul>
+ * <li><b>Retract</b> (default): the active {@code op_mapping} includes {@code UPDATE_BEFORE}
+ * or no updates at all. The output emits {@code INSERT}, {@code UPDATE_BEFORE}, {@code
+ * UPDATE_AFTER}, and {@code DELETE}.
+ * <li><b>Upsert</b>: the {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code
+ * UPDATE_BEFORE}. The output emits {@code INSERT}, {@code UPDATE_AFTER}, and full {@code
Review Comment:
hmmm, it wasn't :D. I see "full"
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -614,7 +614,32 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
/** Java-friendly overload that accepts a list of [[ExplainDetail]]s. */
def verifyRelPlan(query: String, extraDetails: java.util.List[ExplainDetail]): Unit = {
- verifyRelPlan(query, extraDetails.asScala.toSeq: _*)
+ verifyRelPlan(query, extraDetails.asScala: _*)
+ }
+
+ /**
+ * Verify the AST and the optimized rel plan for the given SELECT query. The rendered optimized
+ * rel plan includes the `upsertKeys=[...]` term for rel nodes that derive upsert keys.
+ */
+ def verifyRelPlanWithUpsertKey(query: String, extraDetails: ExplainDetail*): Unit = {
Review Comment:
This does not seem to fit the style of the utilities we have here. We don have unique verifyRelPlan for each param we can configure and rather use the extraDetail list. Take a look at how withDuplicateChanges and withQueryBlockAlias were implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3267449713
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -1057,7 +1073,8 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
withRowType,
expectedPlans,
() => assertEqualsOrExpand("sql", query),
- withQueryBlockAlias)
+ withQueryBlockAlias,
+ withUpsertKey = withUpsertKey)
Review Comment:
let's either used named args or not in one call
```suggestion
withQueryBlockAlias,
withUpsertKey)
```
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/utils/TableTestBase.scala:
##########
@@ -614,7 +614,7 @@ abstract class TableTestUtilBase(test: TableTestBase, isStreamingMode: Boolean)
/** Java-friendly overload that accepts a list of [[ExplainDetail]]s. */
def verifyRelPlan(query: String, extraDetails: java.util.List[ExplainDetail]): Unit = {
- verifyRelPlan(query, extraDetails.asScala.toSeq: _*)
+ verifyRelPlan(query, extraDetails.asScala: _*)
Review Comment:
Did you mean to change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3267582769
##########
docs/content/docs/sql/reference/queries/changelog.md:
##########
@@ -149,6 +149,44 @@ Prefer row semantics, when possible. `PARTITION BY` is only necessary when downs
If you are producing an upsert table — that is, you are emitting `UPDATE_AFTER` but no `UPDATE_BEFORE` from your CDC input stream — the partition key you select here will be considered both the primary key and the upsert key by the engine. Make sure the `PARTITION BY` key matches your primary key exactly.
+#### Upsert table
+
+To generate an upsert table, two requirements must be met:
+
+* **Key partitioning**: use `PARTITION BY <key>`, where the partition key corresponds to the unique/primary key of the dataset.
+* **Op mapping configuration**: the `op_mapping` must include `UPDATE_AFTER` and must NOT include `UPDATE_BEFORE`.
+
+The engine assumes that the keys provided in the `PARTITION BY` clause function as the unique upsert keys. The resulting output changelog becomes an upsert table keyed on these partition columns. Each incoming row is evaluated and produces `INSERT`, `UPDATE_AFTER`, or `DELETE` events, using the partition key as the explicit upsert key. Therefore, if the incoming changelog contains unique keys (such as a primary key), they **must** be used in the `PARTITION BY` clause.
+
+<span class="label label-danger">Note</span>
+- An `op_mapping` that produces `UPDATE_AFTER` without `UPDATE_BEFORE` describes an upsert changelog and requires a key. `PARTITION BY` must be present on the table argument; otherwise the call would produce key-less updates and is rejected at validation time with a `ValidationException`.
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260060876
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/FromChangelogTestPrograms.java:
##########
@@ -200,6 +200,69 @@ public class FromChangelogTestPrograms {
+ "input => TABLE cdc_stream PARTITION BY id)")
.build();
+ public static final TableTestProgram UPSERT_PARTITION_BY =
+ TableTestProgram.of(
+ "from-changelog-upsert-partition-by",
+ "PARTITION BY + op_mapping without UPDATE_BEFORE produces an "
+ + "upsert changelog keyed on the partition columns")
+ .setupTableSource(
+ SourceTestStep.newBuilder("cdc_stream")
+ .addSchema("name STRING", "id INT", "op STRING")
+ .producedValues(
+ Row.of("Alice", 1, "INSERT"),
+ Row.of("Bob", 2, "INSERT"),
+ Row.of("Alice2", 1, "UPDATE_AFTER"),
+ Row.of("Bob", 2, "DELETE"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .consumedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.UPDATE_AFTER, 1, "Alice2"),
+ Row.ofKind(RowKind.DELETE, 2, "Bob"))
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM FROM_CHANGELOG("
+ + "input => TABLE cdc_stream PARTITION BY id, "
+ + "op_mapping => MAP["
+ + "'INSERT', 'INSERT', "
+ + "'UPDATE_AFTER', 'UPDATE_AFTER', "
+ + "'DELETE', 'DELETE'])")
+ .build();
+
+ public static final TableTestProgram UPSERT_PARTITION_BY_CUSTOM_MAPPING =
Review Comment:
Sry for the confusion. It is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3259745468
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
+ final boolean hasUpdateBefore =
+ mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
+ if (!hasUpdateAfter || hasUpdateBefore) {
+ return Optional.empty();
+ }
+ final boolean partitioned =
+ callContext
+ .getTableSemantics(ARG_TABLE)
+ .map(ts -> ts.partitionByColumns().length > 0)
+ .orElse(false);
Review Comment:
The extraction would not be as beautiful as (you and) I imagined to be. One is using `CallContetxt` the other one is using `ChangelogContext`. I will try to play around and make it nice 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3260047759
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -95,6 +98,50 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(DataTypes.ROW(outputFields).notNull());
};
+ // --------------------------------------------------------------------------------------------
+ // Changelog mode inference
+ // --------------------------------------------------------------------------------------------
+
+ /**
+ * Emits an upsert changelog when the input is partitioned (set semantics) and the resolved
+ * {@code op_mapping} maps to {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}. In all other
+ * cases the output is a retract changelog. When upsert mode is selected, the partition key acts
+ * as the upsert key.
+ *
+ * <p>Upsert mode uses full deletes ({@link ChangelogMode#upsert(boolean) upsert(false)})
+ * because the runtime forwards each input delete row with all fields populated; only the {@link
+ * org.apache.flink.types.RowKind} is rewritten.
Review Comment:
Removed for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39538][table] Support upsert output mode for FROM_CHANGELOG [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28164:
URL: https://github.com/apache/flink/pull/28164#discussion_r3259869929
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
+ final CallContext callContext,
+ final Map<String, String> mapping,
+ final boolean throwOnFailure) {
+ final boolean hasUpdateAfter =
+ mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
Review Comment:
https://github.com/apache/flink/pull/28164#discussion_r3259867576
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/FromChangelogTypeStrategy.java:
##########
@@ -200,22 +247,47 @@ private static Optional<List<DataType>> validateOpMapping(
if (validationError.isPresent()) {
return validationError;
}
-
- final boolean hasUpdateBefore =
- mapping.values().stream().anyMatch(v -> UPDATE_BEFORE.equals(v.trim()));
- final boolean hasUpdateAfter =
- mapping.values().stream().anyMatch(v -> UPDATE_AFTER.equals(v.trim()));
- if (hasUpdateAfter && !hasUpdateBefore) {
- return callContext.fail(
- throwOnFailure,
- "The 'op_mapping' must include UPDATE_BEFORE for retract mode. "
- + "Upsert mode (without UPDATE_BEFORE) is not supported "
- + "in this version.");
+ final Optional<List<DataType>> upsertKeyError =
+ validateUpsertRequiresPartitionBy(callContext, mapping, throwOnFailure);
+ if (upsertKeyError.isPresent()) {
+ return upsertKeyError;
}
}
return Optional.empty();
}
+ /**
+ * An {@code op_mapping} that produces {@code UPDATE_AFTER} without {@code UPDATE_BEFORE}
+ * describes an upsert changelog. Upsert mode requires a key, so the input table must use set
+ * semantics via {@code PARTITION BY}.
+ */
+ private static Optional<List<DataType>> validateUpsertRequiresPartitionBy(
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org