You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2026/08/31 22:04:44 UTC
[PR] [FLINK-40528][table] Make codegen tolerat to partial deletes [flink]
snuyanzin opened a new pull request, #29060:
URL: https://github.com/apache/flink/pull/29060
## What is the purpose of the change
The PR makes codegen tolerant to partial deletes
## Brief change log
codegen
## Verifying this change
tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
- The S3 file system connector: ( no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable )
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` and replace the placeholder in the "Generated-by"
line with the tool name and version. Otherwise remove the "Generated-by" line.
See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [ ] Yes (please specify the tool below)
Generated-by: [Tool Name and Version]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29060:
URL: https://github.com/apache/flink/pull/29060#discussion_r3912668876
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
+ * NOT NULL ARRAY column that is wrapped in a ROW(...) projection.
+ */
+ public static final TableTestProgram JOIN_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "join-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: probe-side delete-by-key tombstone carries null"
+ + " for a NOT NULL ARRAY column wrapped in a ROW(...) projection"
+ + " across a join")
+ .setupTableSource(
+ SourceTestStep.newBuilder("left_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ Row.ofKind(RowKind.INSERT, 3, new Integer[] {5}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 3, new Integer[] {6}))
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("right_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.INSERT, 3, "Emily"),
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "BOB"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>",
+ "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .testMaterializedData()
+ .consumedValues(
+ "+I[2, +I[2, [3]], BOB]", "+I[3, +I[3, [6]], Emily]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT l.id, ROW(l.id, l.arr), r.name"
+ + " FROM left_t l JOIN right_t r ON l.id = r.id")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is used as an
+ * element of an {@code ARRAY[...]} literal (element type {@code INT NOT NULL}) wrapped in a
+ * {@code ROW(...)} projection. Validates that array construction sets the element to null
+ * instead of writing the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_ARRAY_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-array-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as an element of an ARRAY[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [10, 99]]]",
+ "+I[2, +I[2, [20, 99]]]",
+ "-D[1, +I[1, [null, 99]]]",
+ "+U[2, +I[2, [30, 99]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY literal variant but for a {@code MAP[...]} literal whose value comes from a
+ * NOT NULL {@code INT} column (value type {@code INT NOT NULL}).
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_MAP_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-map-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as a value of a MAP[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {99=10}]]",
+ "+I[2, +I[2, {99=20}]]",
+ "-D[1, +I[1, {99=null}]]",
+ "+U[2, +I[2, {99=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, MAP[99, v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is nested in a
+ * {@code ROW(...)} serialized by {@code JSON_OBJECT}. The nested field is NOT NULL, so the JSON
+ * row converter must still guard on the runtime null and emit {@code null} instead of reading
+ * the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column nested in a ROW serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"r\":{\"EXPR$0\":10}}]",
+ "+I[2, {\"r\":{\"EXPR$0\":20}}]",
+ "-D[1, {\"r\":{\"EXPR$0\":null}}]",
+ "+U[2, {\"r\":{\"EXPR$0\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('r' VALUE ROW(v)) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} element nested in
+ * an {@code ARRAY[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT element nested in an ARRAY serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"a\":[10,99]}]",
+ "+I[2, {\"a\":[20,99]}]",
+ "-D[1, {\"a\":[null,99]}]",
+ "+U[2, {\"a\":[30,99]}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('a' VALUE ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} value nested in a
+ * {@code MAP[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT value nested in a MAP serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"m\":{\"k\":10}}]",
+ "+I[2, {\"m\":{\"k\":20}}]",
+ "-D[1, {\"m\":{\"k\":null}}]",
+ "+U[2, {\"m\":{\"k\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('m' VALUE MAP['k', v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is CAST to
+ * another primitive type. The cast framework skips the runtime null guard when the input type
+ * is NOT NULL and the target is a primitive Java type, so it reads the primitive default (0)
+ * instead of producing null.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST =
Review Comment:
> INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST
this is required
it highlights another finding flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/CodeGenUtils.scala
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
+ * NOT NULL ARRAY column that is wrapped in a ROW(...) projection.
+ */
+ public static final TableTestProgram JOIN_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "join-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: probe-side delete-by-key tombstone carries null"
+ + " for a NOT NULL ARRAY column wrapped in a ROW(...) projection"
+ + " across a join")
+ .setupTableSource(
+ SourceTestStep.newBuilder("left_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ Row.ofKind(RowKind.INSERT, 3, new Integer[] {5}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 3, new Integer[] {6}))
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("right_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.INSERT, 3, "Emily"),
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "BOB"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>",
+ "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .testMaterializedData()
+ .consumedValues(
+ "+I[2, +I[2, [3]], BOB]", "+I[3, +I[3, [6]], Emily]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT l.id, ROW(l.id, l.arr), r.name"
+ + " FROM left_t l JOIN right_t r ON l.id = r.id")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is used as an
+ * element of an {@code ARRAY[...]} literal (element type {@code INT NOT NULL}) wrapped in a
+ * {@code ROW(...)} projection. Validates that array construction sets the element to null
+ * instead of writing the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_ARRAY_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-array-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as an element of an ARRAY[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [10, 99]]]",
+ "+I[2, +I[2, [20, 99]]]",
+ "-D[1, +I[1, [null, 99]]]",
+ "+U[2, +I[2, [30, 99]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY literal variant but for a {@code MAP[...]} literal whose value comes from a
+ * NOT NULL {@code INT} column (value type {@code INT NOT NULL}).
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_MAP_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-map-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as a value of a MAP[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {99=10}]]",
+ "+I[2, +I[2, {99=20}]]",
+ "-D[1, +I[1, {99=null}]]",
+ "+U[2, +I[2, {99=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, MAP[99, v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is nested in a
+ * {@code ROW(...)} serialized by {@code JSON_OBJECT}. The nested field is NOT NULL, so the JSON
+ * row converter must still guard on the runtime null and emit {@code null} instead of reading
+ * the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column nested in a ROW serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"r\":{\"EXPR$0\":10}}]",
+ "+I[2, {\"r\":{\"EXPR$0\":20}}]",
+ "-D[1, {\"r\":{\"EXPR$0\":null}}]",
+ "+U[2, {\"r\":{\"EXPR$0\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('r' VALUE ROW(v)) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} element nested in
+ * an {@code ARRAY[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT element nested in an ARRAY serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"a\":[10,99]}]",
+ "+I[2, {\"a\":[20,99]}]",
+ "-D[1, {\"a\":[null,99]}]",
+ "+U[2, {\"a\":[30,99]}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('a' VALUE ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} value nested in a
+ * {@code MAP[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT value nested in a MAP serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"m\":{\"k\":10}}]",
+ "+I[2, {\"m\":{\"k\":20}}]",
+ "-D[1, {\"m\":{\"k\":null}}]",
+ "+U[2, {\"m\":{\"k\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('m' VALUE MAP['k', v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is CAST to
+ * another primitive type. The cast framework skips the runtime null guard when the input type
+ * is NOT NULL and the target is a primitive Java type, so it reads the primitive default (0)
+ * instead of producing null.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST =
Review Comment:
> INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST
this is required
it highlights another finding flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/CodeGenUtils.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #29060:
URL: https://github.com/apache/flink/pull/29060#discussion_r3904609057
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/GeneratedExpression.scala:
##########
@@ -56,6 +56,9 @@ case class GeneratedExpression(
*/
def literal: Boolean = literalValue.isDefined
+ /** Whether this expression is statically proven never to be null at runtime. */
+ def isProvenNotNull: Boolean = nullTerm == GeneratedExpression.NEVER_NULL
Review Comment:
Which expressions are proven not null at runtime if we do not trust the types?
It would only be literals and expressions on literals, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #29060:
URL: https://github.com/apache/flink/pull/29060#discussion_r3902967247
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala:
##########
@@ -1226,7 +1226,7 @@ object ScalarOperatorGens {
val tpe = fieldTypes(idx)
if (element.literal) {
""
- } else if (tpe.isNullable) {
+ } else if (tpe.isNullable || element.nullTerm != NEVER_NULL) {
Review Comment:
Can we extract this as a small helper to slightly improve readability?
```
/** True only when codegen proved this statically; a dynamic nullTerm may still be null at runtime. */
def isProvenNonNull: Boolean = nullTerm == NEVER_NULL
...
} else if (tpe.isNullable || !element.isProvenNonNull) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #29060:
URL: https://github.com/apache/flink/pull/29060#discussion_r3906098729
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
+ * NOT NULL ARRAY column that is wrapped in a ROW(...) projection.
+ */
+ public static final TableTestProgram JOIN_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "join-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: probe-side delete-by-key tombstone carries null"
+ + " for a NOT NULL ARRAY column wrapped in a ROW(...) projection"
+ + " across a join")
+ .setupTableSource(
+ SourceTestStep.newBuilder("left_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ Row.ofKind(RowKind.INSERT, 3, new Integer[] {5}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 3, new Integer[] {6}))
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("right_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.INSERT, 3, "Emily"),
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "BOB"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>",
+ "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .testMaterializedData()
+ .consumedValues(
+ "+I[2, +I[2, [3]], BOB]", "+I[3, +I[3, [6]], Emily]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT l.id, ROW(l.id, l.arr), r.name"
+ + " FROM left_t l JOIN right_t r ON l.id = r.id")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is used as an
+ * element of an {@code ARRAY[...]} literal (element type {@code INT NOT NULL}) wrapped in a
+ * {@code ROW(...)} projection. Validates that array construction sets the element to null
+ * instead of writing the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_ARRAY_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-array-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as an element of an ARRAY[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [10, 99]]]",
+ "+I[2, +I[2, [20, 99]]]",
+ "-D[1, +I[1, [null, 99]]]",
+ "+U[2, +I[2, [30, 99]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY literal variant but for a {@code MAP[...]} literal whose value comes from a
+ * NOT NULL {@code INT} column (value type {@code INT NOT NULL}).
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_MAP_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-map-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as a value of a MAP[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {99=10}]]",
+ "+I[2, +I[2, {99=20}]]",
+ "-D[1, +I[1, {99=null}]]",
+ "+U[2, +I[2, {99=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, MAP[99, v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is nested in a
+ * {@code ROW(...)} serialized by {@code JSON_OBJECT}. The nested field is NOT NULL, so the JSON
+ * row converter must still guard on the runtime null and emit {@code null} instead of reading
+ * the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column nested in a ROW serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
Review Comment:
imo you can drop this everywhere
```suggestion
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
Review Comment:
test is inner join, which one did you want?
```suggestion
* A INNER JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
+ * NOT NULL ARRAY column that is wrapped in a ROW(...) projection.
+ */
+ public static final TableTestProgram JOIN_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "join-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: probe-side delete-by-key tombstone carries null"
+ + " for a NOT NULL ARRAY column wrapped in a ROW(...) projection"
+ + " across a join")
+ .setupTableSource(
+ SourceTestStep.newBuilder("left_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ Row.ofKind(RowKind.INSERT, 3, new Integer[] {5}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 3, new Integer[] {6}))
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("right_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.INSERT, 3, "Emily"),
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "BOB"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>",
+ "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .testMaterializedData()
+ .consumedValues(
+ "+I[2, +I[2, [3]], BOB]", "+I[3, +I[3, [6]], Emily]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT l.id, ROW(l.id, l.arr), r.name"
+ + " FROM left_t l JOIN right_t r ON l.id = r.id")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is used as an
+ * element of an {@code ARRAY[...]} literal (element type {@code INT NOT NULL}) wrapped in a
+ * {@code ROW(...)} projection. Validates that array construction sets the element to null
+ * instead of writing the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_ARRAY_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-array-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as an element of an ARRAY[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [10, 99]]]",
+ "+I[2, +I[2, [20, 99]]]",
+ "-D[1, +I[1, [null, 99]]]",
+ "+U[2, +I[2, [30, 99]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY literal variant but for a {@code MAP[...]} literal whose value comes from a
+ * NOT NULL {@code INT} column (value type {@code INT NOT NULL}).
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_MAP_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-map-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as a value of a MAP[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {99=10}]]",
+ "+I[2, +I[2, {99=20}]]",
+ "-D[1, +I[1, {99=null}]]",
+ "+U[2, +I[2, {99=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, MAP[99, v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is nested in a
+ * {@code ROW(...)} serialized by {@code JSON_OBJECT}. The nested field is NOT NULL, so the JSON
+ * row converter must still guard on the runtime null and emit {@code null} instead of reading
+ * the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column nested in a ROW serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"r\":{\"EXPR$0\":10}}]",
+ "+I[2, {\"r\":{\"EXPR$0\":20}}]",
+ "-D[1, {\"r\":{\"EXPR$0\":null}}]",
+ "+U[2, {\"r\":{\"EXPR$0\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('r' VALUE ROW(v)) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} element nested in
+ * an {@code ARRAY[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT element nested in an ARRAY serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"a\":[10,99]}]",
+ "+I[2, {\"a\":[20,99]}]",
+ "-D[1, {\"a\":[null,99]}]",
+ "+U[2, {\"a\":[30,99]}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('a' VALUE ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} value nested in a
+ * {@code MAP[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT value nested in a MAP serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"m\":{\"k\":10}}]",
+ "+I[2, {\"m\":{\"k\":20}}]",
+ "-D[1, {\"m\":{\"k\":null}}]",
+ "+U[2, {\"m\":{\"k\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('m' VALUE MAP['k', v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is CAST to
+ * another primitive type. The cast framework skips the runtime null guard when the input type
+ * is NOT NULL and the target is a primitive Java type, so it reads the primitive default (0)
+ * instead of producing null.
+ */
Review Comment:
Remove as well? Probably just AI overexplaining here. I think there are multiple comments on top of the tests which you can simply delete or reduce
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/DeletesByKeyPrograms.java:
##########
@@ -293,5 +295,532 @@ public final class DeletesByKeyPrograms {
"INSERT INTO sink_t SELECT l.id, r.name, l.`value` FROM left_t l JOIN right_t r ON l.id = r.id")
.build();
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL ARRAY wrapped in a {@code ROW(...)}
+ * projection.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY column wrapped in a ROW(...) projection; validates"
+ + " that row construction does not fail on the null value")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER, 2, new Integer[] {3, 4}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [1, 2]]]",
+ "+I[2, +I[2, [3]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [3, 4]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code MAP} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL MAP column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "m MAP<INT, INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Map.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Map.of(2, 20)),
+ // Delete by key: NOT NULL map column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Map.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {1=10}]]",
+ "+I[2, +I[2, {2=20}]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, {2=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, m) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY variant but for a NOT NULL {@code ROW} column wrapped in {@code ROW(...)}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ROW column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "nested ROW<x INT, y INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, Row.of(1, 10)),
+ Row.ofKind(RowKind.INSERT, 2, Row.of(2, 20)),
+ // Delete by key: NOT NULL row column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, Row.of(2, 30)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ROW<x INT, y INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, +I[1, 10]]]",
+ "+I[2, +I[2, +I[2, 20]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, +I[2, 30]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, nested) FROM source_t")
+ .build();
+
+ /** Same as the ARRAY variant but for a NOT NULL {@code ARRAY<ROW>} column. */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY_OF_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-array-of-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL ARRAY<ROW> column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<ROW<x INT, y INT>> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(
+ RowKind.INSERT,
+ 1,
+ new Row[] {Row.of(1, 10)}),
+ Row.ofKind(
+ RowKind.INSERT,
+ 2,
+ new Row[] {Row.of(2, 20)}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(
+ RowKind.UPDATE_AFTER,
+ 2,
+ new Row[] {Row.of(2, 30)}))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<ROW<x INT, y INT>>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [+I[1, 10]]]]",
+ "+I[2, +I[2, [+I[2, 20]]]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, [+I[2, 30]]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, arr) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL {@code STRING} column. The reference-typed
+ * String path does not fail today, but the case is kept for coverage.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-string",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL STRING column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED", "s STRING NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "a"),
+ Row.ofKind(RowKind.INSERT, 2, "b"),
+ // Delete by key: NOT NULL string column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "c"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b STRING>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, a]]",
+ "+I[2, +I[2, b]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, c]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, s) FROM source_t")
+ .build();
+
+ /**
+ * Same shape as the ARRAY variant but for a NOT NULL primitive {@code INT} column. The
+ * primitive path is already guarded; kept for coverage.
+ */
+ public static final TableTestProgram
+ INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_PRIMITIVE =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-nested-not-null-primitive",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column wrapped in a ROW(...) projection")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b INT>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, 10]]",
+ "+I[2, +I[2, 20]]",
+ "-D[1, +I[1, null]]",
+ "+U[2, +I[2, 30]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, v) FROM source_t")
+ .build();
+
+ /**
+ * A LEFT JOIN whose probe (left) side produces a delete-by-key tombstone carrying null for a
+ * NOT NULL ARRAY column that is wrapped in a ROW(...) projection.
+ */
+ public static final TableTestProgram JOIN_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_ARRAY =
+ TableTestProgram.of(
+ "join-delete-on-key-with-nested-not-null-array",
+ "No ChangelogNormalize: probe-side delete-by-key tombstone carries null"
+ + " for a NOT NULL ARRAY column wrapped in a ROW(...) projection"
+ + " across a join")
+ .setupTableSource(
+ SourceTestStep.newBuilder("left_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "arr ARRAY<INT> NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, new Integer[] {1, 2}),
+ Row.ofKind(RowKind.INSERT, 2, new Integer[] {3}),
+ Row.ofKind(RowKind.INSERT, 3, new Integer[] {5}),
+ // Delete by key: NOT NULL array column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 3, new Integer[] {6}))
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("right_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, "Alice"),
+ Row.ofKind(RowKind.INSERT, 2, "Bob"),
+ Row.ofKind(RowKind.INSERT, 3, "Emily"),
+ Row.ofKind(RowKind.DELETE, 1, null),
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, "BOB"))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>",
+ "name STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .testMaterializedData()
+ .consumedValues(
+ "+I[2, +I[2, [3]], BOB]", "+I[3, +I[3, [6]], Emily]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT l.id, ROW(l.id, l.arr), r.name"
+ + " FROM left_t l JOIN right_t r ON l.id = r.id")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is used as an
+ * element of an {@code ARRAY[...]} literal (element type {@code INT NOT NULL}) wrapped in a
+ * {@code ROW(...)} projection. Validates that array construction sets the element to null
+ * instead of writing the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_ARRAY_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-array-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as an element of an ARRAY[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b ARRAY<INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, [10, 99]]]",
+ "+I[2, +I[2, [20, 99]]]",
+ "-D[1, +I[1, [null, 99]]]",
+ "+U[2, +I[2, [30, 99]]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the ARRAY literal variant but for a {@code MAP[...]} literal whose value comes from a
+ * NOT NULL {@code INT} column (value type {@code INT NOT NULL}).
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_MAP_LITERAL =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-not-null-map-literal",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column used as a value of a MAP[...] literal")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema(
+ "id INT PRIMARY KEY NOT ENFORCED",
+ "r ROW<a INT, b MAP<INT, INT>>")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, +I[1, {99=10}]]",
+ "+I[2, +I[2, {99=20}]]",
+ "-D[1, +I[1, {99=null}]]",
+ "+U[2, +I[2, {99=30}]]")
+ .build())
+ .runSql("INSERT INTO sink_t SELECT id, ROW(id, MAP[99, v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is nested in a
+ * {@code ROW(...)} serialized by {@code JSON_OBJECT}. The nested field is NOT NULL, so the JSON
+ * row converter must still guard on the runtime null and emit {@code null} instead of reading
+ * the primitive default.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ROW =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-row",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT column nested in a ROW serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"r\":{\"EXPR$0\":10}}]",
+ "+I[2, {\"r\":{\"EXPR$0\":20}}]",
+ "-D[1, {\"r\":{\"EXPR$0\":null}}]",
+ "+U[2, {\"r\":{\"EXPR$0\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('r' VALUE ROW(v)) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} element nested in
+ * an {@code ARRAY[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_ARRAY =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-array",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT element nested in an ARRAY serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"a\":[10,99]}]",
+ "+I[2, {\"a\":[20,99]}]",
+ "-D[1, {\"a\":[null,99]}]",
+ "+U[2, {\"a\":[30,99]}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('a' VALUE ARRAY[v, 99]) FROM source_t")
+ .build();
+
+ /**
+ * Same as the JSON_OBJECT nested ROW variant but for a NOT NULL {@code INT} value nested in a
+ * {@code MAP[...]} serialized by {@code JSON_OBJECT}.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_JSON_OBJECT_NESTED_MAP =
+ TableTestProgram.of(
+ "select-delete-on-key-to-delete-on-key-with-json-object-nested-map",
+ "No ChangelogNormalize: a delete-by-key tombstone carries null for a NOT"
+ + " NULL INT value nested in a MAP serialized by JSON_OBJECT")
+ .setupTableSource(
+ SourceTestStep.newBuilder("source_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "v INT NOT NULL")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("source.produces-delete-by-key", "true")
+ .producedValues(
+ Row.ofKind(RowKind.INSERT, 1, 10),
+ Row.ofKind(RowKind.INSERT, 2, 20),
+ // Delete by key: NOT NULL int column is null
+ Row.ofKind(RowKind.DELETE, 1, null),
+ // Update after only
+ Row.ofKind(RowKind.UPDATE_AFTER, 2, 30))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink_t")
+ .addSchema("id INT PRIMARY KEY NOT ENFORCED", "j STRING")
+ .addOption("changelog-mode", "I,UA,D")
+ .addOption("sink.supports-delete-by-key", "true")
+ .consumedValues(
+ "+I[1, {\"m\":{\"k\":10}}]",
+ "+I[2, {\"m\":{\"k\":20}}]",
+ "-D[1, {\"m\":{\"k\":null}}]",
+ "+U[2, {\"m\":{\"k\":30}}]")
+ .build())
+ .runSql(
+ "INSERT INTO sink_t SELECT id, JSON_OBJECT('m' VALUE MAP['k', v]) FROM source_t")
+ .build();
+
+ /**
+ * A delete-by-key tombstone carries null for a NOT NULL {@code INT} column that is CAST to
+ * another primitive type. The cast framework skips the runtime null guard when the input type
+ * is NOT NULL and the target is a primitive Java type, so it reads the primitive default (0)
+ * instead of producing null.
+ */
+ public static final TableTestProgram INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST =
Review Comment:
nit: these are a lot of tests. I'm ok with it if they're really necessary. Maybe we can reduce the number of tests without losing coverage?
For example, I think these two were passing before the change as well. Is there value in keeping them?
`INSERT_SELECT_DELETE_BY_KEY_WITH_NOT_NULL_CAST` and `INSERT_SELECT_DELETE_BY_KEY_WITH_NESTED_NOT_NULL_STRING`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerat to partial deletes [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29060:
URL: https://github.com/apache/flink/pull/29060#issuecomment-5485368779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4c9d2b4982fd27a3b6c86cd45becd110814c9ef2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4c9d2b4982fd27a3b6c86cd45becd110814c9ef2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c9d2b4982fd27a3b6c86cd45becd110814c9ef2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on PR #29060:
URL: https://github.com/apache/flink/pull/29060#issuecomment-5526063333
lgtm. Did you want want to take a look at this @twalthr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #29060:
URL: https://github.com/apache/flink/pull/29060#issuecomment-5926427964
closing as fixed by different approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40528][table] Make codegen tolerant to partial deletes [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin closed pull request #29060: [FLINK-40528][table] Make codegen tolerant to partial deletes
URL: https://github.com/apache/flink/pull/29060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org