You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/09/28 20:37:01 UTC
[PR] [FLINK-40839][table] Support VARIANT and BITMAP columns omitted from an INSERT column list [flink]
raminqaf opened a new pull request, #29330:
URL: https://github.com/apache/flink/pull/29330
## What is the purpose of the change
An INSERT with a column list fails when an omitted column is of type VARIANT or BITMAP.
```sql
CREATE TABLE snk (k INT, v VARIANT, b BITMAP) WITH ('connector' = 'blackhole');
INSERT INTO snk (k) SELECT k FROM src;
```
| Omitted column type | Before | After |
|-----------------------------------------------------------------------|---------------------------------------------------------------------------|--------------------|
| `VARIANT`, `ARRAY<VARIANT>`, `ROW<f VARIANT>`, `MAP<STRING, VARIANT>` | `SQL validation failed. Unsupported type when convertTypeToSpec: VARIANT` | filled with `NULL` |
| `BITMAP`, `ARRAY<BITMAP>`, `ROW<f BITMAP>` | `SQL validation failed. Unsupported type when convertTypeToSpec: OTHER` | filled with `NULL` |
`PreValidateReWriter` pads every omitted column with `CAST(NULL AS <column type>)`. The cast target comes from `SqlTypeUtil#convertTypeToSpec`, which had no branch for either type.
## Brief change log
Both changes are in Flink's copy of `SqlTypeUtil`:
- VARIANT: backport of CALCITE-7293, which adds VARIANT to `isAtomic`.
- BITMAP: `convertTypeToSpec` maps `BitmapRelDataType` to `SqlBitmapTypeNameSpec`.
Notes for reviewers:
- The VARIANT line is the same one the Calcite 1.42.0 upgrade in #28981 brings for FLINK-40001. Whichever PR merges second resolves a one-line conflict, and the backport marker can then go.
- Flink disables Calcite's implicit type coercion, so the `isAtomic` change has no effect outside `convertTypeToSpec`.
- Calcite cannot fix BITMAP. `BitmapRelDataType` reports `SqlTypeName.OTHER`, so it needs a Flink branch.
## Verifying this change
This change added tests and can be verified as follows:
- `SqlTypeUtilTest` converts VARIANT and BITMAP to a type spec and unparses it. It covers nullable and NOT NULL, and both types nested in ARRAY, ROW and MAP.
- `VariantSemanticTest` and `BitmapSemanticTest` run `INSERT INTO sink_t (k) SELECT k FROM t` end to end and check that the omitted columns arrive as `NULL`.
- Without the fix, all 11 new cases fail with the errors above.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on PR #29330:
URL: https://github.com/apache/flink/pull/29330#issuecomment-5890614668
@flinkbot run azure re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29330:
URL: https://github.com/apache/flink/pull/29330#discussion_r4134306713
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/calcite/SqlTypeUtilTest.java:
##########
@@ -42,14 +54,35 @@ class SqlTypeUtilTest {
*/
@Test
void testConvertRowTypeToSpecAndUnparse() {
- FlinkTypeFactory typeFactory =
- new FlinkTypeFactory(
- Thread.currentThread().getContextClassLoader(), FlinkTypeSystem.INSTANCE);
RowType rowType =
RowType.of(
new LogicalType[] {new IntType(), new VarCharType(1)},
new String[] {"a", "b"});
- RelDataType relDataType = LogicalRelDataTypeConverter.toRelDataType(rowType, typeFactory);
+ assertThat(convertToSpecAndUnparse(rowType))
+ .hasToString("ROW(\"a\" INTEGER, \"b\" VARCHAR(1) CHARACTER SET \"UTF-16LE\")");
+ }
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("variantTypes")
+ void testConvertTypeToSpecAndUnparse(DataType dataType, String expected) {
+ assertThat(convertToSpecAndUnparse(dataType.getLogicalType())).isEqualTo(expected);
+ }
+
+ private static Stream<Arguments> variantTypes() {
+ return Stream.of(
+ Arguments.of(VARIANT(), "VARIANT"),
+ Arguments.of(VARIANT().notNull(), "VARIANT"),
+ Arguments.of(ARRAY(VARIANT()), "VARIANT ARRAY"),
+ Arguments.of(ROW(FIELD("v", VARIANT())), "ROW(\"v\" VARIANT)"),
+ Arguments.of(MAP(INT(), VARIANT()), "MAP< INTEGER, VARIANT >"));
+ }
+
+ private static String convertToSpecAndUnparse(LogicalType logicalType) {
+ FlinkTypeFactory typeFactory =
+ new FlinkTypeFactory(
+ Thread.currentThread().getContextClassLoader(), FlinkTypeSystem.INSTANCE);
+ RelDataType relDataType =
+ LogicalRelDataTypeConverter.toRelDataType(logicalType, typeFactory);
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT and BITMAP columns omitted from an INSERT column list [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29330:
URL: https://github.com/apache/flink/pull/29330#issuecomment-5878281292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2845779de7c415f74de5152ccf1d5afd20ff4fad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2845779de7c415f74de5152ccf1d5afd20ff4fad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2845779de7c415f74de5152ccf1d5afd20ff4fad UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #29330:
URL: https://github.com/apache/flink/pull/29330#issuecomment-5906391328
seems azure issue
GHA passed https://github.com/raminqaf/flink/actions/runs/36563259568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on PR #29330:
URL: https://github.com/apache/flink/pull/29330#issuecomment-5890771891
@flinkbot run azure re-run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #29330:
URL: https://github.com/apache/flink/pull/29330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on PR #29330:
URL: https://github.com/apache/flink/pull/29330#issuecomment-5891566762
@flinkbot run azure re-run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40839][table] Support VARIANT columns omitted from an INSERT column list [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29330:
URL: https://github.com/apache/flink/pull/29330#discussion_r4132768329
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/calcite/SqlTypeUtilTest.java:
##########
@@ -42,14 +54,35 @@ class SqlTypeUtilTest {
*/
@Test
void testConvertRowTypeToSpecAndUnparse() {
- FlinkTypeFactory typeFactory =
- new FlinkTypeFactory(
- Thread.currentThread().getContextClassLoader(), FlinkTypeSystem.INSTANCE);
RowType rowType =
RowType.of(
new LogicalType[] {new IntType(), new VarCharType(1)},
new String[] {"a", "b"});
- RelDataType relDataType = LogicalRelDataTypeConverter.toRelDataType(rowType, typeFactory);
+ assertThat(convertToSpecAndUnparse(rowType))
+ .hasToString("ROW(\"a\" INTEGER, \"b\" VARCHAR(1) CHARACTER SET \"UTF-16LE\")");
+ }
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("variantTypes")
+ void testConvertTypeToSpecAndUnparse(DataType dataType, String expected) {
+ assertThat(convertToSpecAndUnparse(dataType.getLogicalType())).isEqualTo(expected);
+ }
+
+ private static Stream<Arguments> variantTypes() {
+ return Stream.of(
+ Arguments.of(VARIANT(), "VARIANT"),
+ Arguments.of(VARIANT().notNull(), "VARIANT"),
+ Arguments.of(ARRAY(VARIANT()), "VARIANT ARRAY"),
+ Arguments.of(ROW(FIELD("v", VARIANT())), "ROW(\"v\" VARIANT)"),
+ Arguments.of(MAP(INT(), VARIANT()), "MAP< INTEGER, VARIANT >"));
+ }
+
+ private static String convertToSpecAndUnparse(LogicalType logicalType) {
+ FlinkTypeFactory typeFactory =
+ new FlinkTypeFactory(
+ Thread.currentThread().getContextClassLoader(), FlinkTypeSystem.INSTANCE);
+ RelDataType relDataType =
+ LogicalRelDataTypeConverter.toRelDataType(logicalType, typeFactory);
Review Comment:
nit: can we reuse typefactory and SqlPrettyWriter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org