You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/09/02 16:14:30 UTC
[PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
raminqaf opened a new pull request, #29073:
URL: https://github.com/apache/flink/pull/29073
## What is the purpose of the change
Let `CAST` and `TRY_CAST` convert a `VARIANT` into an `ARRAY<T>`. This is the first of three stacked PRs for FLINK-37926 (VARIANT to constructed types); the follow-ups add `ROW`/`STRUCTURED` and `MAP<STRING, V>` and build on this branch. It extends FLINK-37925 (VARIANT to scalar types, already merged): a constructed cast is that same scalar cast applied to every element, plus a shape check at each level, so no new leaf semantics are introduced.
A variant array casts to `ARRAY<T>` by casting each element to `T` with the full VARIANT-to-`T` rule, recursively. The recursion bottoms out at the scalar cast, which keeps one mental model for the whole type and one deliberate strictness: a stored value keeps its kind, so a stored string is never parsed into a number. `CAST(PARSE_JSON('["1","2"]') AS ARRAY<INT>)` therefore fails; read it as `ARRAY<STRING>` and convert with a regular cast instead.
## Brief change log
- Add `VariantToArrayCastRule`: shape-check the variant is an array, then cast each element to the target element type, recursively reusing the existing VARIANT cast rules (including the `VARIANT`-to-`VARIANT` identity, so `ARRAY<VARIANT>` shreds exactly one level).
- A JSON `null` element maps to SQL `NULL` for a nullable element type and fails the cast for a `NOT NULL` one; an empty array casts to an empty `ARRAY<T>`; a top-level JSON `null` casts to SQL `NULL` for a nullable target before the shape check. `CAST` fails on any bad element and `TRY_CAST` returns `NULL` for the whole array (all-or-nothing), matching nested `ARRAY` casts today.
- Add a shared `AbstractVariantToConstructedCastRule` base (JSON-null passthrough and `canFail`), reused by the follow-up ROW and MAP PRs.
- Castability is explicit only and recursive: `VARIANT` to `ARRAY<T>` is castable iff `VARIANT` to `T` is, so an element type with no variant counterpart (e.g. `INTERVAL`) is rejected at plan time.
- Extend the cast-rules table and the data types docs (EN and zh).
## Verifying this change
This change added tests and can be verified as follows:
- `LogicalTypeCastsTest`: `VARIANT` to `ARRAY` castability, including nesting, `ARRAY<VARIANT>`, and an unsupported leaf.
- `CastRuleProviderTest#testResolveVariantToArray`: rule resolution and recursive resolvability.
- `CastRulesTest`: codegen behavior for `ARRAY<INT>` (nullable and `NOT NULL`), `ARRAY<STRING>`, `ARRAY<DOUBLE>`, nested `ARRAY<ARRAY<INT>>`, `ARRAY<VARIANT>`, empty and null-element arrays, a heterogeneous array, and shape mismatches.
- `CastFunctionITCase`: end-to-end `CAST`/`TRY_CAST` over `PARSE_JSON`, covering the same shapes, the strict-leaf failures, and validation errors.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no (a new opt-in cast rule; existing paths are unchanged)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs (`data-types.md`, EN and zh) and JavaDocs
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` and replace the placeholder in the "Generated-by"
line with the tool name and version. Otherwise remove the "Generated-by" line.
See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [X] Yes (please specify the tool below)
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "mateczagany (via GitHub)" <gi...@apache.org>.
mateczagany commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3923526447
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
+ /* Example generated code for ARRAY<INT>:
+
+ int arraySize$2 =
+ org.apache.flink.table.runtime.functions.VariantCastUtils.arraySize(
+ variant$1, "ARRAY<INT>");
+ java.lang.Integer[] objArray$3 = new java.lang.Integer[arraySize$2];
+ for (int i$4 = 0; i$4 < arraySize$2; i$4++) {
+ org.apache.flink.types.variant.Variant element$5 = variant$1.getElement(i$4);
+ if (!element$5.isNull()) {
+ result$6 =
+ ((int) org.apache.flink.table.runtime.functions.VariantCastUtils.toIntegral(
+ element$5, -2147483648L, 2147483647L, "INTEGER"));
+ objArray$3[i$4] = result$6;
+ }
+ }
+ result$0 = new org.apache.flink.table.data.GenericArrayData(objArray$3);
+
+ A JSON null element leaves the slot null (SQL NULL); a NOT NULL element type emits a throw instead.
Review Comment:
For `ARRAY<VARIANT>` this drops a value: the identity cast keeps a variant null at top level, but here it becomes SQL NULL.
I think by skipping the `isNull` when the `elementType` is `VARIANT` casts could keep null elements as variant null instead.
This would also match Spark, Trino and Snowflake in this regard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924818776
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
+
+- Each element casts to `T`. A JSON null element maps to SQL `NULL` when `T` is nullable and fails
+ the cast when `T` is `NOT NULL`. An empty array casts to an empty `ARRAY<T>`.
+- `ARRAY<VARIANT>` is the identity leaf: it shreds one level and keeps each element as a variant. A
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3921822589
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
Review Comment:
please put in javadoc sample of generated code as done for other rules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3922743741
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
Review Comment:
Added an example code for `ARRAY<INT>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924813974
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
Review Comment:
Simplified
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
+
+- Each element casts to `T`. A JSON null element maps to SQL `NULL` when `T` is nullable and fails
+ the cast when `T` is `NOT NULL`. An empty array casts to an empty `ARRAY<T>`.
+- `ARRAY<VARIANT>` is the identity leaf: it shreds one level and keeps each element as a variant. A
+ JSON null element stays a variant null rather than becoming SQL `NULL`.
+
+If any element cast fails, the whole cast fails, and `TRY_CAST` returns `NULL` for the entire value
+rather than a partial result. An element type with no variant counterpart, such as
+`ARRAY<INTERVAL YEAR TO MONTH>`, is rejected at validation. A top-level JSON `null` casts to SQL `NULL` for a nullable target before any
+shape check runs.
Review Comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3922739438
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -188,6 +188,57 @@ class CastRulesTest {
.build())
.build();
+ /** {@code [1, 2, 3]}, the design's running array value. */
+ private static final Variant VARIANT_INT_ARRAY =
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().of(2))
+ .add(Variant.newBuilder().of(3))
+ .build();
+
+ /** {@code [1, null, 3]}, an array carrying a JSON null element. */
+ private static final Variant VARIANT_INT_ARRAY_WITH_NULL =
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().ofNull())
+ .add(Variant.newBuilder().of(3))
+ .build();
+
+ /** {@code ["1", "2"]}, an array of stored strings that a numeric leaf must not parse. */
+ private static final Variant VARIANT_STRING_ARRAY =
Review Comment:
updated the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3921868153
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -188,6 +188,57 @@ class CastRulesTest {
.build())
.build();
+ /** {@code [1, 2, 3]}, the design's running array value. */
+ private static final Variant VARIANT_INT_ARRAY =
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().of(2))
+ .add(Variant.newBuilder().of(3))
+ .build();
+
+ /** {@code [1, null, 3]}, an array carrying a JSON null element. */
+ private static final Variant VARIANT_INT_ARRAY_WITH_NULL =
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().ofNull())
+ .add(Variant.newBuilder().of(3))
+ .build();
+
+ /** {@code ["1", "2"]}, an array of stored strings that a numeric leaf must not parse. */
+ private static final Variant VARIANT_STRING_ARRAY =
Review Comment:
what about different length?
```java
Variant.newBuilder()
.array()
.add(Variant.newBuilder().of("1"))
.add(Variant.newBuilder().of("22"))
.add(Variant.newBuilder().of("333"))
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924887349
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -379,6 +384,116 @@ private static List<TestSetSpec> variantCasts() {
TINYINT()));
}
+ private static List<TestSetSpec> variantArrayCasts() {
+ return List.of(
+ TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to an ARRAY")
+ .onFieldsWithData("unused")
+ .andDataTypes(STRING())
+ // ARRAY: each element casts by the same VARIANT-to-element rule.
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, 2, 3},
+ ARRAY(INT()).notNull())
+ // an approximate leaf takes any numeric kind
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(DOUBLE())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<DOUBLE>)",
+ new Double[] {1.0, 2.0, 3.0},
+ ARRAY(DOUBLE()).notNull())
+ // each element renders to string like the scalar cast
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<STRING>)",
+ new String[] {"1", "2", "3"},
+ ARRAY(STRING()).notNull())
+ // a heterogeneous array renders every element to string
+ .testResult(
+ call("PARSE_JSON", "[1, \"a\", 2, \"b\"]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, \"a\", 2, \"b\"]') AS ARRAY<STRING>)",
+ new String[] {"1", "a", "2", "b"},
+ ARRAY(STRING()).notNull())
+ .testResult(
+ call("PARSE_JSON", "[]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[]') AS ARRAY<INT>)",
+ new Integer[] {},
+ ARRAY(INT()).notNull())
+ // a JSON null element maps to SQL NULL for a nullable element type
+ .testResult(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, null, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, null, 3},
+ ARRAY(INT()).notNull())
+ // a JSON null element fails a NOT NULL element type
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT().notNull())),
Review Comment:
Created a ticket for this:
https://issues.apache.org/jira/browse/FLINK-40545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924825849
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -167,27 +168,83 @@ class CastRulesTest {
/** U+1D54F, one code point but two UTF-16 units and four UTF-8 bytes. */
private static final String NON_BMP = "𝕏";
+ private static final VariantBuilder VARIANT_BUILDER = Variant.newBuilder();
private static final Variant VARIANT_ARRAY =
- Variant.newBuilder()
+ VARIANT_BUILDER
.array()
- .add(Variant.newBuilder().of(1))
- .add(Variant.newBuilder().of("two"))
- .add(Variant.newBuilder().of(false))
- .add(Variant.newBuilder().ofNull())
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of("two"))
+ .add(VARIANT_BUILDER.of(false))
+ .add(VARIANT_BUILDER.ofNull())
.build();
private static final Variant VARIANT_OBJECT =
- Variant.newBuilder()
+ VARIANT_BUILDER
.object()
.add(
"k",
- Variant.newBuilder()
+ VARIANT_BUILDER
.array()
- .add(Variant.newBuilder().of(1))
- .add(Variant.newBuilder().of(2))
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of(2))
.build())
.build();
+ /** {@code [1, 2, 3]}, the design's running array value. */
+ private static final Variant VARIANT_INT_ARRAY =
+ VARIANT_BUILDER
+ .array()
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of(2))
+ .add(VARIANT_BUILDER.of(3))
+ .build();
+
+ /** {@code [1, null, 3]}, an array carrying a JSON null element. */
Review Comment:
https://github.com/apache/flink/pull/29073#discussion_r3924823272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3923538990
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -379,6 +384,113 @@ private static List<TestSetSpec> variantCasts() {
TINYINT()));
}
+ private static List<TestSetSpec> variantArrayCasts() {
+ final String obj = "{\"id\": 7, \"name\": \"ada\", \"active\": true}";
Review Comment:
Inlined it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924799242
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -379,6 +384,116 @@ private static List<TestSetSpec> variantCasts() {
TINYINT()));
}
+ private static List<TestSetSpec> variantArrayCasts() {
+ return List.of(
+ TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to an ARRAY")
+ .onFieldsWithData("unused")
+ .andDataTypes(STRING())
+ // ARRAY: each element casts by the same VARIANT-to-element rule.
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, 2, 3},
+ ARRAY(INT()).notNull())
+ // an approximate leaf takes any numeric kind
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(DOUBLE())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<DOUBLE>)",
+ new Double[] {1.0, 2.0, 3.0},
+ ARRAY(DOUBLE()).notNull())
+ // each element renders to string like the scalar cast
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<STRING>)",
+ new String[] {"1", "2", "3"},
+ ARRAY(STRING()).notNull())
+ // a heterogeneous array renders every element to string
+ .testResult(
+ call("PARSE_JSON", "[1, \"a\", 2, \"b\"]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, \"a\", 2, \"b\"]') AS ARRAY<STRING>)",
+ new String[] {"1", "a", "2", "b"},
+ ARRAY(STRING()).notNull())
+ .testResult(
+ call("PARSE_JSON", "[]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[]') AS ARRAY<INT>)",
+ new Integer[] {},
+ ARRAY(INT()).notNull())
+ // a JSON null element maps to SQL NULL for a nullable element type
+ .testResult(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, null, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, null, 3},
+ ARRAY(INT()).notNull())
+ // a JSON null element fails a NOT NULL element type
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT().notNull())),
Review Comment:
there's no dedicated Table API `parseJson()` — `PARSE_JSON` exists only as `BuiltInFunctionDefinitions.PARSE_JSON`, invoked via `call()`. So `call()` can't be avoided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924823272
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/AbstractVariantToConstructedCastRule.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+
+/**
+ * Base class for the rules that cast a {@link
+ * org.apache.flink.table.types.logical.LogicalTypeRoot#VARIANT} to a constructed target, imposing a
+ * schema on a variant. A constructed cast is the scalar cast applied to every leaf plus a shape
+ * check at each level, so the recursion bottoms out at the same scalar cast the primitive and
+ * string rules perform and no new leaf semantics are introduced.
+ *
+ * <p>A constructed cast can always fail, on a shape mismatch, an unreadable leaf, or a missing
+ * {@code NOT NULL} field, so {@code TRY_CAST} wraps the whole value and returns {@code NULL} for
+ * any failure rather than a partial result.
+ */
+abstract class AbstractVariantToConstructedCastRule<OUT>
+ extends AbstractNullAwareCodeGeneratorCastRule<Variant, OUT> {
+
+ protected AbstractVariantToConstructedCastRule(CastRulePredicate predicate) {
+ super(predicate);
+ }
+
+ @Override
+ public boolean canFail(LogicalType inputLogicalType, LogicalType targetLogicalType) {
+ return true;
+ }
+
+ /**
+ * Treats a variant that stores a JSON {@code null} as a {@code NULL} input, so a top-level JSON
+ * null casts to SQL {@code NULL} before any shape check runs. Only applied for a nullable
+ * target: a {@code NOT NULL} result cannot carry {@code NULL}, so a null-valued variant then
+ * fails the shape check as a regular mismatch.
Review Comment:
Replaced all the JSON to VARIANT
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}. The exception is a {@code VARIANT}
+ * element type: there the element cast is the identity, so a JSON {@code null} element is kept as a
+ * variant null rather than downgraded to SQL {@code NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
+ /* Example generated code for ARRAY<INT>:
+
+ int arraySize$2 =
+ org.apache.flink.table.runtime.functions.VariantCastUtils.arraySize(
+ variant$1, "ARRAY<INT>");
+ java.lang.Integer[] objArray$3 = new java.lang.Integer[arraySize$2];
+ for (int i$4 = 0; i$4 < arraySize$2; i$4++) {
+ org.apache.flink.types.variant.Variant element$5 = variant$1.getElement(i$4);
+ if (!element$5.isNull()) {
+ result$6 =
+ ((int) org.apache.flink.table.runtime.functions.VariantCastUtils.toIntegral(
+ element$5, -2147483648L, 2147483647L, "INTEGER"));
+ objArray$3[i$4] = result$6;
+ }
+ }
+ result$0 = new org.apache.flink.table.data.GenericArrayData(objArray$3);
+
+ A JSON null element leaves the slot null (SQL NULL); a NOT NULL element type emits a throw instead.
+
+ */
+ @Override
+ protected String generateCodeBlockInternal(
+ CodeGeneratorCastRule.Context context,
+ String inputTerm,
+ String returnVariable,
+ LogicalType inputLogicalType,
+ LogicalType targetLogicalType) {
+ final LogicalType elementType = ((ArrayType) targetLogicalType).getElementType();
+ final String elementTypeTerm = arrayElementType(elementType);
+ final String sizeTerm = newName(context.getCodeGeneratorContext(), "arraySize");
+ final String arrayTerm = newName(context.getCodeGeneratorContext(), "objArray");
+ final String elementTerm = newName(context.getCodeGeneratorContext(), "element");
+
+ // For a typed element the JSON null is handled in the loop below, so the inner cast is the
Review Comment:
https://github.com/apache/flink/pull/29073#discussion_r3924823272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924850425
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java:
##########
@@ -666,6 +668,12 @@ private static boolean supportsCasting(
return supportsStructuredCasting(
sourceType, targetType, (s, t) -> supportsCasting(s, t, allowExplicit));
+ } else if (sourceRoot == VARIANT && targetRoot == ARRAY) {
+ // A variant array casts to ARRAY<T> when VARIANT casts to the single element type T.
+ // Explicit only, so no accidental coercion. Each runtime element is cast to T by the
+ // array cast rule; a per-element mismatch fails there, not here.
+ return allowExplicit
+ && supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);
Review Comment:
We should keep the `allowExplicit` so the user must write `CAST(v AS ARRAY<...>)` the cast to an ARRAY from a variant is explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29073:
URL: https://github.com/apache/flink/pull/29073#issuecomment-5512862200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "230c20113a20b5383332150d511a9631b4479c8a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "230c20113a20b5383332150d511a9631b4479c8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 230c20113a20b5383332150d511a9631b4479c8a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3922741291
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -138,245 +139,339 @@ Stream<TestSetSpec> getTestSetSpecs() {
}
private static List<TestSetSpec> variantCasts() {
- // A variant is produced with PARSE_JSON since there is no VARIANT literal. Numeric casts
- // succeed only when the value is preserved exactly, otherwise CAST fails and TRY_CAST
- // returns NULL.
- return List.of(
- TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to a primitive")
- .onFieldsWithData("unused")
- .andDataTypes(STRING())
- // An integer converts to any integer target while the value stays in
- // range, and to FLOAT or DOUBLE which are approximate by definition.
- .testResult(
- call("PARSE_JSON", "42").cast(TINYINT()),
- "CAST(PARSE_JSON('42') AS TINYINT)",
- (byte) 42,
- TINYINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(SMALLINT()),
- "CAST(PARSE_JSON('42') AS SMALLINT)",
- (short) 42,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(INT()),
- "CAST(PARSE_JSON('42') AS INT)",
- 42,
- INT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(BIGINT()),
- "CAST(PARSE_JSON('42') AS BIGINT)",
- 42L,
- BIGINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(FLOAT()),
- "CAST(PARSE_JSON('42') AS FLOAT)",
- 42.0f,
- FLOAT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(DOUBLE()),
- "CAST(PARSE_JSON('42') AS DOUBLE)",
- 42.0d,
- DOUBLE().notNull())
- // An out-of-range value is rejected rather than wrapped.
- .testResult(
- call("PARSE_JSON", "1000").cast(SMALLINT()),
- "CAST(PARSE_JSON('1000') AS SMALLINT)",
- (short) 1000,
- SMALLINT().notNull())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "1000").cast(TINYINT()), "overflowed")
- .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS TINYINT)", "overflowed")
- .testResult(
- call("PARSE_JSON", "1000").tryCast(TINYINT()),
- "TRY_CAST(PARSE_JSON('1000') AS TINYINT)",
- null,
- TINYINT())
- // A decimal reaches an integer target when it is already integral.
- .testResult(
- call("PARSE_JSON", "7.0").cast(INT()),
- "CAST(PARSE_JSON('7.0') AS INT)",
- 7,
- INT().notNull())
- // A fractional value is rejected, since converting would drop digits.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(INT()), "lose precision")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(INT()),
- "TRY_CAST(PARSE_JSON('123.456') AS INT)",
- null,
- INT())
- // A DECIMAL target has to hold the value exactly.
- .testResult(
- call("PARSE_JSON", "123.456").cast(DECIMAL(6, 3)),
- "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))",
- new BigDecimal("123.456"),
- DECIMAL(6, 3).notNull())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(DECIMAL(6, 2)), "lose precision")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(6, 2)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6, 2))",
- null,
- DECIMAL(6, 2))
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(DECIMAL(5, 3)), "overflowed")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(5, 3)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5, 3))",
- null,
- DECIMAL(5, 3))
- // An integer is exact, so it reaches a DECIMAL that has room for it.
- .testResult(
- call("PARSE_JSON", "42").cast(DECIMAL(5, 2)),
- "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))",
- new BigDecimal("42.00"),
- DECIMAL(5, 2).notNull())
- // A decimal reaches an approximate target, where losing digits is expected.
- .testResult(
- call("PARSE_JSON", "123.456").cast(FLOAT()),
- "CAST(PARSE_JSON('123.456') AS FLOAT)",
- 123.456f,
- FLOAT().notNull())
- .testResult(
- call("PARSE_JSON", "123.456").cast(DOUBLE()),
- "CAST(PARSE_JSON('123.456') AS DOUBLE)",
- 123.456d,
- DOUBLE().notNull())
- // A magnitude the target cannot represent is still rejected.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "1e40").cast(FLOAT()), "overflowed")
- .testResult(
- call("PARSE_JSON", "1e40").tryCast(FLOAT()),
- "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)",
- null,
- FLOAT())
- .testResult(
- call("PARSE_JSON", "1e20").cast(DOUBLE()),
- "CAST(PARSE_JSON('1e20') AS DOUBLE)",
- 1e20,
- DOUBLE().notNull())
- .testResult(
- call("PARSE_JSON", "true").cast(BOOLEAN()),
- "CAST(PARSE_JSON('true') AS BOOLEAN)",
- true,
- BOOLEAN().notNull())
- // CAST returns the raw scalar value (string unquoted)
- .testResult(
- call("PARSE_JSON", "\"foo\"").cast(STRING()),
- "CAST(PARSE_JSON('\"foo\"') AS STRING)",
- "foo",
- STRING().notNull())
- .testResult(
- call("PARSE_JSON", "123.456").cast(STRING()),
- "CAST(PARSE_JSON('123.456') AS STRING)",
- "123.456",
- STRING().notNull())
- // The rendering matches a regular cast of the stored kind, so a boolean
- // becomes TRUE rather than the JSON true.
- .testResult(
- call("PARSE_JSON", "true").cast(STRING()),
- "CAST(PARSE_JSON('true') AS STRING)",
- "TRUE",
- STRING().notNull())
- // An object or array has no scalar value, so the error points to
- // JSON_STRING.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "[\"a\", \"b\"]").cast(STRING()), "JSON_STRING")
- .testResult(
- call("PARSE_JSON", "[\"a\", \"b\"]").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)",
- null,
- STRING())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "{\"a\": 1}").cast(STRING()), "JSON_STRING")
- .testResult(
- call("PARSE_JSON", "{\"a\": 1}").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
- null,
- STRING())
- // A bounded CHAR/VARCHAR target trims a longer value, and CHAR pads a
- // shorter one to its fixed width, the same as a regular cast into it.
- .testResult(
- call("PARSE_JSON", "\"ab\"").cast(VARCHAR(3)),
- "CAST(PARSE_JSON('\"ab\"') AS VARCHAR(3))",
- "ab",
- VARCHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"foobar\"").cast(VARCHAR(3)),
- "CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))",
- "foo",
- VARCHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"foobar\"").tryCast(VARCHAR(3)),
- "TRY_CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))",
- "foo",
- VARCHAR(3))
- .testResult(
- call("PARSE_JSON", "\"abc\"").cast(CHAR(3)),
- "CAST(PARSE_JSON('\"abc\"') AS CHAR(3))",
- "abc",
- CHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"abcdef\"").cast(CHAR(3)),
- "CAST(PARSE_JSON('\"abcdef\"') AS CHAR(3))",
- "abc",
- CHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"ab\"").cast(CHAR(5)),
- "CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
- "ab ",
- CHAR(5).notNull())
- .testResult(
- call("PARSE_JSON", "\"ab\"").tryCast(CHAR(5)),
- "TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
- "ab ",
- CHAR(5))
- // A variant holding a JSON null casts to SQL NULL, not to the text 'null'.
- // The length of that text must not be checked against the target either.
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(STRING()),
- "CAST(TRY_PARSE_JSON('null') AS STRING)",
- null,
- STRING())
- .testResult(
- call("PARSE_JSON", "null").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('null') AS STRING)",
- null,
- STRING())
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(CHAR(2)),
- "CAST(TRY_PARSE_JSON('null') AS CHAR(2))",
- null,
- CHAR(2))
- .testResult(
- call("PARSE_JSON", "null").tryCast(VARCHAR(2)),
- "TRY_CAST(PARSE_JSON('null') AS VARCHAR(2))",
- null,
- VARCHAR(2))
- // TRY_CAST of a value whose kind does not match the target returns NULL
- .testResult(
- call("PARSE_JSON", "\"foo\"").tryCast(INT()),
- "TRY_CAST(PARSE_JSON('\"foo\"') AS INT)",
- null,
- INT())
- // A variant that stores a JSON null casts to SQL NULL when the target is
- // nullable: a nullable variant source, or TRY_CAST which forces nullable.
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(INT()),
- "CAST(TRY_PARSE_JSON('null') AS INT)",
- null,
- INT())
- .testResult(
- call("PARSE_JSON", "null").tryCast(BOOLEAN()),
- "TRY_CAST(PARSE_JSON('null') AS BOOLEAN)",
- null,
- BOOLEAN())
- // A nullable variant with a concrete value still casts normally.
- .testResult(
- call("TRY_PARSE_JSON", "42").cast(TINYINT()),
- "CAST(TRY_PARSE_JSON('42') AS TINYINT)",
- (byte) 42,
- TINYINT()));
+ return List.of(variantPrimitiveCasts(), variantArrayCasts());
+ }
+
+ private static TestSetSpec variantPrimitiveCasts() {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #29073:
URL: https://github.com/apache/flink/pull/29073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3921850328
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -138,245 +139,339 @@ Stream<TestSetSpec> getTestSetSpecs() {
}
private static List<TestSetSpec> variantCasts() {
- // A variant is produced with PARSE_JSON since there is no VARIANT literal. Numeric casts
- // succeed only when the value is preserved exactly, otherwise CAST fails and TRY_CAST
- // returns NULL.
- return List.of(
- TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to a primitive")
- .onFieldsWithData("unused")
- .andDataTypes(STRING())
- // An integer converts to any integer target while the value stays in
- // range, and to FLOAT or DOUBLE which are approximate by definition.
- .testResult(
- call("PARSE_JSON", "42").cast(TINYINT()),
- "CAST(PARSE_JSON('42') AS TINYINT)",
- (byte) 42,
- TINYINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(SMALLINT()),
- "CAST(PARSE_JSON('42') AS SMALLINT)",
- (short) 42,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(INT()),
- "CAST(PARSE_JSON('42') AS INT)",
- 42,
- INT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(BIGINT()),
- "CAST(PARSE_JSON('42') AS BIGINT)",
- 42L,
- BIGINT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(FLOAT()),
- "CAST(PARSE_JSON('42') AS FLOAT)",
- 42.0f,
- FLOAT().notNull())
- .testResult(
- call("PARSE_JSON", "42").cast(DOUBLE()),
- "CAST(PARSE_JSON('42') AS DOUBLE)",
- 42.0d,
- DOUBLE().notNull())
- // An out-of-range value is rejected rather than wrapped.
- .testResult(
- call("PARSE_JSON", "1000").cast(SMALLINT()),
- "CAST(PARSE_JSON('1000') AS SMALLINT)",
- (short) 1000,
- SMALLINT().notNull())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "1000").cast(TINYINT()), "overflowed")
- .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS TINYINT)", "overflowed")
- .testResult(
- call("PARSE_JSON", "1000").tryCast(TINYINT()),
- "TRY_CAST(PARSE_JSON('1000') AS TINYINT)",
- null,
- TINYINT())
- // A decimal reaches an integer target when it is already integral.
- .testResult(
- call("PARSE_JSON", "7.0").cast(INT()),
- "CAST(PARSE_JSON('7.0') AS INT)",
- 7,
- INT().notNull())
- // A fractional value is rejected, since converting would drop digits.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(INT()), "lose precision")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(INT()),
- "TRY_CAST(PARSE_JSON('123.456') AS INT)",
- null,
- INT())
- // A DECIMAL target has to hold the value exactly.
- .testResult(
- call("PARSE_JSON", "123.456").cast(DECIMAL(6, 3)),
- "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))",
- new BigDecimal("123.456"),
- DECIMAL(6, 3).notNull())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(DECIMAL(6, 2)), "lose precision")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(6, 2)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6, 2))",
- null,
- DECIMAL(6, 2))
- .testTableApiRuntimeError(
- call("PARSE_JSON", "123.456").cast(DECIMAL(5, 3)), "overflowed")
- .testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(5, 3)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5, 3))",
- null,
- DECIMAL(5, 3))
- // An integer is exact, so it reaches a DECIMAL that has room for it.
- .testResult(
- call("PARSE_JSON", "42").cast(DECIMAL(5, 2)),
- "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))",
- new BigDecimal("42.00"),
- DECIMAL(5, 2).notNull())
- // A decimal reaches an approximate target, where losing digits is expected.
- .testResult(
- call("PARSE_JSON", "123.456").cast(FLOAT()),
- "CAST(PARSE_JSON('123.456') AS FLOAT)",
- 123.456f,
- FLOAT().notNull())
- .testResult(
- call("PARSE_JSON", "123.456").cast(DOUBLE()),
- "CAST(PARSE_JSON('123.456') AS DOUBLE)",
- 123.456d,
- DOUBLE().notNull())
- // A magnitude the target cannot represent is still rejected.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "1e40").cast(FLOAT()), "overflowed")
- .testResult(
- call("PARSE_JSON", "1e40").tryCast(FLOAT()),
- "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)",
- null,
- FLOAT())
- .testResult(
- call("PARSE_JSON", "1e20").cast(DOUBLE()),
- "CAST(PARSE_JSON('1e20') AS DOUBLE)",
- 1e20,
- DOUBLE().notNull())
- .testResult(
- call("PARSE_JSON", "true").cast(BOOLEAN()),
- "CAST(PARSE_JSON('true') AS BOOLEAN)",
- true,
- BOOLEAN().notNull())
- // CAST returns the raw scalar value (string unquoted)
- .testResult(
- call("PARSE_JSON", "\"foo\"").cast(STRING()),
- "CAST(PARSE_JSON('\"foo\"') AS STRING)",
- "foo",
- STRING().notNull())
- .testResult(
- call("PARSE_JSON", "123.456").cast(STRING()),
- "CAST(PARSE_JSON('123.456') AS STRING)",
- "123.456",
- STRING().notNull())
- // The rendering matches a regular cast of the stored kind, so a boolean
- // becomes TRUE rather than the JSON true.
- .testResult(
- call("PARSE_JSON", "true").cast(STRING()),
- "CAST(PARSE_JSON('true') AS STRING)",
- "TRUE",
- STRING().notNull())
- // An object or array has no scalar value, so the error points to
- // JSON_STRING.
- .testTableApiRuntimeError(
- call("PARSE_JSON", "[\"a\", \"b\"]").cast(STRING()), "JSON_STRING")
- .testResult(
- call("PARSE_JSON", "[\"a\", \"b\"]").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)",
- null,
- STRING())
- .testTableApiRuntimeError(
- call("PARSE_JSON", "{\"a\": 1}").cast(STRING()), "JSON_STRING")
- .testResult(
- call("PARSE_JSON", "{\"a\": 1}").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
- null,
- STRING())
- // A bounded CHAR/VARCHAR target trims a longer value, and CHAR pads a
- // shorter one to its fixed width, the same as a regular cast into it.
- .testResult(
- call("PARSE_JSON", "\"ab\"").cast(VARCHAR(3)),
- "CAST(PARSE_JSON('\"ab\"') AS VARCHAR(3))",
- "ab",
- VARCHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"foobar\"").cast(VARCHAR(3)),
- "CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))",
- "foo",
- VARCHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"foobar\"").tryCast(VARCHAR(3)),
- "TRY_CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))",
- "foo",
- VARCHAR(3))
- .testResult(
- call("PARSE_JSON", "\"abc\"").cast(CHAR(3)),
- "CAST(PARSE_JSON('\"abc\"') AS CHAR(3))",
- "abc",
- CHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"abcdef\"").cast(CHAR(3)),
- "CAST(PARSE_JSON('\"abcdef\"') AS CHAR(3))",
- "abc",
- CHAR(3).notNull())
- .testResult(
- call("PARSE_JSON", "\"ab\"").cast(CHAR(5)),
- "CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
- "ab ",
- CHAR(5).notNull())
- .testResult(
- call("PARSE_JSON", "\"ab\"").tryCast(CHAR(5)),
- "TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
- "ab ",
- CHAR(5))
- // A variant holding a JSON null casts to SQL NULL, not to the text 'null'.
- // The length of that text must not be checked against the target either.
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(STRING()),
- "CAST(TRY_PARSE_JSON('null') AS STRING)",
- null,
- STRING())
- .testResult(
- call("PARSE_JSON", "null").tryCast(STRING()),
- "TRY_CAST(PARSE_JSON('null') AS STRING)",
- null,
- STRING())
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(CHAR(2)),
- "CAST(TRY_PARSE_JSON('null') AS CHAR(2))",
- null,
- CHAR(2))
- .testResult(
- call("PARSE_JSON", "null").tryCast(VARCHAR(2)),
- "TRY_CAST(PARSE_JSON('null') AS VARCHAR(2))",
- null,
- VARCHAR(2))
- // TRY_CAST of a value whose kind does not match the target returns NULL
- .testResult(
- call("PARSE_JSON", "\"foo\"").tryCast(INT()),
- "TRY_CAST(PARSE_JSON('\"foo\"') AS INT)",
- null,
- INT())
- // A variant that stores a JSON null casts to SQL NULL when the target is
- // nullable: a nullable variant source, or TRY_CAST which forces nullable.
- .testResult(
- call("TRY_PARSE_JSON", "null").cast(INT()),
- "CAST(TRY_PARSE_JSON('null') AS INT)",
- null,
- INT())
- .testResult(
- call("PARSE_JSON", "null").tryCast(BOOLEAN()),
- "TRY_CAST(PARSE_JSON('null') AS BOOLEAN)",
- null,
- BOOLEAN())
- // A nullable variant with a concrete value still casts normally.
- .testResult(
- call("TRY_PARSE_JSON", "42").cast(TINYINT()),
- "CAST(TRY_PARSE_JSON('42') AS TINYINT)",
- (byte) 42,
- TINYINT()));
+ return List.of(variantPrimitiveCasts(), variantArrayCasts());
+ }
+
+ private static TestSetSpec variantPrimitiveCasts() {
Review Comment:
can we keep `List` here?
then diff will be less drastic
I guess anyway there is no major reason to go away from `List`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "manner (via GitHub)" <gi...@apache.org>.
manner commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3923324520
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -379,6 +384,113 @@ private static List<TestSetSpec> variantCasts() {
TINYINT()));
}
+ private static List<TestSetSpec> variantArrayCasts() {
+ final String obj = "{\"id\": 7, \"name\": \"ada\", \"active\": true}";
Review Comment:
Is there a reason this is the only test string that's not inlined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924820610
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/AbstractVariantToConstructedCastRule.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+
+/**
+ * Base class for the rules that cast a {@link
+ * org.apache.flink.table.types.logical.LogicalTypeRoot#VARIANT} to a constructed target, imposing a
Review Comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3923987296
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
+ /* Example generated code for ARRAY<INT>:
+
+ int arraySize$2 =
+ org.apache.flink.table.runtime.functions.VariantCastUtils.arraySize(
+ variant$1, "ARRAY<INT>");
+ java.lang.Integer[] objArray$3 = new java.lang.Integer[arraySize$2];
+ for (int i$4 = 0; i$4 < arraySize$2; i$4++) {
+ org.apache.flink.types.variant.Variant element$5 = variant$1.getElement(i$4);
+ if (!element$5.isNull()) {
+ result$6 =
+ ((int) org.apache.flink.table.runtime.functions.VariantCastUtils.toIntegral(
+ element$5, -2147483648L, 2147483647L, "INTEGER"));
+ objArray$3[i$4] = result$6;
+ }
+ }
+ result$0 = new org.apache.flink.table.data.GenericArrayData(objArray$3);
+
+ A JSON null element leaves the slot null (SQL NULL); a NOT NULL element type emits a throw instead.
Review Comment:
Excellent catch @mateczagany! I have addressed your feedback and the codegen should be able to handle this now. I have added a test:
```
CAST VARIANT -> ARRAY<VARIANT>
[1, null, 3]
```
It is green now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37926][table] Support casting from VARIANT to ARRAY [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #29073:
URL: https://github.com/apache/flink/pull/29073#discussion_r3924446501
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
Review Comment:
```suggestion
`VARIANT`, so it casts to the element type `T` by the same rules, recursively.
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
+
+- Each element casts to `T`. A JSON null element maps to SQL `NULL` when `T` is nullable and fails
+ the cast when `T` is `NOT NULL`. An empty array casts to an empty `ARRAY<T>`.
+- `ARRAY<VARIANT>` is the identity leaf: it shreds one level and keeps each element as a variant. A
Review Comment:
```suggestion
- `ARRAY<VARIANT>` is the identity element: it shreds one level and keeps each element as a variant. A
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java:
##########
@@ -666,6 +668,12 @@ private static boolean supportsCasting(
return supportsStructuredCasting(
sourceType, targetType, (s, t) -> supportsCasting(s, t, allowExplicit));
+ } else if (sourceRoot == VARIANT && targetRoot == ARRAY) {
+ // A variant array casts to ARRAY<T> when VARIANT casts to the single element type T.
+ // Explicit only, so no accidental coercion. Each runtime element is cast to T by the
+ // array cast rule; a per-element mismatch fails there, not here.
+ return allowExplicit
+ && supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);
Review Comment:
why this?
```suggestion
return supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1588,6 +1588,33 @@ rendering. Use `JSON_STRING` for the JSON representation instead, where a string
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
+A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
+casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
+`VARIANT`, so it casts to the element type `T` by the same rules, recursively, bottoming out at the
+scalar cast above. A leaf is never parsed either, so a stored string does not reach an integer
+target. Cast the leaf to `STRING` first and convert with a regular cast.
+
+- Each element casts to `T`. A JSON null element maps to SQL `NULL` when `T` is nullable and fails
+ the cast when `T` is `NOT NULL`. An empty array casts to an empty `ARRAY<T>`.
+- `ARRAY<VARIANT>` is the identity leaf: it shreds one level and keeps each element as a variant. A
+ JSON null element stays a variant null rather than becoming SQL `NULL`.
+
+If any element cast fails, the whole cast fails, and `TRY_CAST` returns `NULL` for the entire value
+rather than a partial result. An element type with no variant counterpart, such as
+`ARRAY<INTERVAL YEAR TO MONTH>`, is rejected at validation. A top-level JSON `null` casts to SQL `NULL` for a nullable target before any
+shape check runs.
Review Comment:
```suggestion
`ARRAY<INTERVAL YEAR TO MONTH>`, is rejected at validation.
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/AbstractVariantToConstructedCastRule.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+
+/**
+ * Base class for the rules that cast a {@link
+ * org.apache.flink.table.types.logical.LogicalTypeRoot#VARIANT} to a constructed target, imposing a
Review Comment:
nit: import for better readability
```suggestion
* LogicalTypeRoot#VARIANT} to a constructed target, imposing a
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/AbstractVariantToConstructedCastRule.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+
+/**
+ * Base class for the rules that cast a {@link
+ * org.apache.flink.table.types.logical.LogicalTypeRoot#VARIANT} to a constructed target, imposing a
+ * schema on a variant. A constructed cast is the scalar cast applied to every leaf plus a shape
+ * check at each level, so the recursion bottoms out at the same scalar cast the primitive and
+ * string rules perform and no new leaf semantics are introduced.
+ *
+ * <p>A constructed cast can always fail, on a shape mismatch, an unreadable leaf, or a missing
+ * {@code NOT NULL} field, so {@code TRY_CAST} wraps the whole value and returns {@code NULL} for
+ * any failure rather than a partial result.
+ */
+abstract class AbstractVariantToConstructedCastRule<OUT>
+ extends AbstractNullAwareCodeGeneratorCastRule<Variant, OUT> {
+
+ protected AbstractVariantToConstructedCastRule(CastRulePredicate predicate) {
+ super(predicate);
+ }
+
+ @Override
+ public boolean canFail(LogicalType inputLogicalType, LogicalType targetLogicalType) {
+ return true;
+ }
+
+ /**
+ * Treats a variant that stores a JSON {@code null} as a {@code NULL} input, so a top-level JSON
+ * null casts to SQL {@code NULL} before any shape check runs. Only applied for a nullable
+ * target: a {@code NOT NULL} result cannot carry {@code NULL}, so a null-valued variant then
+ * fails the shape check as a regular mismatch.
Review Comment:
I would not mention JSON here. Variant is not JSON, it is a dedicated type.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.casting;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.planner.codegen.CodeGenUtils;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.Variant;
+
+import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.newArray;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
+import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
+
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeRoot#ARRAY} cast rule.
+ *
+ * <p>The variant must be an array, otherwise the cast fails. Each element is itself a variant and
+ * casts to the target element type by the full {@code VARIANT}-to-element rule, recursively. An
+ * element that stores a JSON {@code null} maps to SQL {@code NULL} when the element type is
+ * nullable and fails the cast when it is {@code NOT NULL}. The exception is a {@code VARIANT}
+ * element type: there the element cast is the identity, so a JSON {@code null} element is kept as a
+ * variant null rather than downgraded to SQL {@code NULL}.
+ */
+class VariantToArrayCastRule extends AbstractVariantToConstructedCastRule<ArrayData> {
+
+ static final VariantToArrayCastRule INSTANCE = new VariantToArrayCastRule();
+
+ private VariantToArrayCastRule() {
+ super(
+ CastRulePredicate.builder()
+ .predicate(
+ (input, target) ->
+ input.is(LogicalTypeRoot.VARIANT)
+ && target.is(LogicalTypeRoot.ARRAY)
+ && CastRuleProvider.resolve(
+ input,
+ ((ArrayType) target)
+ .getElementType())
+ != null)
+ .build());
+ }
+
+ /* Example generated code for ARRAY<INT>:
+
+ int arraySize$2 =
+ org.apache.flink.table.runtime.functions.VariantCastUtils.arraySize(
+ variant$1, "ARRAY<INT>");
+ java.lang.Integer[] objArray$3 = new java.lang.Integer[arraySize$2];
+ for (int i$4 = 0; i$4 < arraySize$2; i$4++) {
+ org.apache.flink.types.variant.Variant element$5 = variant$1.getElement(i$4);
+ if (!element$5.isNull()) {
+ result$6 =
+ ((int) org.apache.flink.table.runtime.functions.VariantCastUtils.toIntegral(
+ element$5, -2147483648L, 2147483647L, "INTEGER"));
+ objArray$3[i$4] = result$6;
+ }
+ }
+ result$0 = new org.apache.flink.table.data.GenericArrayData(objArray$3);
+
+ A JSON null element leaves the slot null (SQL NULL); a NOT NULL element type emits a throw instead.
+
+ */
+ @Override
+ protected String generateCodeBlockInternal(
+ CodeGeneratorCastRule.Context context,
+ String inputTerm,
+ String returnVariable,
+ LogicalType inputLogicalType,
+ LogicalType targetLogicalType) {
+ final LogicalType elementType = ((ArrayType) targetLogicalType).getElementType();
+ final String elementTypeTerm = arrayElementType(elementType);
+ final String sizeTerm = newName(context.getCodeGeneratorContext(), "arraySize");
+ final String arrayTerm = newName(context.getCodeGeneratorContext(), "objArray");
+ final String elementTerm = newName(context.getCodeGeneratorContext(), "element");
+
+ // For a typed element the JSON null is handled in the loop below, so the inner cast is the
Review Comment:
```suggestion
// For a typed element the VARIANT null is handled in the loop below, so the inner cast is the
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -379,6 +384,116 @@ private static List<TestSetSpec> variantCasts() {
TINYINT()));
}
+ private static List<TestSetSpec> variantArrayCasts() {
+ return List.of(
+ TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to an ARRAY")
+ .onFieldsWithData("unused")
+ .andDataTypes(STRING())
+ // ARRAY: each element casts by the same VARIANT-to-element rule.
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, 2, 3},
+ ARRAY(INT()).notNull())
+ // an approximate leaf takes any numeric kind
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(DOUBLE())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<DOUBLE>)",
+ new Double[] {1.0, 2.0, 3.0},
+ ARRAY(DOUBLE()).notNull())
+ // each element renders to string like the scalar cast
+ .testResult(
+ call("PARSE_JSON", "[1, 2, 3]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS ARRAY<STRING>)",
+ new String[] {"1", "2", "3"},
+ ARRAY(STRING()).notNull())
+ // a heterogeneous array renders every element to string
+ .testResult(
+ call("PARSE_JSON", "[1, \"a\", 2, \"b\"]").cast(ARRAY(STRING())),
+ "CAST(PARSE_JSON('[1, \"a\", 2, \"b\"]') AS ARRAY<STRING>)",
+ new String[] {"1", "a", "2", "b"},
+ ARRAY(STRING()).notNull())
+ .testResult(
+ call("PARSE_JSON", "[]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[]') AS ARRAY<INT>)",
+ new Integer[] {},
+ ARRAY(INT()).notNull())
+ // a JSON null element maps to SQL NULL for a nullable element type
+ .testResult(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT())),
+ "CAST(PARSE_JSON('[1, null, 3]') AS ARRAY<INT>)",
+ new Integer[] {1, null, 3},
+ ARRAY(INT()).notNull())
+ // a JSON null element fails a NOT NULL element type
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT().notNull())),
Review Comment:
is there no "Table API" way of parsing JSON? Can we avoid `call()`?
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -167,27 +168,83 @@ class CastRulesTest {
/** U+1D54F, one code point but two UTF-16 units and four UTF-8 bytes. */
private static final String NON_BMP = "𝕏";
+ private static final VariantBuilder VARIANT_BUILDER = Variant.newBuilder();
private static final Variant VARIANT_ARRAY =
- Variant.newBuilder()
+ VARIANT_BUILDER
.array()
- .add(Variant.newBuilder().of(1))
- .add(Variant.newBuilder().of("two"))
- .add(Variant.newBuilder().of(false))
- .add(Variant.newBuilder().ofNull())
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of("two"))
+ .add(VARIANT_BUILDER.of(false))
+ .add(VARIANT_BUILDER.ofNull())
.build();
private static final Variant VARIANT_OBJECT =
- Variant.newBuilder()
+ VARIANT_BUILDER
.object()
.add(
"k",
- Variant.newBuilder()
+ VARIANT_BUILDER
.array()
- .add(Variant.newBuilder().of(1))
- .add(Variant.newBuilder().of(2))
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of(2))
.build())
.build();
+ /** {@code [1, 2, 3]}, the design's running array value. */
+ private static final Variant VARIANT_INT_ARRAY =
+ VARIANT_BUILDER
+ .array()
+ .add(VARIANT_BUILDER.of(1))
+ .add(VARIANT_BUILDER.of(2))
+ .add(VARIANT_BUILDER.of(3))
+ .build();
+
+ /** {@code [1, null, 3]}, an array carrying a JSON null element. */
Review Comment:
```suggestion
/** {@code [1, null, 3]}, an array carrying a VARIANT null element. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org