You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/07/16 09:56:03 UTC
[PR] [FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string [flink]
raminqaf opened a new pull request, #28758:
URL: https://github.com/apache/flink/pull/28758
## What is the purpose of the change
Follow-up to the initial `VARIANT`-to-primitive cast support (FLINK-37925). It tightens numeric cast semantics and completes the set of supported cast targets:
- Numeric casts from a `VARIANT` now reject values that do not fit the target instead of silently wrapping. An out-of-range integer or an overflowing `DECIMAL`
fails `CAST` and returns `NULL` for `TRY_CAST`. `FLOAT` and `DOUBLE` keep lenient IEEE conversion, where overflow becomes infinity. This follows the behavior of Spark's variant casts.
- `CAST(VARIANT AS CHAR/VARCHAR)` is now allowed and returns the JSON string representation, so the earlier `JSON_STRING`-only restriction and its cast hint are removed.
This makes a `VARIANT`-to-integer cast stricter than a regular numeric narrowing cast, which wraps. That is intentional: a `VARIANT` carries untrusted semi-structured data, so silently corrupting an out-of-range value is worse than failing. `TRY_CAST` is the escape hatch, and the docs show a double-cast pattern for wrap-around narrowing.
## Brief change log
- Add `VariantCastUtils` (flink-table-runtime) that performs checked narrowing: truncate any fraction toward zero, range-check, and throw on overflow. It uses `BigDecimal`/`BigInteger` so it is correct for every stored numeric kind.
- Route integer and `DECIMAL` targets in `VariantToPrimitiveCastRule` through the checked helpers; keep `FLOAT`/`DOUBLE` lenient.
- Express `VARIANT` cast validation in the per-target rules of `LogicalTypeCasts`; allow `CAST` to `CHAR`/`VARCHAR` and drop the `JSON_STRING` cast hint.
- Document `VARIANT` value encoding, the `PARSE_JSON` type mapping, and timestamp microsecond precision; fix the `getInstant` javadoc to reference `Type.TIMESTAMP_LTZ`.
## Verifying this change
This change added tests and can be verified as follows:
- Extended `CastFunctionITCase#variantCasts` with overflow cases (`CAST` fails, `TRY_CAST` returns `NULL`), the `1e20 AS INT` case, and the lenient `FLOAT`/`DOUBLE` cases; the full `CastFunctionITCase` (752 tests) passes.
- Updated `LogicalTypeCastsTest` for the `VARIANT` cast-support matrix.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes, javadoc-only change to the `@PublicEvolving` `Variant` interface (no
signature change)
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, only on the `VARIANT`-to-numeric cast path
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no (it refines existing FLINK-37925 behavior)
- If yes, how is the feature documented? docs and JavaDocs
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [X] Yes (please specify the tool below)
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3690107631
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, targetType);
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, targetType);
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, targetType);
+ }
+ return decimal;
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ final String targetType =
+ String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR", targetLength);
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ targetType));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = variant.get().toString();
Review Comment:
See PR description and https://github.com/apache/flink/pull/28758#discussion_r3690104002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3704703986
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
I implemented in a way that the user casts Bytes -> String and if it is broken it gets a proper error message and sees the bytes. Unfortunately, if we have a STRING and want to cast it to Bytes, this will fail.
If we want to allow this for debugging reasons, which I think is very useful to just see the bytes. One way to solve this is to introduce a `getStringBytes` in `BinaryVariant` that fetches the Binary form of String. This will cause that the implementation diverges from Spark. Can be also done in a Follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3666548107
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ if (value instanceof BigDecimal) {
+ return (BigDecimal) value;
+ }
+ if (value instanceof BigInteger) {
+ return new BigDecimal((BigInteger) value);
+ }
+ if (value instanceof Float || value instanceof Double) {
+ return BigDecimal.valueOf(value.doubleValue());
+ }
+ return BigDecimal.valueOf(value.longValue());
+ }
+
+ // The exact mathematical value of a number, used to detect precision loss in floating-point
+ // casts. Unlike toBigDecimal, floats and doubles use their exact binary value rather than the
+ // shortest round-trippable decimal, so 0.1d becomes 0.1000000000000000055..., not 0.1.
+ private static BigDecimal exactValue(Number value) {
Review Comment:
I have changed the behavior here a bit. Now we are strict with the DECIMAL type too.
`CAST(PARSE_JSON('123.456') AS DECIMAL(5,2))` would fail instead of silently approximating (before it resulted in `123.46`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656122885
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
Review Comment:
can we use typeName instead of hardcoded string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610225458
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
`NaN` is not part of the JSON specification: https://www.json.org/json-en.html
Jackson does not allow it by default and you need to switch a flag to allow it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3718619712
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.binary.StringUtf8Utils;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * <p>A stored binary value has to be well-formed UTF-8, since a character string cannot carry
+ * bytes that no character maps to. Invalid input is rejected rather than decoded into {@code
+ * U+FFFD}, which would silently substitute a character the value never held.
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ value = variant.get().toString();
+ break;
+ case DECIMAL:
+ // toPlainString rather than toString, so that a small scale is not rendered in
+ // scientific notation, matching a regular DECIMAL to string cast.
+ value = variant.getDecimal().toPlainString();
+ break;
+ case STRING:
+ value = variant.getString();
+ break;
+ case BYTES:
+ // SQL reads a binary value as UTF-8, the same as a regular BINARY to string cast.
+ final byte[] utf8 = variant.getBytes();
+ final int invalidAt =
+ StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0, utf8.length);
+ if (invalidAt >= 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast the VARIANT binary value to %s because it is not "
+ + "valid UTF-8; the first invalid byte is at index %d "
+ + "of %d. Cast to BYTES to inspect the raw value, or "
+ + "wrap that in MAKE_VALID_UTF8 to replace every "
+ + "invalid byte with the U+FFFD replacement character.",
+ characterTarget(targetLength, charTarget),
+ invalidAt,
+ utf8.length));
+ }
+ value = new String(utf8, StandardCharsets.UTF_8);
+ break;
+ case DATE:
+ value = DateTimeUtils.formatDate((int) variant.getDate().toEpochDay());
+ break;
+ case TIMESTAMP:
+ // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A
+ // variant keeps microseconds, so the precision is always 6.
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromLocalDateTime(variant.getDateTime()),
+ DateTimeUtils.UTC_ZONE,
+ TIMESTAMP_PRECISION);
+ break;
+ case TIMESTAMP_LTZ:
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromInstant(variant.getInstant()),
+ sessionZone,
+ TIMESTAMP_PRECISION);
+ break;
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ characterTarget(targetLength, charTarget)));
+ default:
+ // An object or array has no scalar rendering.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ }
+ final boolean fits =
+ charTarget ? value.length() == targetLength : value.length() <= targetLength;
Review Comment:
this might be a problem for multibyte characters
eg. such query
```sql
SELECT CAST('𝕏' AS CHAR(1)), TRY_CAST(PARSE_JSON('"𝕏"') AS CHAR(1));
```
returns `𝕏` NULL
I would expect same result for 2 projections
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656077829
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE} keep lenient IEEE conversion (overflow
+ * becomes infinity). A non-numeric variant raises {@link ClassCastException}, which fails
+ * {@code CAST} and yields {@code null} for {@code TRY_CAST}.
*/
private static String numericExpression(String inputTerm, LogicalType target) {
final String number = cast(className(Number.class), methodCall(inputTerm, "get"));
- if (!target.is(LogicalTypeRoot.DECIMAL)) {
- return methodCall(number, numberAccessor(target));
- }
- final DecimalType decimalType = (DecimalType) target;
- return staticCall(
- DecimalData.class,
- "fromBigDecimal",
- constructorCall(BigDecimal.class, methodCall(number, "toString")),
- decimalType.getPrecision(),
- decimalType.getScale());
- }
-
- private static String numberAccessor(LogicalType target) {
switch (target.getTypeRoot()) {
case TINYINT:
- return "byteValue";
+ return staticCall(VariantCastUtils.class, "toByteExact", number);
case SMALLINT:
- return "shortValue";
+ return staticCall(VariantCastUtils.class, "toShortExact", number);
case INTEGER:
- return "intValue";
+ return staticCall(VariantCastUtils.class, "toIntExact", number);
case BIGINT:
- return "longValue";
+ return staticCall(VariantCastUtils.class, "toLongExact", number);
case FLOAT:
- return "floatValue";
+ return methodCall(number, "floatValue");
Review Comment:
Introduced `toFloatExact` and `toDoubleExact`. In case there is a precision loss the CAST will fail and TRY_CAST will return null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656154488
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ if (value instanceof BigDecimal) {
+ return (BigDecimal) value;
+ }
+ if (value instanceof BigInteger) {
+ return new BigDecimal((BigInteger) value);
+ }
+ if (value instanceof Float || value instanceof Double) {
+ return BigDecimal.valueOf(value.doubleValue());
+ }
+ return BigDecimal.valueOf(value.longValue());
+ }
+
+ // The exact mathematical value of a number, used to detect precision loss in floating-point
+ // casts. Unlike toBigDecimal, floats and doubles use their exact binary value rather than the
+ // shortest round-trippable decimal, so 0.1d becomes 0.1000000000000000055..., not 0.1.
+ private static BigDecimal exactValue(Number value) {
Review Comment:
what is the reason in some cases we should use `toBigDecimal` and can not use `exactValue` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613589952
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
+ // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")
Review Comment:
yes, however it does not answer why we expect `NULL` after that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3636605204
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+ * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+ * {@code JSON_STRING}); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException}.
+ *
+ * <p>The target length is enforced strictly, with no padding or truncation: a {@code VARCHAR}
+ * value must not be longer than {@code targetLength}, and a {@code CHAR} value must match it
+ * exactly. A value that does not fit raises {@link TableRuntimeException}.
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : convertToBigDecimal(value);
+ }
+
+ private static BigDecimal convertToBigDecimal(Number number) {
+ if (number == null) {
Review Comment:
Why this null check here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3702542676
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
Good catch! Yes, let's address this in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3594523534
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1523,11 +1523,47 @@ without requiring upfront schema definition. For example, if a new field is adde
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A value
+outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for `TRY_CAST`. Other
+targets require the stored value to be of the matching kind, and a mismatch is handled the same way.
+Casting to `CHAR`/`VARCHAR` (or calling `JSON_STRING`) returns the JSON string representation. Casting
Review Comment:
@twalthr I want to emphasize on this statement. A `CAST(v.a AS STRING`) is the same `VARIANT → STRING` cast. A `v.a` (variant field access) returns another `VARIANT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656188157
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
Review Comment:
if `variant.get() == null`
then `String.valueOf(...)` returns `"null"` (String containing 4 letters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610225458
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
`NaN` and `Infinity` are not part of the JSON specification: https://www.rfc-editor.org/info/rfc8259/#section-6
Jackson does not allow it by default and you need to switch a flag to allow it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3615543738
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
why microsecond not nanoseconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3622420298
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
Review Comment:
did you double check this behavior with other vendors? sounds incorrect to me.
```
while objects and arrays return their JSON
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
Review Comment:
```suggestion
Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows. To narrow a
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
Review Comment:
incomplete list anyway.
```suggestion
example `"foo"`).
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the stored value, then apply a
Review Comment:
```suggestion
`VARIANT`, first cast it to a type that fits the stored value, then apply a
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
Review Comment:
```suggestion
CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after overflow)
```
##########
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java:
##########
@@ -284,11 +285,12 @@ private static Stream<Arguments> testData() {
new VarBinaryType(VarBinaryType.MAX_LENGTH),
false,
true),
+ Arguments.of(new VariantType(), new CharType(), false, true),
+ Arguments.of(new VariantType(), VarCharType.STRING_TYPE, false, true),
// variant identity cast is implicit
Arguments.of(new VariantType(), new VariantType(), true, true),
// TIME, character strings and constructed targets are not castable from variant
Review Comment:
```suggestion
// TIME and constructed targets are not castable from variant
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}, following Spark's variant cast semantics; {@code FLOAT} and {@code DOUBLE}
Review Comment:
not sure if we follow 100%, better remove
```suggestion
* {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE}
```
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
Review Comment:
is it guaranteed that this becomes a `TableRuntimeException` eventually? or how do we treat other cast cases currently?
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java:
##########
@@ -45,6 +51,6 @@ public String generateStringExpression(
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
- return methodCall(inputTerm, "toString");
+ return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);
Review Comment:
according to LogicalTypeCasts, we support VARCHAR and CHAR both with various length. this cast rule should consider the target length as well. I guess it should throw if length exceeds?
##########
flink-core/src/main/java/org/apache/flink/types/variant/Variant.java:
##########
@@ -127,18 +127,19 @@ public interface Variant {
/**
* Get the scalar value of variant as LocalDateTime, if the variant type is {@link
- * Type#TIMESTAMP}.
+ * Type#TIMESTAMP}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#TIMESTAMP}.
*/
LocalDateTime getDateTime() throws VariantTypeException;
/**
- * Get the scalar value of variant as Instant, if the variant type is {@link Type#TIMESTAMP}.
+ * Get the scalar value of variant as Instant, if the variant type is {@link
Review Comment:
can you apply to all JavaDocs in this class?
```suggestion
* Get the scalar value of variant as {@link Instant}, if the variant type is {@link
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
+ // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")
Review Comment:
to conclude on this: whereas for regular casts people opt-in on the overflow, for variant casts they should first extract the correct value before opt-in to overflow.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
Review Comment:
```suggestion
// An integer value outside the target range fails
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
Review Comment:
this is what JSON_STRING should be good for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3636506400
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
Review Comment:
also include the `1e400` example here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3702656723
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -1544,35 +1544,79 @@ Stream<CastTestSpecBuilder> testCases() {
.fromCase(BITMAP(), DEFAULT_BITMAP, DEFAULT_BITMAP.toBytes())
.fromCase(BITMAP(), Bitmap.empty(), Bitmap.empty().toBytes())
.fromCase(BITMAP(), null, null),
- // From VARIANT to primitive types. Numeric targets are lenient: a variant holding
- // any numeric kind converts to the requested numeric type (widening and narrowing).
- // Non-numeric targets are strict: the stored kind must match, otherwise the cast
- // fails and TRY_CAST returns null.
+ // From VARIANT to primitive types. A cast succeeds only when the target holds the
+ // stored value unaltered, except for the approximate FLOAT and DOUBLE.
+ // A character string renders like a regular cast of the stored kind, so these
+ // expectations reuse the constants of the native cases above.
+ CastTestSpecBuilder.testCastTo(STRING())
+ .fromCase(VARIANT(), Variant.newBuilder().of(true), fromString("TRUE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(false), fromString("FALSE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of("foo"), fromString("foo"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(42), fromString("42"))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new BigDecimal("123.456")),
+ fromString("123.456"))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(LocalDate.parse("2021-09-24")),
+ DATE_STRING)
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(TIMESTAMP.toLocalDateTime()),
+ TIMESTAMP_STRING)
+ .fromCase(
+ VARIANT(),
+ CET_CONTEXT,
+ Variant.newBuilder().of(TIMESTAMP.toInstant()),
+ TIMESTAMP_STRING_CET)
+ // a binary value has no scalar rendering
Review Comment:
A binary value now decodes as UTF-8, the same as a regular BINARY to string cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3702936897
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
still my comment is valid:
> better to let toStringValue return StringData. we could call fromUtf8Bytes if variant can't give us good guarantees
This is in sync with `FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities``
We can offer VARIANT STRING to BYTES as a non-strict accessor for which users can call `MAKE_VALID_UTF8` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3702363160
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
Confirmed a bug, and it looks like a port regression. `BinaryVariantUtil` comes from Spark's `VariantUtil`, where the line reads:
```java
return new String(value, start, length, StandardCharsets.UTF_8);
```
Flink's copy dropped the charset argument:
```java
return new String(value, start, length);
```
That constructor falls back to the JVM default charset, which is platform dependent. Writing is unaffected, since `BinaryVariantInternalBuilder` encodes with explicit UTF-8, so the two directions currently disagree and a non-ASCII variant string can decode incorrectly.
`getString` is used well beyond casting, so I would rather not widen this PR. Follow-up ticket will restore the charset and adopt your suggestion of returning `StringData` from `toStringValue`, which lets the string path use `fromUtf8Bytes` and skip the decode plus re-encode round trip. I will link it here.
Source https://github.com/apache/spark/blob/84122de296b13f146d24bca21e29301702af13ec/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java#L522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613589952
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
+ // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")
Review Comment:
yes, however it does not answer we expect `NULL` after that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613292046
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
+ // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")
Review Comment:
why do we do this this way?
Better to have same way as regular cast
Then there is `SqlConformance#checkedArithmetic` which we can make use to switch between different behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610225458
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
`NaN` and `Infinity` are not part of the JSON specification: https://www.rfc-editor.org/info/rfc8259/#section-6
>Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.
Jackson does not allow it by default and you need to switch a flag to allow it. To represent them, the user has to define them as string and cast them as float. The workaround is documented in the variant type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3621070903
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
@davidradl This is how the implementation of the `BinaryVariant` defines it.
```java
@Override
public Instant getInstant() throws VariantTypeException {
checkType(Type.TIMESTAMP_LTZ, getType());
return microsToInstant(BinaryVariantUtil.getLong(value, pos));
}
```
The implementation is done based on the [Encoding types defined in Parquet](https://parquet.apache.org/docs/file-format/types/variantencoding/). Flink implements the types `0` (NULL)-`16` (STRING). 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3631060099
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -67,18 +67,35 @@ public static DecimalData toDecimalExact(Number value, int precision, int scale)
}
/**
- * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
- * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+ * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+ * {@code JSON_STRING}); objects and arrays use their JSON representation.
Review Comment:
> objects and arrays use their JSON representation
Let's not do this. People can use `JSON_STRING` for this. Let stay conservative and throw an error. Maybe pointing towards `JSON_STRING`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643808454
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
Review Comment:
Addressed
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3690105826
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, targetType);
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, targetType);
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, targetType);
+ }
+ return decimal;
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ final String targetType =
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3604974563
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
why do we assume that
IIRC Jackson has a property allowing to read them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610239635
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
There is no `BigDecimal.valueOf(Number val)`. Another way of converting it to BigDecimal would be something like this
```java
private static BigDecimal convertToBigDecimal(Number number) {
if (number == null) {
return null;
}
if (number instanceof BigDecimal) {
return (BigDecimal) number;
}
if (number instanceof BigInteger) {
return new BigDecimal((BigInteger) number);
}
if (number instanceof Integer
|| number instanceof Long
|| number instanceof Short
|| number instanceof Byte) {
return BigDecimal.valueOf(number.longValue());
}
return BigDecimal.valueOf(number.doubleValue());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610239635
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
There is no `BigDecimal.valueOf(Number val)`. Another way of converting it to BigDecimal would be something like this
```java
private static BigDecimal convertToBigDecimal(Number number) {
if (number instanceof Integer
|| number instanceof Long
|| number instanceof Short
|| number instanceof Byte) {
return BigDecimal.valueOf(number.longValue());
}
return BigDecimal.valueOf(number.doubleValue());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3672165521
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
Review Comment:
I have applied the feedback and improved the runtime code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643809559
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
Review Comment:
Added
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3636549712
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/basics/WordCountSQLExample.java:
##########
@@ -34,15 +34,7 @@ public static void main(String[] args) throws Exception {
// execute a Flink SQL job and print the result locally
tableEnv.executeSql(
// define the aggregation
- "SELECT word, SUM(frequency) AS `count`\n"
- // read from an artificial fixed-size table with rows and columns
- + "FROM (\n"
- + " VALUES ('Hello', 1), ('Ciao', 1), ('Hello', 2)\n"
- + ")\n"
- // name the table and its columns
- + "AS WordTable(word, frequency)\n"
- // group for aggregation
- + "GROUP BY word")
+ "SELECT PARSE_JSON('Infinity')")
Review Comment:
undo
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
```suggestion
are stored with microsecond precision.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define CAST and TRY_CAST from VARIANT to scalar types [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3729302125
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.binary.StringUtf8Utils;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * <p>A stored binary value has to be well-formed UTF-8, since a character string cannot carry
+ * bytes that no character maps to. Invalid input is rejected rather than decoded into {@code
+ * U+FFFD}, which would silently substitute a character the value never held.
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ value = variant.get().toString();
+ break;
+ case DECIMAL:
+ // toPlainString rather than toString, so that a small scale is not rendered in
+ // scientific notation, matching a regular DECIMAL to string cast.
+ value = variant.getDecimal().toPlainString();
+ break;
+ case STRING:
+ value = variant.getString();
+ break;
+ case BYTES:
+ // SQL reads a binary value as UTF-8, the same as a regular BINARY to string cast.
+ final byte[] utf8 = variant.getBytes();
+ final int invalidAt =
+ StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0, utf8.length);
+ if (invalidAt >= 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast the VARIANT binary value to %s because it is not "
+ + "valid UTF-8; the first invalid byte is at index %d "
+ + "of %d. Cast to BYTES to inspect the raw value, or "
+ + "wrap that in MAKE_VALID_UTF8 to replace every "
+ + "invalid byte with the U+FFFD replacement character.",
+ characterTarget(targetLength, charTarget),
+ invalidAt,
+ utf8.length));
+ }
+ value = new String(utf8, StandardCharsets.UTF_8);
+ break;
+ case DATE:
+ value = DateTimeUtils.formatDate((int) variant.getDate().toEpochDay());
+ break;
+ case TIMESTAMP:
+ // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A
+ // variant keeps microseconds, so the precision is always 6.
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromLocalDateTime(variant.getDateTime()),
+ DateTimeUtils.UTC_ZONE,
+ TIMESTAMP_PRECISION);
+ break;
+ case TIMESTAMP_LTZ:
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromInstant(variant.getInstant()),
+ sessionZone,
+ TIMESTAMP_PRECISION);
+ break;
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ characterTarget(targetLength, charTarget)));
+ default:
+ // An object or array has no scalar rendering.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ }
+ final boolean fits =
+ charTarget ? value.length() == targetLength : value.length() <= targetLength;
Review Comment:
Fixed now we have:
```
SELECT CAST('𝕏' AS CHAR(1)), TRY_CAST(PARSE_JSON('"𝕏"') AS CHAR(1), TRY_CAST(PARSE_JSON('"𝕏𝕏𝕏"') AS CHAR(2));
```
returns
```
+--------------------------------+--------------------------------+--------------------------------+
| EXPR$0 | EXPR$1 | EXPR$2 |
+--------------------------------+--------------------------------+--------------------------------+
| 𝕏 | 𝕏 | 𝕏𝕏 |
+--------------------------------+--------------------------------+--------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28758:
URL: https://github.com/apache/flink/pull/28758#issuecomment-4990589186
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aa795acb4c173c07293667278817db569c6ee300",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa795acb4c173c07293667278817db569c6ee300",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa795acb4c173c07293667278817db569c6ee300 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628412665
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java:
##########
@@ -45,6 +51,6 @@ public String generateStringExpression(
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
- return methodCall(inputTerm, "toString");
+ return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);
Review Comment:
when the target is bounded `CHAR(n)/VARCHAR(n)`, `CharVarCharTrimPadCastRule` kicks in, calls the `variant→string` rule, and trims/pads — the identical path every other to-string cast uses.
I added tests to prove it, and they pass:
- CAST(PARSE_JSON('"foobar"') AS VARCHAR(3)) → foo (trimmed)
- CAST(PARSE_JSON('"ab"') AS CHAR(5)) → ab (padded)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3621085558
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
Review Comment:
@davidradl I think the implementation of `DATE` is also based on Parquet: https://parquet.apache.org/docs/file-format/types/logicaltypes/
> DATE is used for a logical date type, without a time of day. It must annotate an int32 that stores the number of days from the Unix epoch, 1 January 1970.
```java
@Override
public LocalDate getDate() throws VariantTypeException {
checkType(Type.DATE, getType());
return LocalDate.ofEpochDay(BinaryVariantUtil.getLong(value, pos));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613433686
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
+ // CAST and returns NULL for TRY_CAST, instead of wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()), "overflowed")
Review Comment:
The idea here is that the user first casts (converts) the value from variant to the actual type it has and then performs an explicit cast if they want too. In this case the 128 is an `INT` so the user first needs to cast it `INT` and then if they want to explicitly cast to `TINYINT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3615556979
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
Review Comment:
I am curious why Date does not go to Flink date which would seem more natural
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628468877
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the target range fails
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3621070903
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
@davidradl This is how the implementation of the `BinaryVariant` defines it.
```
@Override
public Instant getInstant() throws VariantTypeException {
checkType(Type.TIMESTAMP_LTZ, getType());
return microsToInstant(BinaryVariantUtil.getLong(value, pos));
}
```
The implementation is done based on the [Encoding types defined in Parquet](https://parquet.apache.org/docs/file-format/types/variantencoding/). Flink implements the types `0` (NULL)-`16` (STRING). 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628361042
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
Review Comment:
Removed
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
Review Comment:
addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610239635
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
There is no `BigDecimal.valueOf(Number val)`. Another way of converting it to BigDecimal would be something like this
```java
public static BigDecimal convertToBigDecimal(Number number) {
if (number instanceof Integer
|| number instanceof Long
|| number instanceof Short
|| number instanceof Byte) {
return BigDecimal.valueOf(number.longValue());
}
return BigDecimal.valueOf(number.doubleValue());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3636524801
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
+binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
+(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
+padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
+otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
Review Comment:
people should know how they work already
```suggestion
padding or truncation.
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
+binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
+(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
+padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
+otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows. To narrow a
+`VARIANT`, first cast it to a type that fits the stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after overflow)
+```
+{{< /hint >}}
Review Comment:
I would vote for dropping this paragraph entirely. Or make it less prominent. A `hint info` is definitely too much. A sentence like `Numeric overflows are prohibited.` should be enough after: `targets accept any wider numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. Numeric overflows are prohibited.`
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
Review Comment:
```suggestion
targets accept any wider numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
```
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+ * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+ * {@code JSON_STRING}); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException}.
+ *
+ * <p>The target length is enforced strictly, with no padding or truncation: a {@code VARCHAR}
+ * value must not be longer than {@code targetLength}, and a {@code CHAR} value must match it
+ * exactly. A value that does not fit raises {@link TableRuntimeException}.
Review Comment:
nit: Too much comment. Code is shorter than this JavaDoc.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE} keep lenient IEEE conversion (overflow
+ * becomes infinity). A non-numeric variant raises {@link ClassCastException}, which fails
+ * {@code CAST} and yields {@code null} for {@code TRY_CAST}.
*/
private static String numericExpression(String inputTerm, LogicalType target) {
final String number = cast(className(Number.class), methodCall(inputTerm, "get"));
- if (!target.is(LogicalTypeRoot.DECIMAL)) {
- return methodCall(number, numberAccessor(target));
- }
- final DecimalType decimalType = (DecimalType) target;
- return staticCall(
- DecimalData.class,
- "fromBigDecimal",
- constructorCall(BigDecimal.class, methodCall(number, "toString")),
- decimalType.getPrecision(),
- decimalType.getScale());
- }
-
- private static String numberAccessor(LogicalType target) {
switch (target.getTypeRoot()) {
case TINYINT:
- return "byteValue";
+ return staticCall(VariantCastUtils.class, "toByteExact", number);
case SMALLINT:
- return "shortValue";
+ return staticCall(VariantCastUtils.class, "toShortExact", number);
case INTEGER:
- return "intValue";
+ return staticCall(VariantCastUtils.class, "toIntExact", number);
case BIGINT:
- return "longValue";
+ return staticCall(VariantCastUtils.class, "toLongExact", number);
case FLOAT:
- return "floatValue";
+ return methodCall(number, "floatValue");
case DOUBLE:
- return "doubleValue";
+ return methodCall(number, "doubleValue");
Review Comment:
Are there any data loss checks in this method? Asking AI leads to exactness loss.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
Review Comment:
Where is max precision of 38 checked?
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE} keep lenient IEEE conversion (overflow
+ * becomes infinity). A non-numeric variant raises {@link ClassCastException}, which fails
+ * {@code CAST} and yields {@code null} for {@code TRY_CAST}.
*/
private static String numericExpression(String inputTerm, LogicalType target) {
final String number = cast(className(Number.class), methodCall(inputTerm, "get"));
- if (!target.is(LogicalTypeRoot.DECIMAL)) {
- return methodCall(number, numberAccessor(target));
- }
- final DecimalType decimalType = (DecimalType) target;
- return staticCall(
- DecimalData.class,
- "fromBigDecimal",
- constructorCall(BigDecimal.class, methodCall(number, "toString")),
- decimalType.getPrecision(),
- decimalType.getScale());
- }
-
- private static String numberAccessor(LogicalType target) {
switch (target.getTypeRoot()) {
case TINYINT:
- return "byteValue";
+ return staticCall(VariantCastUtils.class, "toByteExact", number);
case SMALLINT:
- return "shortValue";
+ return staticCall(VariantCastUtils.class, "toShortExact", number);
case INTEGER:
- return "intValue";
+ return staticCall(VariantCastUtils.class, "toIntExact", number);
case BIGINT:
- return "longValue";
+ return staticCall(VariantCastUtils.class, "toLongExact", number);
case FLOAT:
- return "floatValue";
+ return methodCall(number, "floatValue");
Review Comment:
Are there any data loss checks in this method? Asking AI leads to exactness loss.
> can I store a full long in a float?
>
> The short answer is yes, in terms of size (range), but no, in terms of exactness (precision).
>
> If you attempt to store a large long inside a float, the application will not crash, but you will likely lose data through rounding errors.
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
Review Comment:
Let's not go into the details of CHAR/VARCHAR at this line
```suggestion
handled the same way. Casting a `VARIANT` to `STRING` extracts the scalar value, so a
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643805919
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/basics/WordCountSQLExample.java:
##########
@@ -34,15 +34,7 @@ public static void main(String[] args) throws Exception {
// execute a Flink SQL job and print the result locally
tableEnv.executeSql(
// define the aggregation
- "SELECT word, SUM(frequency) AS `count`\n"
- // read from an artificial fixed-size table with rows and columns
- + "FROM (\n"
- + " VALUES ('Hello', 1), ('Ciao', 1), ('Hello', 2)\n"
- + ")\n"
- // name the table and its columns
- + "AS WordTable(word, frequency)\n"
- // group for aggregation
- + "GROUP BY word")
+ "SELECT PARSE_JSON('Infinity')")
Review Comment:
ups. Reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3684320053
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -1544,35 +1544,55 @@ Stream<CastTestSpecBuilder> testCases() {
.fromCase(BITMAP(), DEFAULT_BITMAP, DEFAULT_BITMAP.toBytes())
.fromCase(BITMAP(), Bitmap.empty(), Bitmap.empty().toBytes())
.fromCase(BITMAP(), null, null),
- // From VARIANT to primitive types. Numeric targets are lenient: a variant holding
- // any numeric kind converts to the requested numeric type (widening and narrowing).
- // Non-numeric targets are strict: the stored kind must match, otherwise the cast
- // fails and TRY_CAST returns null.
+ // From VARIANT to primitive types. A cast succeeds only when the target holds the
+ // stored value unaltered: an integer widens or narrows while it stays in range, a
+ // DECIMAL has to fit the precision and scale, and a timestamp the precision. FLOAT
+ // and DOUBLE are approximate, so they take any numeric kind and reject only a
+ // magnitude out of range. Reading one kind as another is never implicit.
CastTestSpecBuilder.testCastTo(BOOLEAN())
.fromCase(VARIANT(), Variant.newBuilder().of(true), true)
.fromCase(VARIANT(), Variant.newBuilder().of(false), false)
.fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(TINYINT())
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (byte) 42)
+ // a wider integer kind narrows while the value is in range
.fromCase(VARIANT(), Variant.newBuilder().of(42), (byte) 42)
+ // out of range is rejected instead of wrapping
+ .fail(VARIANT(), Variant.newBuilder().of(1000), TableRuntimeException.class)
.fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(SMALLINT())
.fromCase(VARIANT(), Variant.newBuilder().of((short) 42), (short) 42)
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (short) 42)
+ .fromCase(VARIANT(), Variant.newBuilder().of(1000), (short) 1000)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(40000),
+ TableRuntimeException.class)
.fail(
VARIANT(),
Variant.newBuilder().of(true),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(INT())
- // widening: a JSON integer is stored in the smallest type but still casts
- // up
+ .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
+ // every integer kind converts as long as the value fits
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of((short) 42), 42)
- .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of(42L), 42)
- // narrowing from a floating point or decimal value truncates
- .fromCase(VARIANT(), Variant.newBuilder().of(3.9d), 3)
- .fromCase(VARIANT(), Variant.newBuilder().of(new BigDecimal("7.2")), 7)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(2147483648L),
+ TableRuntimeException.class)
+ // an approximate or decimal kind is not read as an integer, whether or not
+ // the value happens to be integral
+ .fail(VARIANT(), Variant.newBuilder().of(7.0d), TableRuntimeException.class)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(new BigDecimal("7.0")),
+ TableRuntimeException.class)
Review Comment:
This one we could rediscuss, right?
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -138,75 +138,203 @@ Stream<TestSetSpec> getTestSetSpecs() {
}
private static List<TestSetSpec> variantCasts() {
- // A variant is produced with PARSE_JSON so that the source column stays a STRING literal;
- // there is no VARIANT literal to feed a source column directly. A JSON integer is stored in
- // the smallest integer type that fits (42 -> TINYINT), and numeric casts are lenient, so it
- // still widens to INT.
+ // A variant is produced with PARSE_JSON since there is no VARIANT literal. Numeric casts
+ // succeed only when the value is preserved exactly, otherwise CAST fails and TRY_CAST
+ // returns NULL.
return List.of(
TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to a primitive")
.onFieldsWithData("unused")
.andDataTypes(STRING())
+ // An integer converts to any integer target while the value stays in
+ // range, and to FLOAT or DOUBLE which are approximate by definition.
+ .testResult(
+ call("PARSE_JSON", "42").cast(TINYINT()),
+ "CAST(PARSE_JSON('42') AS TINYINT)",
+ (byte) 42,
+ TINYINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(SMALLINT()),
+ "CAST(PARSE_JSON('42') AS SMALLINT)",
+ (short) 42,
+ SMALLINT().notNull())
.testResult(
call("PARSE_JSON", "42").cast(INT()),
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
+ call("PARSE_JSON", "42").cast(BIGINT()),
+ "CAST(PARSE_JSON('42') AS BIGINT)",
+ 42L,
+ BIGINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(FLOAT()),
+ "CAST(PARSE_JSON('42') AS FLOAT)",
+ 42.0f,
+ FLOAT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(DOUBLE()),
+ "CAST(PARSE_JSON('42') AS DOUBLE)",
+ 42.0d,
+ DOUBLE().notNull())
+ // An out-of-range value is rejected rather than wrapped.
+ .testResult(
+ call("PARSE_JSON", "1000").cast(SMALLINT()),
+ "CAST(PARSE_JSON('1000') AS SMALLINT)",
+ (short) 1000,
SMALLINT().notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1000").cast(TINYINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS TINYINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "1000").tryCast(TINYINT()),
+ "TRY_CAST(PARSE_JSON('1000') AS TINYINT)",
+ null,
+ TINYINT())
+ // A decimal is not read as an integer, since that would drop digits.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(INT()), "Cannot cast a VARIANT")
.testResult(
- call("PARSE_JSON", "2147483648").cast(INT()),
- "CAST(PARSE_JSON('2147483648') AS INT)",
- -2147483648,
- INT().notNull())
+ call("PARSE_JSON", "123.456").tryCast(INT()),
+ "TRY_CAST(PARSE_JSON('123.456') AS INT)",
+ null,
+ INT())
+ // A DECIMAL target has to hold the value exactly.
.testResult(
- call("PARSE_JSON", "9223372036854775808").cast(BIGINT()),
- "CAST(PARSE_JSON('9223372036854775808') AS BIGINT)",
- -9223372036854775808L,
- BIGINT().notNull())
- // An out-of-range floating point value saturates when cast to an integer,
- // and a fractional value is truncated toward zero.
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6, 3)),
+ "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))",
+ new BigDecimal("123.456"),
+ DECIMAL(6, 3).notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6, 2)), "lose precision")
.testResult(
- call("PARSE_JSON", "1e20").cast(INT()),
- "CAST(PARSE_JSON('1e20') AS INT)",
- 2147483647,
- INT().notNull())
+ call("PARSE_JSON", "123.456").tryCast(DECIMAL(6, 2)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6, 2))",
+ null,
+ DECIMAL(6, 2))
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(5, 3)), "overflowed")
.testResult(
- call("PARSE_JSON", "3.9").cast(INT()),
- "CAST(PARSE_JSON('3.9') AS INT)",
- 3,
- INT().notNull())
- // A value beyond the FLOAT range becomes infinity.
+ call("PARSE_JSON", "123.456").tryCast(DECIMAL(5, 3)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5, 3))",
+ null,
+ DECIMAL(5, 3))
+ // An integer is exact, so it reaches a DECIMAL that has room for it.
+ .testResult(
+ call("PARSE_JSON", "42").cast(DECIMAL(5, 2)),
+ "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))",
+ new BigDecimal("42.00"),
+ DECIMAL(5, 2).notNull())
+ // A decimal reaches an approximate target, where losing digits is expected.
.testResult(
- call("PARSE_JSON", "1e40").cast(FLOAT()),
- "CAST(PARSE_JSON('1e40') AS FLOAT)",
- Float.POSITIVE_INFINITY,
+ call("PARSE_JSON", "123.456").cast(FLOAT()),
+ "CAST(PARSE_JSON('123.456') AS FLOAT)",
+ 123.456f,
FLOAT().notNull())
- // DECIMAL overflow yields NULL instead of wrapping; TRY_CAST surfaces it.
.testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(4, 2)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(4, 2))",
+ call("PARSE_JSON", "123.456").cast(DOUBLE()),
+ "CAST(PARSE_JSON('123.456') AS DOUBLE)",
+ 123.456d,
+ DOUBLE().notNull())
+ // A magnitude the target cannot represent is still rejected.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1e40").cast(FLOAT()), "overflowed")
+ .testResult(
+ call("PARSE_JSON", "1e40").tryCast(FLOAT()),
+ "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)",
null,
- DECIMAL(4, 2))
+ FLOAT())
.testResult(
- call("PARSE_JSON", "42").cast(BIGINT()),
- "CAST(PARSE_JSON('42') AS BIGINT)",
- 42L,
- BIGINT().notNull())
+ call("PARSE_JSON", "1e20").cast(DOUBLE()),
+ "CAST(PARSE_JSON('1e20') AS DOUBLE)",
+ 1e20,
+ DOUBLE().notNull())
.testResult(
call("PARSE_JSON", "true").cast(BOOLEAN()),
"CAST(PARSE_JSON('true') AS BOOLEAN)",
true,
BOOLEAN().notNull())
+ // CAST returns the raw scalar value (string unquoted)
+ .testResult(
+ call("PARSE_JSON", "\"foo\"").cast(STRING()),
+ "CAST(PARSE_JSON('\"foo\"') AS STRING)",
+ "foo",
+ STRING().notNull())
+ .testResult(
+ call("PARSE_JSON", "123.456").cast(STRING()),
+ "CAST(PARSE_JSON('123.456') AS STRING)",
+ "123.456",
+ STRING().notNull())
+ .testResult(
+ call("PARSE_JSON", "true").cast(STRING()),
+ "CAST(PARSE_JSON('true') AS STRING)",
+ "true",
+ STRING().notNull())
Review Comment:
This we did not discuss. Let's not allow this for now. Only string can be cast to string.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
Review Comment:
only format on error, keep the hot path performant
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, targetType);
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, targetType);
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, targetType);
+ }
+ return decimal;
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ final String targetType =
+ String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR", targetLength);
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ targetType));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = variant.get().toString();
Review Comment:
this is again JSON and not SQL. e.g. `TRUE` vs `true` for variant boolean to string. as mentioned above. only string variant can cast to string. if you want to support more, come up with a proper string representation for each variant type. incl TS_LTZ in users time zone.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, targetType);
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, targetType);
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, targetType);
+ }
+ return decimal;
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ final String targetType =
Review Comment:
same comment as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define CAST and TRY_CAST from VARIANT to scalar types [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3729302125
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.binary.StringUtf8Utils;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * <p>A stored binary value has to be well-formed UTF-8, since a character string cannot carry
+ * bytes that no character maps to. Invalid input is rejected rather than decoded into {@code
+ * U+FFFD}, which would silently substitute a character the value never held.
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ value = variant.get().toString();
+ break;
+ case DECIMAL:
+ // toPlainString rather than toString, so that a small scale is not rendered in
+ // scientific notation, matching a regular DECIMAL to string cast.
+ value = variant.getDecimal().toPlainString();
+ break;
+ case STRING:
+ value = variant.getString();
+ break;
+ case BYTES:
+ // SQL reads a binary value as UTF-8, the same as a regular BINARY to string cast.
+ final byte[] utf8 = variant.getBytes();
+ final int invalidAt =
+ StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0, utf8.length);
+ if (invalidAt >= 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast the VARIANT binary value to %s because it is not "
+ + "valid UTF-8; the first invalid byte is at index %d "
+ + "of %d. Cast to BYTES to inspect the raw value, or "
+ + "wrap that in MAKE_VALID_UTF8 to replace every "
+ + "invalid byte with the U+FFFD replacement character.",
+ characterTarget(targetLength, charTarget),
+ invalidAt,
+ utf8.length));
+ }
+ value = new String(utf8, StandardCharsets.UTF_8);
+ break;
+ case DATE:
+ value = DateTimeUtils.formatDate((int) variant.getDate().toEpochDay());
+ break;
+ case TIMESTAMP:
+ // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A
+ // variant keeps microseconds, so the precision is always 6.
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromLocalDateTime(variant.getDateTime()),
+ DateTimeUtils.UTC_ZONE,
+ TIMESTAMP_PRECISION);
+ break;
+ case TIMESTAMP_LTZ:
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromInstant(variant.getInstant()),
+ sessionZone,
+ TIMESTAMP_PRECISION);
+ break;
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ characterTarget(targetLength, charTarget)));
+ default:
+ // An object or array has no scalar rendering.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ }
+ final boolean fits =
+ charTarget ? value.length() == targetLength : value.length() <= targetLength;
Review Comment:
Fixed now we have:
```
SELECT CAST('𝕏' AS CHAR(1)),
TRY_CAST(PARSE_JSON('"𝕏"') AS CHAR(1),
TRY_CAST(PARSE_JSON('"𝕏𝕏𝕏"') AS CHAR(2));
```
returns
```
+--------------------------------+--------------------------------+--------------------------------+
| EXPR$0 | EXPR$1 | EXPR$2 |
+--------------------------------+--------------------------------+--------------------------------+
| 𝕏 | 𝕏 | 𝕏𝕏 |
+--------------------------------+--------------------------------+--------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3666552666
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
Review Comment:
Thanks for the catch. Changed and added test for null casts to STRING, CHAR, and VARCHAR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3690104002
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -138,75 +138,203 @@ Stream<TestSetSpec> getTestSetSpecs() {
}
private static List<TestSetSpec> variantCasts() {
- // A variant is produced with PARSE_JSON so that the source column stays a STRING literal;
- // there is no VARIANT literal to feed a source column directly. A JSON integer is stored in
- // the smallest integer type that fits (42 -> TINYINT), and numeric casts are lenient, so it
- // still widens to INT.
+ // A variant is produced with PARSE_JSON since there is no VARIANT literal. Numeric casts
+ // succeed only when the value is preserved exactly, otherwise CAST fails and TRY_CAST
+ // returns NULL.
return List.of(
TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to a primitive")
.onFieldsWithData("unused")
.andDataTypes(STRING())
+ // An integer converts to any integer target while the value stays in
+ // range, and to FLOAT or DOUBLE which are approximate by definition.
+ .testResult(
+ call("PARSE_JSON", "42").cast(TINYINT()),
+ "CAST(PARSE_JSON('42') AS TINYINT)",
+ (byte) 42,
+ TINYINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(SMALLINT()),
+ "CAST(PARSE_JSON('42') AS SMALLINT)",
+ (short) 42,
+ SMALLINT().notNull())
.testResult(
call("PARSE_JSON", "42").cast(INT()),
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing), like a regular numeric
- // cast.
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
+ call("PARSE_JSON", "42").cast(BIGINT()),
+ "CAST(PARSE_JSON('42') AS BIGINT)",
+ 42L,
+ BIGINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(FLOAT()),
+ "CAST(PARSE_JSON('42') AS FLOAT)",
+ 42.0f,
+ FLOAT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(DOUBLE()),
+ "CAST(PARSE_JSON('42') AS DOUBLE)",
+ 42.0d,
+ DOUBLE().notNull())
+ // An out-of-range value is rejected rather than wrapped.
+ .testResult(
+ call("PARSE_JSON", "1000").cast(SMALLINT()),
+ "CAST(PARSE_JSON('1000') AS SMALLINT)",
+ (short) 1000,
SMALLINT().notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1000").cast(TINYINT()), "overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS TINYINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "1000").tryCast(TINYINT()),
+ "TRY_CAST(PARSE_JSON('1000') AS TINYINT)",
+ null,
+ TINYINT())
+ // A decimal is not read as an integer, since that would drop digits.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(INT()), "Cannot cast a VARIANT")
.testResult(
- call("PARSE_JSON", "2147483648").cast(INT()),
- "CAST(PARSE_JSON('2147483648') AS INT)",
- -2147483648,
- INT().notNull())
+ call("PARSE_JSON", "123.456").tryCast(INT()),
+ "TRY_CAST(PARSE_JSON('123.456') AS INT)",
+ null,
+ INT())
+ // A DECIMAL target has to hold the value exactly.
.testResult(
- call("PARSE_JSON", "9223372036854775808").cast(BIGINT()),
- "CAST(PARSE_JSON('9223372036854775808') AS BIGINT)",
- -9223372036854775808L,
- BIGINT().notNull())
- // An out-of-range floating point value saturates when cast to an integer,
- // and a fractional value is truncated toward zero.
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6, 3)),
+ "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))",
+ new BigDecimal("123.456"),
+ DECIMAL(6, 3).notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6, 2)), "lose precision")
.testResult(
- call("PARSE_JSON", "1e20").cast(INT()),
- "CAST(PARSE_JSON('1e20') AS INT)",
- 2147483647,
- INT().notNull())
+ call("PARSE_JSON", "123.456").tryCast(DECIMAL(6, 2)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6, 2))",
+ null,
+ DECIMAL(6, 2))
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(5, 3)), "overflowed")
.testResult(
- call("PARSE_JSON", "3.9").cast(INT()),
- "CAST(PARSE_JSON('3.9') AS INT)",
- 3,
- INT().notNull())
- // A value beyond the FLOAT range becomes infinity.
+ call("PARSE_JSON", "123.456").tryCast(DECIMAL(5, 3)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5, 3))",
+ null,
+ DECIMAL(5, 3))
+ // An integer is exact, so it reaches a DECIMAL that has room for it.
+ .testResult(
+ call("PARSE_JSON", "42").cast(DECIMAL(5, 2)),
+ "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))",
+ new BigDecimal("42.00"),
+ DECIMAL(5, 2).notNull())
+ // A decimal reaches an approximate target, where losing digits is expected.
.testResult(
- call("PARSE_JSON", "1e40").cast(FLOAT()),
- "CAST(PARSE_JSON('1e40') AS FLOAT)",
- Float.POSITIVE_INFINITY,
+ call("PARSE_JSON", "123.456").cast(FLOAT()),
+ "CAST(PARSE_JSON('123.456') AS FLOAT)",
+ 123.456f,
FLOAT().notNull())
- // DECIMAL overflow yields NULL instead of wrapping; TRY_CAST surfaces it.
.testResult(
- call("PARSE_JSON", "123.456").tryCast(DECIMAL(4, 2)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(4, 2))",
+ call("PARSE_JSON", "123.456").cast(DOUBLE()),
+ "CAST(PARSE_JSON('123.456') AS DOUBLE)",
+ 123.456d,
+ DOUBLE().notNull())
+ // A magnitude the target cannot represent is still rejected.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1e40").cast(FLOAT()), "overflowed")
+ .testResult(
+ call("PARSE_JSON", "1e40").tryCast(FLOAT()),
+ "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)",
null,
- DECIMAL(4, 2))
+ FLOAT())
.testResult(
- call("PARSE_JSON", "42").cast(BIGINT()),
- "CAST(PARSE_JSON('42') AS BIGINT)",
- 42L,
- BIGINT().notNull())
+ call("PARSE_JSON", "1e20").cast(DOUBLE()),
+ "CAST(PARSE_JSON('1e20') AS DOUBLE)",
+ 1e20,
+ DOUBLE().notNull())
.testResult(
call("PARSE_JSON", "true").cast(BOOLEAN()),
"CAST(PARSE_JSON('true') AS BOOLEAN)",
true,
BOOLEAN().notNull())
+ // CAST returns the raw scalar value (string unquoted)
+ .testResult(
+ call("PARSE_JSON", "\"foo\"").cast(STRING()),
+ "CAST(PARSE_JSON('\"foo\"') AS STRING)",
+ "foo",
+ STRING().notNull())
+ .testResult(
+ call("PARSE_JSON", "123.456").cast(STRING()),
+ "CAST(PARSE_JSON('123.456') AS STRING)",
+ "123.456",
+ STRING().notNull())
+ .testResult(
+ call("PARSE_JSON", "true").cast(STRING()),
+ "CAST(PARSE_JSON('true') AS STRING)",
+ "true",
+ STRING().notNull())
Review Comment:
Now the cast to string produces SQL string.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. {@code FLOAT} and {@code DOUBLE} are
+ * the exception: they are approximate by definition, so they accept any numeric kind and reject
+ * only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads an integer variant as a {@code long} and checks it against the target range. Only the
+ * integer kinds are accepted, so an approximate or decimal value is rejected rather than
+ * rounded.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ final long value = ((Number) variant.get()).longValue();
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final String targetType = String.format("DECIMAL(%d, %d)", precision, scale);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3690102532
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -1544,35 +1544,55 @@ Stream<CastTestSpecBuilder> testCases() {
.fromCase(BITMAP(), DEFAULT_BITMAP, DEFAULT_BITMAP.toBytes())
.fromCase(BITMAP(), Bitmap.empty(), Bitmap.empty().toBytes())
.fromCase(BITMAP(), null, null),
- // From VARIANT to primitive types. Numeric targets are lenient: a variant holding
- // any numeric kind converts to the requested numeric type (widening and narrowing).
- // Non-numeric targets are strict: the stored kind must match, otherwise the cast
- // fails and TRY_CAST returns null.
+ // From VARIANT to primitive types. A cast succeeds only when the target holds the
+ // stored value unaltered: an integer widens or narrows while it stays in range, a
+ // DECIMAL has to fit the precision and scale, and a timestamp the precision. FLOAT
+ // and DOUBLE are approximate, so they take any numeric kind and reject only a
+ // magnitude out of range. Reading one kind as another is never implicit.
CastTestSpecBuilder.testCastTo(BOOLEAN())
.fromCase(VARIANT(), Variant.newBuilder().of(true), true)
.fromCase(VARIANT(), Variant.newBuilder().of(false), false)
.fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(TINYINT())
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (byte) 42)
+ // a wider integer kind narrows while the value is in range
.fromCase(VARIANT(), Variant.newBuilder().of(42), (byte) 42)
+ // out of range is rejected instead of wrapping
+ .fail(VARIANT(), Variant.newBuilder().of(1000), TableRuntimeException.class)
.fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(SMALLINT())
.fromCase(VARIANT(), Variant.newBuilder().of((short) 42), (short) 42)
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (short) 42)
+ .fromCase(VARIANT(), Variant.newBuilder().of(1000), (short) 1000)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(40000),
+ TableRuntimeException.class)
.fail(
VARIANT(),
Variant.newBuilder().of(true),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(INT())
- // widening: a JSON integer is stored in the smallest type but still casts
- // up
+ .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
+ // every integer kind converts as long as the value fits
.fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of((short) 42), 42)
- .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of(42L), 42)
- // narrowing from a floating point or decimal value truncates
- .fromCase(VARIANT(), Variant.newBuilder().of(3.9d), 3)
- .fromCase(VARIANT(), Variant.newBuilder().of(new BigDecimal("7.2")), 7)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(2147483648L),
+ TableRuntimeException.class)
+ // an approximate or decimal kind is not read as an integer, whether or not
+ // the value happens to be integral
+ .fail(VARIANT(), Variant.newBuilder().of(7.0d), TableRuntimeException.class)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(new BigDecimal("7.0")),
+ TableRuntimeException.class)
Review Comment:
Now casts with trailing zeros to integer types are allowed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3690628523
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java:
##########
@@ -1544,35 +1544,79 @@ Stream<CastTestSpecBuilder> testCases() {
.fromCase(BITMAP(), DEFAULT_BITMAP, DEFAULT_BITMAP.toBytes())
.fromCase(BITMAP(), Bitmap.empty(), Bitmap.empty().toBytes())
.fromCase(BITMAP(), null, null),
- // From VARIANT to primitive types. Numeric targets are lenient: a variant holding
- // any numeric kind converts to the requested numeric type (widening and narrowing).
- // Non-numeric targets are strict: the stored kind must match, otherwise the cast
- // fails and TRY_CAST returns null.
+ // From VARIANT to primitive types. A cast succeeds only when the target holds the
+ // stored value unaltered, except for the approximate FLOAT and DOUBLE.
+ // A character string renders like a regular cast of the stored kind, so these
+ // expectations reuse the constants of the native cases above.
+ CastTestSpecBuilder.testCastTo(STRING())
+ .fromCase(VARIANT(), Variant.newBuilder().of(true), fromString("TRUE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(false), fromString("FALSE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of("foo"), fromString("foo"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(42), fromString("42"))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new BigDecimal("123.456")),
+ fromString("123.456"))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(LocalDate.parse("2021-09-24")),
+ DATE_STRING)
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(TIMESTAMP.toLocalDateTime()),
+ TIMESTAMP_STRING)
+ .fromCase(
+ VARIANT(),
+ CET_CONTEXT,
+ Variant.newBuilder().of(TIMESTAMP.toInstant()),
+ TIMESTAMP_STRING_CET)
+ // a binary value has no scalar rendering
Review Comment:
according to SQL rules we assume UTF-8 as a string
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
Review Comment:
if you use the toString of BigDecimal, double check that leading/trailing zeros have the same as with regular DECIMAL AS STRING casts
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
better to let toStringValue return StringData. we could call fromUtf8Bytes if variant can't give us good guarantees
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
+ break;
+ case STRING:
+ value = variant.getString();
Review Comment:
Is it guaranteed that this is always a UTF8 string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628471218
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
Review Comment:
Changed it to `TableRuntimeException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3621070903
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
@davidradl This is how the implementation of the `BinaryVariant` defines it.
```java
@Override
public Instant getInstant() throws VariantTypeException {
checkType(Type.TIMESTAMP_LTZ, getType());
return microsToInstant(BinaryVariantUtil.getLong(value, pos));
}
```
The implementation is done based on the [Encoding types defined in Parquet](https://parquet.apache.org/docs/file-format/types/variantencoding/). Flink implements the types from `0` (NULL) to `16` (STRING). 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643801715
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
Review Comment:
Max precision 38 is guaranteed by the target `DecimalType` itself (`DecimalType.MAX_PRECISION`), so precision here is always ≤ 38. `DecimalData.fromBigDecimal` returns `null` when the value doesn't fit the given precision/scale, and we throw on that. So there's no separate check to add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656080661
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE} keep lenient IEEE conversion (overflow
+ * becomes infinity). A non-numeric variant raises {@link ClassCastException}, which fails
+ * {@code CAST} and yields {@code null} for {@code TRY_CAST}.
*/
private static String numericExpression(String inputTerm, LogicalType target) {
final String number = cast(className(Number.class), methodCall(inputTerm, "get"));
- if (!target.is(LogicalTypeRoot.DECIMAL)) {
- return methodCall(number, numberAccessor(target));
- }
- final DecimalType decimalType = (DecimalType) target;
- return staticCall(
- DecimalData.class,
- "fromBigDecimal",
- constructorCall(BigDecimal.class, methodCall(number, "toString")),
- decimalType.getPrecision(),
- decimalType.getScale());
- }
-
- private static String numberAccessor(LogicalType target) {
switch (target.getTypeRoot()) {
case TINYINT:
- return "byteValue";
+ return staticCall(VariantCastUtils.class, "toByteExact", number);
case SMALLINT:
- return "shortValue";
+ return staticCall(VariantCastUtils.class, "toShortExact", number);
case INTEGER:
- return "intValue";
+ return staticCall(VariantCastUtils.class, "toIntExact", number);
case BIGINT:
- return "longValue";
+ return staticCall(VariantCastUtils.class, "toLongExact", number);
case FLOAT:
- return "floatValue";
+ return methodCall(number, "floatValue");
case DOUBLE:
- return "doubleValue";
+ return methodCall(number, "doubleValue");
Review Comment:
Please see: https://github.com/apache/flink/pull/28758#discussion_r3656077829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656225392
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
Review Comment:
this is runtime code
you always know the value of `min`/`max`
can we have already constructed/prepared `BigInteger`'s `min`/`max` and pass them here rather than construct it on every call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656188157
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
Review Comment:
if `variant.get() == null`
then `String.valueOf(...)` returns `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656184480
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
Review Comment:
looks like a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3672165521
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
Review Comment:
I have applied the feedback and improved the runtime code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3672802586
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
Review Comment:
>The toBigInteger() on the line above still allocates once per row. Let me know if you want that removed as well.
this is exactly the point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3666534836
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3604896086
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
why do we need to parse it every time?
can we used `valueOf(...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628460514
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
Review Comment:
For objects, arrays, bytes we rely on the `Variant#toJson()` implementation. All other types (i.e., String) we convert to literal String. This is because for a case like `CAST(PARSE_JSON('foo') AS STRING))` we want to return a string literal `foo` and not a JSON String `"foo"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610327513
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
then the question: why do we support these values in some parts of the code and do not support in others?
Do we have a plan to be consistent for all parts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3614482042
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
In Variant we don't and I don't think we should because it follows the JSON spec. Vendors like Snowflake and Databricks also don't allow it.
For cast we do `CAST('Nan' AS FLOAT)` and it is a valid case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613103046
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3621070903
##########
docs/content/docs/sql/reference/data-types.md:
##########
@@ -1515,19 +1515,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
Review Comment:
@davidradl This is how the implementation of the `BinaryVariant` defines it.
```java
@Override
public Instant getInstant() throws VariantTypeException {
checkType(Type.TIMESTAMP_LTZ, getType());
return microsToInstant(BinaryVariantUtil.getLong(value, pos));
}
```
The implementation is done based on the [Encoding types defined in Parquet](https://parquet.apache.org/docs/file-format/types/variantencoding/). Flink implements the types from `0` (NULL) to `16` (STRING). The rest of the types: 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643803169
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+ * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+ * {@code JSON_STRING}); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException}.
+ *
+ * <p>The target length is enforced strictly, with no padding or truncation: a {@code VARCHAR}
+ * value must not be longer than {@code targetLength}, and a {@code CHAR} value must match it
+ * exactly. A value that does not fit raises {@link TableRuntimeException}.
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : convertToBigDecimal(value);
+ }
+
+ private static BigDecimal convertToBigDecimal(Number number) {
+ if (number == null) {
Review Comment:
Removed. It was not necessary
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link TableRuntimeException} on overflow instead
+ * of wrapping. Casting to a character string extracts the scalar value (a string stays unquoted); a
+ * variant holding an object, array, or binary value is not castable and raises {@link
+ * TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value for a {@code CHAR(targetLength)} or {@code
+ * VARCHAR(targetLength)} target. Scalars use their raw value (a string stays unquoted, unlike
+ * {@code JSON_STRING}); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException}.
+ *
+ * <p>The target length is enforced strictly, with no padding or truncation: a {@code VARCHAR}
+ * value must not be longer than {@code targetLength}, and a {@code CHAR} value must match it
+ * exactly. A value that does not fit raises {@link TableRuntimeException}.
Review Comment:
Shortened it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3702665952
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ value = variant.get().toString();
Review Comment:
Good catch! `CAST(PARSE_JSON('0.0000000001') AS STRING)` returned `1E-10` where native returns `0.0000000001`. Now the behavior is aligned and added a test to cover it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643807000
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
+binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
+(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
+padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
+otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows. To narrow a
+`VARIANT`, first cast it to a type that fits the stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after overflow)
+```
+{{< /hint >}}
Review Comment:
Dropped
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
+binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
+(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
+padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
+otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3643815494
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java:
##########
@@ -45,6 +51,6 @@ public String generateStringExpression(
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
- return methodCall(inputTerm, "toString");
+ return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);
Review Comment:
Reverted the change. Now we follow this rule:
- VARCHAR(n): OK if length ≤ n, else throw.
- CHAR(n): OK only if length == n, else throw.
So `CAST (PARSE_JSON('ab') AS VARCHAR(3)) -> returns ab`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3656194396
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
Review Comment:
then there is a check for length
and in this case it will throw exception for types `CHAR(1)`, `CHAR(2)`, `CHAR(3)`
even if the value is `null`
or did I miss anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628364249
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
Review Comment:
Changed
##########
flink-core/src/main/java/org/apache/flink/types/variant/Variant.java:
##########
@@ -127,18 +127,19 @@ public interface Variant {
/**
* Get the scalar value of variant as LocalDateTime, if the variant type is {@link
- * Type#TIMESTAMP}.
+ * Type#TIMESTAMP}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#TIMESTAMP}.
*/
LocalDateTime getDateTime() throws VariantTypeException;
/**
- * Get the scalar value of variant as Instant, if the variant type is {@link Type#TIMESTAMP}.
+ * Get the scalar value of variant as Instant, if the variant type is {@link
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628362907
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the stored value, then apply a
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3628366412
##########
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java:
##########
@@ -284,11 +285,12 @@ private static Stream<Arguments> testData() {
new VarBinaryType(VarBinaryType.MAX_LENGTH),
false,
true),
+ Arguments.of(new VariantType(), new CharType(), false, true),
+ Arguments.of(new VariantType(), VarCharType.STRING_TYPE, false, true),
// variant identity cast is implicit
Arguments.of(new VariantType(), new VariantType(), true, true),
// TIME, character strings and constructed targets are not castable from variant
Review Comment:
Changed
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and yields {@code null} for
+ * {@code TRY_CAST}, following Spark's variant cast semantics; {@code FLOAT} and {@code DOUBLE}
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3672763250
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are strict: a value that cannot be represented exactly, whether by integer or
+ * decimal overflow or by floating-point precision loss, raises {@link TableRuntimeException}
+ * instead of wrapping or rounding. Casting to a character string extracts the scalar value (a
+ * string stays unquoted); a variant holding an object, array, or binary value is not castable and
+ * raises {@link TableRuntimeException} (use {@code JSON_STRING} for its JSON representation).
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static float toFloatExact(Number value) {
+ final float result = value.floatValue();
+ if (!Float.isFinite(result)) {
+ throw overflow(value, "FLOAT");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "FLOAT");
+ }
+ return result;
+ }
+
+ public static double toDoubleExact(Number value) {
+ final double result = value.doubleValue();
+ if (!Double.isFinite(result)) {
+ throw overflow(value, "DOUBLE");
+ }
+ if (exactValue(value).compareTo(new BigDecimal(result)) != 0) {
+ throw lossyCast(value, "DOUBLE");
+ }
+ return result;
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to its raw string value, enforcing {@code targetLength}
+ * strictly with no padding or truncation ({@code CHAR} requires an exact length, {@code
+ * VARCHAR} an upper bound).
+ */
+ public static String toStringValue(Variant variant, int targetLength, boolean charTarget) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ default:
+ // Scalars are cast to their raw string value.
+ }
+ final String value = String.valueOf(variant.get());
+ final int length = value.length();
+ final boolean fits = charTarget ? length == targetLength : length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT string value of length %d does not fit %s(%d); VARIANT "
+ + "string casts do not pad or truncate.",
+ length, charTarget ? "CHAR" : "VARCHAR", targetLength));
+ }
+ return value;
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw overflow(value, targetType);
Review Comment:
I went one step further and dropped the `BigInteger` comparison entirely:
```java
if (integral.bitLength() > 63) {
throw overflow(value, targetType.name());
}
final long result = integral.longValue();
if (result < min || result > max) {
throw overflow(value, targetType.name());
}
```
`bitLength()` does not allocate. Once it passes, the value provably fits a `long`, so the range check runs on the primitive `min`/`max` that were already being passed in. This avoids the two `BigInteger.valueOf` calls and the two `compareTo` calls per row, and it needs no static constants per type.
The `toBigInteger()` on the line above still allocates once per row. Let me know if you want that removed as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define CAST and TRY_CAST from VARIANT to scalar types [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #28758:
URL: https://github.com/apache/flink/pull/28758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Define VARIANT cast rules so a cast never alters the stored value [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3718619712
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.binary.StringUtf8Utils;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast succeeds only when the target holds the stored value without altering it, so a value is
+ * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches
+ * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE}
+ * are the exception to exactness: they are approximate by definition, so they accept any numeric
+ * kind and reject only a magnitude they cannot represent at all.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the target range. An
+ * approximate or decimal value is accepted only when it is already integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number) variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the
+ * precision and scale without rounding, although trailing zeros may be appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number) variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision, scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so
+ * the value is accepted only when its fractional seconds fit the target precision.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision));
+ }
+ final LocalDateTime value = variant.getDateTime();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP");
+ return TimestampData.fromLocalDateTime(value);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision));
+ }
+ final Instant value = variant.getInstant();
+ checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ");
+ return TimestampData.fromInstant(value);
+ }
+
+ /**
+ * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation
+ * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound).
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) {
+ final byte[] value = variant.getBytes();
+ final boolean fits =
+ fixedLength ? value.length == targetLength : value.length <= targetLength;
+ if (!fits) {
+ throw new TableRuntimeException(
+ String.format(
+ "The VARIANT binary value of length %d does not fit %s(%d); VARIANT "
+ + "casts do not pad or truncate.",
+ value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength));
+ }
+ return value;
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular
+ * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding
+ * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound).
+ *
+ * <p>A stored binary value has to be well-formed UTF-8, since a character string cannot carry
+ * bytes that no character maps to. Invalid input is rejected rather than decoded into {@code
+ * U+FFFD}, which would silently substitute a character the value never held.
+ *
+ * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value
+ */
+ public static String toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ value = variant.get().toString();
+ break;
+ case DECIMAL:
+ // toPlainString rather than toString, so that a small scale is not rendered in
+ // scientific notation, matching a regular DECIMAL to string cast.
+ value = variant.getDecimal().toPlainString();
+ break;
+ case STRING:
+ value = variant.getString();
+ break;
+ case BYTES:
+ // SQL reads a binary value as UTF-8, the same as a regular BINARY to string cast.
+ final byte[] utf8 = variant.getBytes();
+ final int invalidAt =
+ StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0, utf8.length);
+ if (invalidAt >= 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast the VARIANT binary value to %s because it is not "
+ + "valid UTF-8; the first invalid byte is at index %d "
+ + "of %d. Cast to BYTES to inspect the raw value, or "
+ + "wrap that in MAKE_VALID_UTF8 to replace every "
+ + "invalid byte with the U+FFFD replacement character.",
+ characterTarget(targetLength, charTarget),
+ invalidAt,
+ utf8.length));
+ }
+ value = new String(utf8, StandardCharsets.UTF_8);
+ break;
+ case DATE:
+ value = DateTimeUtils.formatDate((int) variant.getDate().toEpochDay());
+ break;
+ case TIMESTAMP:
+ // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A
+ // variant keeps microseconds, so the precision is always 6.
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromLocalDateTime(variant.getDateTime()),
+ DateTimeUtils.UTC_ZONE,
+ TIMESTAMP_PRECISION);
+ break;
+ case TIMESTAMP_LTZ:
+ value =
+ DateTimeUtils.formatTimestamp(
+ TimestampData.fromInstant(variant.getInstant()),
+ sessionZone,
+ TIMESTAMP_PRECISION);
+ break;
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s because the target does not "
+ + "accept NULL.",
+ characterTarget(targetLength, charTarget)));
+ default:
+ // An object or array has no scalar rendering.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character string. Use the "
+ + "JSON_STRING function to obtain its JSON representation.",
+ variant.getType()));
+ }
+ final boolean fits =
+ charTarget ? value.length() == targetLength : value.length() <= targetLength;
Review Comment:
this might be a problem for multibyte characters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610325327
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException} on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE, Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE, Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE, "BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to DECIMAL(%d, %d) overflowed.",
+ value, precision, scale));
+ }
+ return decimal;
+ }
+
+ /**
+ * Casts a {@code VARIANT} to its SQL string value: scalars use their raw value (a string stays
+ * unquoted, unlike {@code JSON_STRING}), objects and arrays use JSON.
+ */
+ public static String toStringValue(Variant variant) {
+ switch (variant.getType()) {
+ case OBJECT:
+ case ARRAY:
+ case BYTES:
+ return variant.toJson();
+ default:
+ return String.valueOf(variant.get());
+ }
+ }
+
+ private static long toIntegralExact(Number value, long min, long max, String targetType) {
+ // toBigInteger truncates any fractional part toward zero, matching regular numeric casts.
+ final BigInteger integral = toBigDecimal(value).toBigInteger();
+ if (integral.compareTo(BigInteger.valueOf(min)) < 0
+ || integral.compareTo(BigInteger.valueOf(max)) > 0) {
+ throw new ArithmeticException(
+ String.format(
+ "Casting the VARIANT value %s to %s overflowed.", value, targetType));
+ }
+ return integral.longValue();
+ }
+
+ private static BigDecimal toBigDecimal(Number value) {
+ return value instanceof BigDecimal ? (BigDecimal) value : new BigDecimal(value.toString());
Review Comment:
i guess if you check for `instanceof Float/Double` the code will be a bit shorter
in general yes, this code will be faster then (except for Double/Float however it is ok)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3610225458
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -120,6 +122,15 @@ void testParseJsonObject() throws IOException {
assertThat(variant.getField("k2").getDecimal()).isEqualTo(BigDecimal.valueOf(1.5));
}
+ @ParameterizedTest
+ @ValueSource(strings = {"NaN", "Infinity", "-Infinity"})
+ void testParseJsonRejectsNonNumericNumbers(final String nonNumericNumber) {
+ // NaN and the infinities are not valid JSON, so parsing fails: PARSE_JSON surfaces the
+ // error and TRY_PARSE_JSON returns NULL.
Review Comment:
`NaN` and `Infinity` are not part of the JSON specification: https://www.rfc-editor.org/info/rfc8259/#section-6
>Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.
Jackson does not allow it by default and you need to switch a flag to allow it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3613254180
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind |
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL` |
+| `true` / `false` | `BOOLEAN` |
+| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
+| Number exceeding 38 digits of precision or scale | `DOUBLE` |
+| String | `STRING` |
+| Array | array (elements encoded by the same rules) |
+| Object | object (values encoded by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
+```
+{{< /hint >}}
Review Comment:
what is difference here from non VARIANT case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org