You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "manner (via GitHub)" <gi...@apache.org> on 2026/09/28 13:11:17 UTC
[PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
manner opened a new pull request, #29323:
URL: https://github.com/apache/flink/pull/29323
## What is the purpose of the change
`BinaryVariantInternalBuilder.appendDecimal` writes `(byte) d.scale()` without validating the decimal. There is no check for a negative scale, and precision/scale above 38 is only guarded by an `assert`. The variant spec requires a scale in [0, 38] and a precision of at most 38.
So `VariantBuilder.of(BigDecimal)`, e.g. from a UDF, can write variants that fail later on read with `MALFORMED_VARIANT`. When the scale byte wraps around, it even reads back a different value: `of(new BigDecimal("1e2147483647")).toJson()` returns `0.1`.
This PR validates in `appendDecimal`, so all callers are covered. It follows the same rules Spark uses in its CSV/XML variant parsers ([SPARK-54099,](https://issues.apache.org/jira/browse/SPARK-54099) [SPARK-55932](https://issues.apache.org/jira/browse/SPARK-55932)).
## Brief change log
- A negative scale is rescaled to 0 with `setScale(0)`, which keeps the value. Non-zero values with a scale below -38 are rejected before that, because they can't fit anyway.
- Decimals whose precision or scale is still above 38 are rejected with a `VariantTypeException`. The builder already uses this exception for out-of-range timestamps.
- Removed the `assert` and documented the behavior on `VariantBuilder.of(BigDecimal)`.
- PARSE_JSON is unchanged. `tryParseDecimal` already checks the range and falls back to double.
## Verifying this change
This change added tests and can be verified as follows:
- `BinaryVariantTest`: negative scale, including a round trip through `getDecimal()`, which strips trailing zeros and returns `1E+2` for a variant holding 100.
- `BinaryVariantTest`: precision and scale of exactly 38.
- `BinaryVariantTest`: out-of-range values: precision 39, scale 39, `1e38`, `-1e999999999` and `1e2147483647`.
- `BinaryVariantInternalBuilderTest`: PARSE_JSON still stores numbers outside the decimal range as double.
- The new builder tests fail without the fix.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes. `VariantBuilder` is `@PublicEvolving`. The signature is unchanged, but `of(BigDecimal)` now throws a `VariantTypeException` for decimals that don't fit, instead of returning a broken variant.
- The serializers: no.
- The runtime per-record code paths (performance sensitive): yes, `appendDecimal` runs per record. The added checks are a few int comparisons, and `setScale` only runs for negative scales.
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? JavaDocs
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4131456837
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
+ result = d.setScale(0);
Review Comment:
Just add a one line of comment on top and say that for for a negative scale the result is precision - scale. So in the case of `12345e34` is 5 - (-34) = 39.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4123902818
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
Review Comment:
I see BinaryVariantUtil.java has method checkDecimal. I wonder if we can reuse this utils method , or create another method that extends the checking. This utils class seems to be the place to put this checking logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #29323:
URL: https://github.com/apache/flink/pull/29323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4146102559
##########
flink-core/src/main/java/org/apache/flink/types/variant/VariantBuilder.java:
##########
@@ -58,7 +58,12 @@ public interface VariantBuilder {
/** Create a variant from a boolean. */
Variant of(boolean b);
- /** Create a variant from a BigDecimal. */
+ /**
+ * Create a variant from a BigDecimal. A negative scale is rescaled to 0, which keeps the
+ * numeric value.
+ *
+ * @throws VariantTypeException if the precision or the scale of the rescaled decimal exceeds 38
Review Comment:
is 38 same value as at `{@link BinaryVariantUtil#MAX_DECIMAL16_PRECISION}` ?
can we refer to this then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4122823757
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
+ result = d.setScale(0);
+ }
+ if (result.scale() > MAX_DECIMAL16_PRECISION
+ || result.precision() > MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
Review Comment:
Can't we move this in the beginning of this method? If `d` has this conditions we throw.
```java
if (d.scale() > MAX_DECIMAL16_PRECISION
|| d.precision() > MAX_DECIMAL16_PRECISION) {
throw decimalOutOfRange(d);
}
```
`result = d.setScale(0)` will never satisfy this condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4146119420
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantTest.java:
##########
@@ -114,6 +114,53 @@ void testScalarVariant() {
assertThat(builder.ofNull().isNull()).isTrue();
}
+ @Test
+ void testDecimalWithNegativeScale() {
+ Variant variant = builder.of(new BigDecimal("-1e5"));
+ assertThat(variant.getType()).isEqualTo(Variant.Type.DECIMAL);
+ assertThat(variant.getDecimal()).isEqualByComparingTo("-100000");
+ assertThat(variant.toJson()).isEqualTo("-100000");
+
+ // 38 digits after rescaling
+ assertThat(builder.of(new BigDecimal("1e37")).getDecimal()).isEqualByComparingTo("1e37");
+ assertThat(builder.of(new BigDecimal("0e999999999")).getDecimal())
+ .isEqualByComparingTo(BigDecimal.ZERO);
+
+ // getDecimal() strips trailing zeros, so its result can have a negative scale
+ BigDecimal stripped = builder.of(BigDecimal.valueOf(100)).getDecimal();
+ assertThat(stripped.scale()).isNegative();
+ assertThat(builder.of(stripped).getDecimal()).isEqualByComparingTo("100");
+ }
+
+ @ParameterizedTest
+ @ValueSource(
+ strings = {
+ "12345678901234567890123456789012345678",
+ "-0.12345678901234567890123456789012345678"
+ })
+ void testDecimalWithMaxPrecisionAndScale(final String decimal) {
+ assertThat(builder.of(new BigDecimal(decimal)).getDecimal()).isEqualByComparingTo(decimal);
+ }
+
+ @ParameterizedTest
+ @ValueSource(
+ strings = {
+ // precision 39
+ "123456789012345678901234567890123456789",
+ // scale 39
+ "0.000000000000000000000000000000000000001",
+ // 39 digits after rescaling the negative scale
+ "1e38",
+ "-1e999999999",
+ // the scale does not fit into a byte
+ "1e2147483647"
+ })
+ void testDecimalOutOfRange(final String decimal) {
+ assertThatThrownBy(() -> builder.of(new BigDecimal(decimal)))
+ .isInstanceOf(VariantTypeException.class)
+ .hasMessageContaining("outside the range supported by variant decimals");
Review Comment:
can we compare against the whole message rather than containing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "manner (via GitHub)" <gi...@apache.org>.
manner commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4132778205
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
Review Comment:
I added a helper `fitsVariantDecimal()` in `BinaryVariantUtil` that's now doing the precision and scale checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "manner (via GitHub)" <gi...@apache.org>.
manner commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4123174937
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
+ result = d.setScale(0);
+ }
+ if (result.scale() > MAX_DECIMAL16_PRECISION
+ || result.precision() > MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
Review Comment:
There are some edge cases where this can still happen. For example, `12345e34` has scale -34 and precision 5. After rescaling to scale 0, it has a precision of 39, which the variant doesn't support.
But we can rewrite the check as:
```java
if (d.signum() != 0 && (long) d.precision() - d.scale() > MAX_DECIMAL16_PRECISION) {
throw decimalOutOfRange(d);
}
```
Then we don't need another check after rescaling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4146004268
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -143,6 +143,19 @@ void testParseJsonRejectsNonFiniteNumbers(final String nonFiniteNumber) {
.isInstanceOf(IOException.class);
}
+ @ParameterizedTest
+ @ValueSource(
+ strings = {
+ "123456789012345678901234567890123456789",
+ "0.000000000000000000000000000000000000001"
+ })
+ void testParseJsonStoresNumbersOutsideDecimalRangeAsDouble(final String number)
+ throws IOException {
+ BinaryVariant variant = BinaryVariantInternalBuilder.parseJson(number, false);
+ assertThat(variant.getType()).isEqualTo(Variant.Type.DOUBLE);
Review Comment:
```suggestion
assertThat(variant.getType()).isSameAs(Variant.Type.DOUBLE);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29323:
URL: https://github.com/apache/flink/pull/29323#discussion_r4123609962
##########
flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantInternalBuilder.java:
##########
@@ -271,6 +276,33 @@ public void appendDecimal(BigDecimal d) {
}
}
+ // The variant spec requires a scale in [0, 38] and a precision of at most 38.
+ private static BigDecimal toVariantDecimal(BigDecimal d) {
+ BigDecimal result = d;
+ if (d.scale() < 0) {
+ // A non-zero value with a scale below -38 has more than 38 digits after rescaling.
+ // Reject it upfront because setScale is expensive for huge exponents like 1e999999999.
+ if (d.signum() != 0 && d.scale() < -MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
+ result = d.setScale(0);
+ }
+ if (result.scale() > MAX_DECIMAL16_PRECISION
+ || result.precision() > MAX_DECIMAL16_PRECISION) {
+ throw decimalOutOfRange(d);
+ }
Review Comment:
Okay, then lets keep it as is. Thanks for the explenation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40832][core] Validate decimal range in VariantBuilder.of(BigDecimal) [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29323:
URL: https://github.com/apache/flink/pull/29323#issuecomment-5870827909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "992c51c0f2eb3130bed02e85520536a30d854188",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "992c51c0f2eb3130bed02e85520536a30d854188",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 992c51c0f2eb3130bed02e85520536a30d854188 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org