You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/10/16 01:01:55 UTC
(spark) branch branch-4.1 updated: [SPARK-60119][4.1][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new a311ee206068 [SPARK-60119][4.1][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
a311ee206068 is described below
commit a311ee20606829e8ee1f3ecd8e32ea3251e81abe
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu Oct 15 18:01:44 2026 -0700
[SPARK-60119][4.1][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
### What changes were proposed in this pull request?
Backport of #59335 to branch-4.1.
The cherry-pick conflicted only on `CastUtils.java`, which does not exist on this branch: `Cast` calls `QueryExecutionErrors.cannotChangeDecimalPrecisionError` directly in both the interpreted and generated code. So instead of removing that method and switching `CastUtils`, this backport keeps it and makes it delegate to `DataTypeErrors.cannotChangeDecimalPrecisionError`, which gives the same error message. The rest of the change is identical to #59335.
In the BigDecimal branch of `Decimal.changePrecision`, when the scale change exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits (`precision - scale`, computed in Long) is checked before calling `setScale`:
- If the value is nonzero and has more integral digits than the target type allows, return `false` (overflow) directly. Rounding cannot bring it back into range.
- If the value is below 0.01 ulp of the target scale, the result depends only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is rescaled instead. This gives the same result for every rounding mode.
The Long branch of `changePrecision` now computes the scale change in Long, so it no longer wraps and throws `ArrayIndexOutOfBoundsException` from `POW_10(diff)` when negative scales are allowed.
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose scale is below -1000 (only possible under `spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of `toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a String. Other values are rendered as before. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy, now delegates to `DataTypeErrors.cannotChangeDecimalPrecisionError`.
### Why are the changes needed?
`setScale` takes time that grows with the scale change, and throws once 10^change exceeds BigInteger's range or the change exceeds the Int range:
```sql
SELECT CAST('1e-2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: BigInteger would overflow supported range (LEGACY, ANSI and TRY)
SELECT CAST('1e-100000000' AS DECIMAL(10,2));
-- 0.00, after ~40s
SET spark.sql.legacy.allowNegativeScaleOfDecimal=true;
SELECT CAST('1e2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: Underflow
```
This branch already has the related fix SPARK-60118, so backporting this keeps decimal casts consistent with master.
### Does this PR introduce _any_ user-facing change?
Yes, the same as #59335:
- Casting a string whose value is far below the target scale's ulp now returns the rounded value (e.g. 0.00) instead of throwing `ArithmeticException`, and returns quickly.
- With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string with a huge exponent returns NULL (non-ANSI, try_cast) or throws `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException: Underflow`, and a decimal stored as a long no longer throws `ArrayIndexOutOfBoundsException` for a scale change beyond the Int range. In that error, a value whose scale is below -1000 is shown in scientific notation.
No input that previously produced a value changes its result.
### How was this patch tested?
The tests from #59335 in `DecimalSuite`, `CastSuiteBase`, `CastWithAnsiOnSuite` and `CastWithAnsiOffSuite` (also run through `TryCastSuite`). On this branch with JDK 17, {'4.3': '512 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`, plus `UdfSerializationSuite` (16 tests)', '4.2': '410 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCa [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59514 from viirya/SPARK-60119-4.1.
Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
.../apache/spark/sql/errors/DataTypeErrors.scala | 11 ++-
.../scala/org/apache/spark/sql/types/Decimal.scala | 27 ++++++--
.../spark/sql/errors/QueryExecutionErrors.scala | 11 +--
.../sql/catalyst/expressions/CastSuiteBase.scala | 6 ++
.../expressions/CastWithAnsiOffSuite.scala | 9 +++
.../catalyst/expressions/CastWithAnsiOnSuite.scala | 39 +++++++++++
.../org/apache/spark/sql/types/DecimalSuite.scala | 80 ++++++++++++++++++++++
7 files changed, 167 insertions(+), 16 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
index dcad3567bef8..b91a51cb331d 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
@@ -196,15 +196,24 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
numericValueOutOfRange(value, decimalPrecision, decimalScale, context)
}
+ // A decimal whose scale is below -MAX_PLAIN_STRING_TRAILING_ZEROS (that is, with more than
+ // this many zeros implied by a negative scale) is rendered in scientific notation in error
+ // messages instead of plain notation.
+ private val MAX_PLAIN_STRING_TRAILING_ZEROS = 1000
+
private def numericValueOutOfRange(
value: Decimal,
decimalPrecision: Int,
decimalScale: Int,
context: QueryContext): ArithmeticException = {
+ // A huge negative scale (legacy mode only) makes the plain string too long to build,
+ // e.g. for 1E+2147483647.
+ val valueString =
+ if (value.scale < -MAX_PLAIN_STRING_TRAILING_ZEROS) value.toString else value.toPlainString
new SparkArithmeticException(
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
messageParameters = Map(
- "value" -> value.toPlainString,
+ "value" -> valueString,
"precision" -> decimalPrecision.toString,
"scale" -> decimalScale.toString,
"config" -> toSQLConf("spark.sql.ansi.enabled")),
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
index 192244fe2b71..1ba307362a51 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
@@ -398,8 +398,9 @@ final class Decimal extends Ordered[Decimal] with Serializable {
// First, update our lv if we can, or transfer over to using a BigDecimal
if (dv.eq(null)) {
if (scale < _scale) {
- // Easier case: we just need to divide our scale down
- val diff = _scale - scale
+ // Easier case: we just need to divide our scale down. The difference is computed in
+ // Long since it can exceed the Int range when negative scales are allowed.
+ val diff = _scale.toLong - scale
// If diff is greater than max number of digits we store in Long, then
// value becomes 0. Otherwise we calculate new value dividing by power of 10.
// In both cases we apply rounding after that.
@@ -411,7 +412,7 @@ final class Decimal extends Ordered[Decimal] with Serializable {
case _ => throw DataTypeErrors.unsupportedRoundingMode(roundMode)
}
} else {
- val pow10diff = POW_10(diff)
+ val pow10diff = POW_10(diff.toInt)
// % and / always round to 0
val droppedDigits = lv % pow10diff
lv /= pow10diff
@@ -440,11 +441,11 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (scale > _scale) {
// We might be able to multiply lv by a power of 10 and not overflow, but if not,
// switch to using a BigDecimal
- val diff = scale - _scale
- val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0))
+ val diff = scale.toLong - _scale
+ val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0L).toInt)
if (diff <= MAX_LONG_DIGITS && lv > -p && lv < p) {
// Multiplying lv by POW_10(diff) will still keep it below MAX_LONG_DIGITS
- lv *= POW_10(diff)
+ lv *= POW_10(diff.toInt)
} else {
// Give up on using Longs; switch to BigDecimal, which we'll modify below
dv = BigDecimal(lv, _scale)
@@ -456,6 +457,20 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (dv.ne(null)) {
// We get here if either we started with a BigDecimal, or we switched to one because we would
// have overflowed our Long; in either case we must rescale dv to the new scale.
+ if (math.abs(dv.scale.toLong - scale) > DecimalType.MAX_PRECISION) {
+ // setScale's cost grows with the scale change, and it throws once 10^change exceeds
+ // BigInteger's range (e.g. for 1e-2147483647) or the change exceeds the Int range.
+ // Skip it when the result is known.
+ val numIntegralDigits = dv.precision.toLong - dv.scale
+ if (dv.signum != 0 && numIntegralDigits > precision.toLong - scale) {
+ return false
+ }
+ if (numIntegralDigits < -scale.toLong - 1) {
+ // |dv| is below 0.01 ulp of the new scale, so the result depends only on the sign
+ // and the rounding mode. Round a value of the same sign and 0.01 ulp instead.
+ dv = BigDecimal(dv.signum, scale + 2)
+ }
+ }
dv = dv.setScale(scale, roundMode)
if (dv.precision > precision) {
return false
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 351868fcc2e2..c7e3848a9747 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -107,15 +107,8 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
decimalPrecision: Int,
decimalScale: Int,
context: QueryContext = null): ArithmeticException = {
- new SparkArithmeticException(
- errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
- messageParameters = Map(
- "value" -> value.toPlainString,
- "precision" -> decimalPrecision.toString,
- "scale" -> decimalScale.toString,
- "config" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
- context = getQueryContext(context),
- summary = getSummary(context))
+ DataTypeErrors.cannotChangeDecimalPrecisionError(
+ value, decimalPrecision, decimalScale, context)
}
def invalidInputSyntaxForBooleanError(
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 5e8df1f14938..ca9a68de66ae 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -1352,6 +1352,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
+ test("SPARK-60119: cast string with a huge negative exponent to decimal") {
+ Seq("1e-2147483647", "-1e-2147483647", "1e-100000000", "0e-2147483647").foreach { str =>
+ checkEvaluation(cast(str, DecimalType(10, 2)), Decimal("0.00"))
+ }
+ }
+
test("cast ANSI intervals to/from decimals") {
Seq(
(Duration.ZERO, DayTimeIntervalType(DAY), DecimalType(10, 3)) -> Decimal(0, 10, 3),
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
index 53a5e51f7d56..a924bbaffd3c 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
@@ -554,6 +554,15 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+ test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
+ withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
+ Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
+ checkEvaluation(cast(str, DecimalType(10, 2)), null)
+ }
+ checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
+ }
+ }
+
test("data type casting II") {
checkEvaluation(
cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
index 6db67d26a4be..e366638e0884 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
@@ -461,6 +461,45 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
castErrMsg("abcd", DecimalType(38, 1)))
}
+ test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
+ withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
+ Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
+ checkExceptionInExpression[ArithmeticException](
+ cast(str, DecimalType(10, 2)),
+ "cannot be represented as Decimal(10, 2)")
+ }
+ checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
+
+ // The value is shown in plain notation unless its scale is hugely negative.
+ Seq("1e2147483647" -> "1E+2147483647", "1e40" -> ("1" + "0" * 40)).foreach {
+ case (str, value) =>
+ def expr: Expression = cast(str, DecimalType(10, 2))
+ if (!isTryCast) {
+ def checkOutOfRange(eval: => Any): Unit = {
+ checkError(
+ exception = intercept[SparkArithmeticException](eval),
+ condition = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
+ parameters = Map(
+ "value" -> value,
+ "precision" -> "10",
+ "scale" -> "2",
+ "config" -> """"spark.sql.ansi.enabled""""),
+ queryContext = Array(ExpectedContext(fragment = "", start = -1, stop = -1)))
+ }
+ // Interpreted, and both codegen paths without falling back to interpreted.
+ checkOutOfRange(evaluateWithoutCodegen(expr))
+ withSQLConf(
+ SQLConf.CODEGEN_FACTORY_MODE.key -> CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
+ checkOutOfRange(evaluateWithMutableProjection(expr))
+ checkOutOfRange(evaluateWithUnsafeProjection(expr))
+ }
+ } else {
+ checkEvaluation(expr, null)
+ }
+ }
+ }
+ }
+
protected def checkCastToBooleanError(l: Literal, to: DataType, tryCastResult: Any): Unit = {
checkExceptionInExpression[SparkRuntimeException](
cast(l, to), """cannot be cast to "BOOLEAN"""")
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
index 35b3e96a7418..0a7b0fafc54d 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
@@ -501,4 +501,84 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
s"unscaled: $unscaled, scaleFrom: $scaleFrom, scaleTo: $scaleTo, mode: $roundMode")
}
}
+
+ test("SPARK-60119: changePrecision with a source scale far from the target scale") {
+ // A value below 0.01 ulp of the target scale rounds to 0 or +/-1 ulp depending only on its
+ // sign and the rounding mode, so it must round like a small value of the same sign.
+ Seq("1e-2147483647", "-1e-2147483647", "9.99e-100000000", "-1e-100000000",
+ "0e-2147483647").foreach { str =>
+ val bd = new java.math.BigDecimal(str)
+ allSupportedRoundModes.foreach { mode =>
+ val expected = BigDecimal(bd.signum, 50).setScale(2, mode)
+ val d = Decimal(bd)
+ assert(d.changePrecision(10, 2, mode), s"$str, $mode")
+ assert(d.toString === expected.toString, s"$str, $mode")
+ }
+ }
+
+ withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
+ Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
+ allSupportedRoundModes.foreach { mode =>
+ val d = Decimal(new java.math.BigDecimal(str))
+ assert(d.toPrecision(10, 2, mode) === null, s"$str, $mode")
+ }
+ }
+ val zero = Decimal(new java.math.BigDecimal("0e2147483647"))
+ assert(zero.changePrecision(10, 2))
+ assert(zero.toString === "0.00")
+
+ // A compact (long) input takes the Long branch first, where the scale change can exceed
+ // the Int range. These switch to the BigDecimal branch, so check their results directly.
+ Seq(
+ (1L, 1, -40, 10, 2, null),
+ (0L, 1, -2147483647, 10, 2, "0.00"),
+ (1L, 1, Int.MinValue, 38, 20, null)).foreach { case (unscaled, p, s, toP, toS, expected) =>
+ allSupportedRoundModes.foreach { mode =>
+ val compact = Decimal.createUnsafe(unscaled, p, s)
+ checkCompact(compact, true)
+ assert(Option(compact.toPrecision(toP, toS, mode)).map(_.toString).orNull === expected,
+ s"$unscaled, ($p, $s) -> ($toP, $toS), $mode")
+ }
+ }
+ // These stay in the Long branch, so compare with the same value in BigDecimal form.
+ Seq(100L, -100L).foreach { unscaled =>
+ allSupportedRoundModes.foreach { mode =>
+ val compact = Decimal.createUnsafe(unscaled, 3, 2)
+ checkCompact(compact, true)
+ val expanded = Decimal(java.math.BigDecimal.valueOf(unscaled, 2))
+ checkCompact(expanded, false)
+ assert(
+ Option(compact.toPrecision(1, -2147483647, mode)).map(_.toString) ===
+ Option(expanded.toPrecision(1, -2147483647, mode)).map(_.toString),
+ s"$unscaled, $mode")
+ }
+ }
+ }
+
+ // Values around the bounds of the shortcuts must round as BigDecimal.setScale does.
+ Seq(false, true).foreach { allowNegativeScale =>
+ withSQLConf(
+ SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> allowNegativeScale.toString) {
+ val targets =
+ Seq((10, 2), (3, 3), (5, 0)) ++ (if (allowNegativeScale) Seq((5, -3)) else Nil)
+ for {
+ // The long mantissas make the scale change large enough to take the shortcuts.
+ mantissa <- Seq("1", "4.9", "5", "5.1", "9.99", "9.995",
+ "1." + "0" * 79 + "1", "4." + "9" * 80, "5." + "0" * 80, "5." + "0" * 79 + "1",
+ "9." + "9" * 80)
+ exponent <- -8 to 12
+ sign <- Seq("", "-")
+ mode <- allSupportedRoundModes
+ (precision, scale) <- targets
+ } {
+ val bd = new java.math.BigDecimal(s"$sign${mantissa}e$exponent")
+ val rounded = BigDecimal(bd).setScale(scale, mode)
+ val expected = if (rounded.precision > precision) null else rounded.toString
+ val d = Decimal(bd).toPrecision(precision, scale, mode)
+ assert(Option(d).map(_.toString).orNull === expected,
+ s"$bd, $mode, ($precision, $scale)")
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org