You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stevomitric (via GitHub)" <gi...@apache.org> on 2026/03/10 13:11:07 UTC
[PR] [SPARK-55932] Fix XML to variant parser hang on negative scale [spark]
stevomitric opened a new pull request, #54726:
URL: https://github.com/apache/spark/pull/54726
### What changes were proposed in this pull request?
Add a guard before `setScale(0):` if `scale < -MAX_DECIMAL16_PRECISION` (-38), skip the decimal path and store the value as a string. Any value with scale below -38 is guaranteed to exceed the 38-digit precision limit after normalization, so the guard is a pure optimization with no correctness impact.
### Why are the changes needed?
When parsing XML or CSV data to Variant type, a decimal string with an extreme positive exponent (e.g., "1e9999999") causes the executor to hang indefinitely.
Java's BigDecimal parses such strings into a compact representation (`unscaledValue=1`, `scale=-9999999`). The variant parser then calls `d.setScale(0)` to normalize the scale before storing in the variant binary format. This forces the JDK to compute `BigInteger.pow(10, 9999999)`, materializing a number with ~10 million digits via multiplication - an operation that takes very long and effectively hangs the task.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit tests in this PR.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55932] Fix XML to variant parser hang on negative scale [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #54726: [SPARK-55932] Fix XML to variant parser hang on negative scale
URL: https://github.com/apache/spark/pull/54726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55932] Fix XML to variant parser hang on negative scale [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #54726:
URL: https://github.com/apache/spark/pull/54726#issuecomment-4035475035
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55932] Fix XML to variant parser hang on negative scale [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #54726:
URL: https://github.com/apache/spark/pull/54726#issuecomment-4031283095
cc @cloud-fan to PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55932] Fix XML to variant parser hang on negative scale [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54726:
URL: https://github.com/apache/spark/pull/54726#discussion_r2911935938
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:
##########
@@ -496,17 +496,24 @@ class UnivocityParser(
def parseDecimal(): DataType = {
try {
var d = decimalParser(s)
- if (d.scale() < 0) {
- d = d.setScale(0)
- }
- if (d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION &&
- d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION) {
- builder.appendDecimal(d)
- // The actual decimal type doesn't matter. `appendDecimal` will use the smallest
- // possible decimal type to store the value.
- DecimalType.USER_DEFAULT
- } else {
+ if (d.scale() < -VariantUtil.MAX_DECIMAL16_PRECISION) {
+ // Scale is so extremely negative that setScale(0) would require computing
+ // bigTenToThe(|scale|), which is prohibitively expensive. The resulting precision
+ // would also exceed MAX_DECIMAL16_PRECISION, so fall through.
Review Comment:
Nit: the XML version says "fall through to string" — suggest matching here for consistency.
```suggestion
// would also exceed MAX_DECIMAL16_PRECISION, so fall through to string.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org