You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dwsmith1983 (via GitHub)" <gi...@apache.org> on 2026/10/07 18:34:37 UTC
[PR] [SPARK-60045][SQL] Avoid per-value regex compilation when parsing decimals in CSV/JSON/XML [spark]
dwsmith1983 opened a new pull request, #59273:
URL: https://github.com/apache/spark/pull/59273
### What changes were proposed in this pull request?
`ExprUtils.getDecimalParser` strips grouping commas with `s.replaceAll(",", "")` for the default `en-US` locale. This PR uses `s.replace(",", "")` instead. The pattern is a literal comma, so the result is the same, but `replace` does not compile a regex.
### Why are the changes needed?
Performance. `String.replaceAll` compiles a `java.util.regex.Pattern` on every call, and this parser runs once per decimal value in the CSV reader (`UnivocityParser`), the JSON reader (decimals given as strings), the XML reader and JSON/XML schema inference.
On a 2M row CSV file with 5 `DECIMAL(18,2)` columns (local[1], JDK 17, best of 3):
| | Before | After |
|---|---|---|
| One parse (`new BigDecimal(s.replaceAll/replace(",", ""))`) | 85.4 ns | 14.8 ns |
| Read the file with the DECIMAL schema | 1638 ms | 842 ms |
| Read the same file with the columns as STRING (reference) | 628 ms | 671 ms |
The DECIMAL read is 1.95x faster. Decimal parsing used to add about 1 s over reading the columns as strings and now adds about 0.17 s.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests already parse grouped values such as `1,000.01` and `158,058,049.001` through this code with the `en-US` locale. They pass: `UnivocityParserSuite`, `CSVInferSchemaSuite`, `JsonInferSchemaSuite`, `JsonExpressionsSuite`, `XmlExpressionsSuite`, and the decimal tests in `CSVv1Suite`, `JsonV1Suite` and `XmlSuite`.
The numbers above come from a local benchmark run before and after the change on the same commit.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60045][SQL] Avoid per-value regex compilation when parsing decimals in CSV/JSON/XML [spark]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #59273:
URL: https://github.com/apache/spark/pull/59273#issuecomment-6094934902
cc @MaxGekk @HyukjinKwon, a one-line change: `replaceAll` compiled a regex for every decimal value parsed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org