You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ozenyunus35 (via GitHub)" <gi...@apache.org> on 2026/10/15 01:02:56 UTC

[PR] fix(spark): format_string %s of a DECIMAL and %g like Java's Formatter [datafusion]

ozenyunus35 opened a new pull request, #26346:
URL: https://github.com/apache/datafusion/pull/26346

   ## Which issue does this PR close?
   
   - Closes #26337.
   
   ## Rationale for this change
   
   Spark's `format_string` uses Java's `Formatter`, and `datafusion-spark` differed from it in two ways: `%s` of a DECIMAL printed the unscaled integer (`123456` for `123.456`), and `%g` / `%G` dropped trailing zeros like C (`1234.5`), while Java always prints the requested number of significant digits (`1234.50`).
   
   ## What changes are included in this PR?
   
   - `%s` / `%S` of a `Decimal128` or `Decimal256` prints the value with its scale in plain notation (`123.456`, `0.50`), through a small `decimal_to_plain_string` helper.
   - `%g` / `%G` no longer strip trailing zeros, for float and decimal arguments. The `strip_trailing_0s` flag and `trim_trailing_0s` were only used for this, so they are removed.
   - While adding cases, `%g` of `0.0` panicked with "attempt to subtract with overflow", because the exponent came from `log10(0)`. A `decimal_exponent` helper returns 0 for zero, so `%g` gives `0.00000` and `%e` gives `0.000000e+00`, as in Java.
   
   I checked every new expected value by running `String.format` with the same pattern on JDK 21 (`BigDecimal` for the decimal `%s` cases).
   
   One thing I left as is: Java's `BigDecimal.toString` switches to E notation for very small values (`1E-7` for `0.0000001`); this PR prints those in plain notation. I couldn't confirm what Spark's own `Decimal.toString` gives there without a Spark install, so I didn't guess.
   
   ## What is the testing strategy for this PR?
   
   - `format_string.slt`: the expectations the issue points at are updated (`%s` / `%S` of `Decimal128` and `Decimal256`, `%g` / `%G` of `Float64`, `Float32`, `Decimal128` and `Decimal256`), and new cases cover `%g` on small, large, zero, negative and rounded values, `%.3g`, `%.0g`, `%10.2g`, `%e` of zero, and `%s` / `%S` / `%10s` of decimals. On `main` the file fails (and the `%g` of `0.0` case panics).
   - `cargo test -p datafusion-spark --lib format_string` and `cargo clippy -p datafusion-spark --all-targets -- -D warnings` pass.
   
   ## Are there any user-facing changes?
   
   Yes: `format_string('%s', <decimal>)` prints the decimal with its scale, `%g` / `%G` keep trailing zeros, and `%g` of zero no longer panics.
   
   I used an AI assistant to help draft parts of the code and tests; I reviewed the changes and ran the tests myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(spark): format_string %s of a DECIMAL and %g like Java's Formatter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26346:
URL: https://github.com/apache/datafusion/pull/26346#issuecomment-6506378673

   Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.
   
   https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful
   
   Review capacity is very limited and this is one of three Spark compatibility PRs opened the same day across unrelated functions, with no engagement since. The policy asks new contributors to work with maintainers on one PR at a time rather than opening several at once.
   
   If this fix matters to you, please pick the single change you care most about, reopen or resubmit it with a short description in your own words, and we are happy to help with review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(spark): format_string %s of a DECIMAL and %g like Java's Formatter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #26346: fix(spark): format_string %s of a DECIMAL and %g like Java's Formatter
URL: https://github.com/apache/datafusion/pull/26346


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org