You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/03/31 21:51:10 UTC
[PR] Fix incorrect decimal conversion for binary-encoded Parquet decimals (precision <= 18) [pinot]
Jackie-Jiang opened a new pull request, #18055:
URL: https://github.com/apache/pinot/pull/18055
## Summary
- The `binaryToDecimal` fast path for `precision <= 18` applied scale via floating-point division (`BigDecimal.valueOf(unscaled / pow(10, scale))`), which has two bugs:
1. **Trailing zeros lost**: e.g. unscaled=110, scale=2 produced `"1.1"` (scale 1) instead of `"1.10"` (scale 2). `BigDecimal.equals` is scale-sensitive, so comparisons fail.
2. **Precision loss**: for large unscaled values near the 18-digit boundary, `double`'s ~15.9 significant digit limit corrupts the result (e.g. `123456789012345678 / 1e6` rounds incorrectly).
3. **Scale dropped on overflow**: the overflow branch returned `new BigDecimal(unscaledNew)` without applying scale.
- Fix: replace the entire method with `new BigDecimal(new BigInteger(bytes), scale)`, which is the standard approach matching the Parquet spec's two's complement big-endian encoding. This is already what the `precision > 18` branch did correctly.
## Test plan
- Added two test cases to `ParquetReaderLogicalTypesTest.testDecimalType`:
- `DECIMAL(10, 2)` binary with value `"1.10"` — catches the trailing-zero/scale bug
- `DECIMAL(18, 6)` binary with value `"123456789012.345678"` — catches the floating-point precision loss bug
- Both assertions would fail against the old code and pass with the fix.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix incorrect decimal conversion for binary-encoded Parquet decimals (precision <= 18) [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18055:
URL: https://github.com/apache/pinot/pull/18055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix incorrect decimal conversion for binary-encoded Parquet decimals (precision <= 18) [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18055:
URL: https://github.com/apache/pinot/pull/18055#issuecomment-4166064651
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18055?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.48%. Comparing base ([`abdb6cb`](https://app.codecov.io/gh/apache/pinot/commit/abdb6cba2d80ff07599194b000dabc95f519cbf6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`42d7944`](https://app.codecov.io/gh/apache/pinot/commit/42d79448e1bd68ce94938b381e91342b6a6ef3ef?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (abdb6cb) and HEAD (42d7944). Click for more details.
>
> <details><summary>HEAD has 36 uploads less than BASE</summary>
>
>| Flag | BASE (abdb6cb) | HEAD (42d7944) |
>|------|------|------|
>|java-21|5|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|10|1|
>|java-11|5|0|
>|unittests2|2|0|
>|integration|6|0|
>|integration2|2|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18055 +/- ##
============================================
- Coverage 63.34% 55.48% -7.87%
+ Complexity 1543 745 -798
============================================
Files 3200 2502 -698
Lines 194222 143160 -51062
Branches 29934 22986 -6948
============================================
- Hits 123026 79429 -43597
+ Misses 61526 56977 -4549
+ Partials 9670 6754 -2916
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.48% <ø> (-7.84%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.48% <ø> (-7.87%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.48% <ø> (-7.86%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.48% <ø> (-0.04%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18055/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18055?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org