You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "PingLiuPing (via GitHub)" <gi...@apache.org> on 2025/08/26 09:43:23 UTC
[I] [C++] Min Max stats are wrong when the value of a column are both infinity or -infinity [arrow]
PingLiuPing opened a new issue, #47426:
URL: https://github.com/apache/arrow/issues/47426
### Describe the bug, including details regarding any error messages, version, and platform.
When writing parquet files, if the value of a float or double column are both infinity or -infinity, the min and max stats of this column is wrong.
For example, suppose the column is float, and all values are -std::numeric_limits<float>::infinity(), the min and max collected in TypeStatisticsImpl are [-Inf, std::numeric_limits<float>::lowest()].
But the actual result should be [-Inf, -Inf].
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Min Max stats are wrong when the value of a column are both infinity or -infinity [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #47426: [C++][Parquet] Min Max stats are wrong when the value of a column are both infinity or -infinity
URL: https://github.com/apache/arrow/issues/47426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Min Max stats are wrong when the value of a column are both infinity or -infinity [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #47426:
URL: https://github.com/apache/arrow/issues/47426#issuecomment-5982515648
Looks good to me; this code is where it was fixed; closing.
https://github.com/apache/arrow/blob/e85e181188ac3c5e8407b2160331faf5e78fd429/cpp/src/parquet/statistics.cc#L90-L107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Min Max stats are wrong when the value of a column are both infinity or -infinity [arrow]
Posted by "PingLiuPing (via GitHub)" <gi...@apache.org>.
PingLiuPing commented on issue #47426:
URL: https://github.com/apache/arrow/issues/47426#issuecomment-6053274950
Thank you for the fix and summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Min Max stats are wrong when the value of a column are both infinity or -infinity [arrow]
Posted by "stackedsax (via GitHub)" <gi...@apache.org>.
stackedsax commented on issue #47426:
URL: https://github.com/apache/arrow/issues/47426#issuecomment-5981157854
This was fixed by #50183 (GH-50182), released in Arrow 25.0.0. `DefaultMin()`/`DefaultMax()` in `cpp/src/parquet/statistics.cc` now seed the running min/max with `+Inf`/`-Inf` for floating-point types (including `FLOAT16`) instead of `numeric_limits<T>::max()`/`lowest()`, so an all-`-Inf` column now reports `[-Inf, -Inf]` as expected. `TestFloatStatistics.Infinities` in `statistics_test.cc` covers FLOAT, DOUBLE and FLOAT16.
Quick check on 25.0.0 or later:
```python
import io, pyarrow as pa, pyarrow.parquet as pq
inf = float("inf")
buf = io.BytesIO()
pq.write_table(pa.table({"c": pa.array([-inf, -inf, -inf])}), buf)
st = pq.ParquetFile(buf).metadata.row_group(0).column(0).statistics
print(st.min, st.max) # -inf -inf
```
Note for anyone arriving from facebookincubator/velox#14603: Velox vendors a copy of this file under `velox/dwio/parquet/writer/arrow/Statistics.cpp`, so it needs to pick up the equivalent change separately.
@thisisnic I think this can be closed as a duplicate of #50182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org