You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2023/11/14 12:15:38 UTC
[I] Parquet: don't truncate for float16 [arrow-rs]
Jefffrey opened a new issue, #5075:
URL: https://github.com/apache/arrow-rs/issues/5075
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
See discussion:
https://github.com/apache/arrow-rs/pull/5003#discussion_r1389675899
https://github.com/apache/arrow-rs/pull/5003 added support for float16 type which is a logical type on top of fixed len byte array
When writing statistics, truncation can occur for binary physical type:
https://github.com/apache/arrow-rs/blob/7ba36b012322e08b06184c806f8ba339181cebc1/parquet/src/column/writer/mod.rs#L634-L643
Which might be troublesome for f16 type, if the `column_index_truncate_length` config is set to 1
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
Ignore truncation for f16
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
Do we need to worry about this for other types based on binary physical types? i.e. decimal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: don't truncate min/max statistics for float16 and decimal when writing file [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5075:
URL: https://github.com/apache/arrow-rs/issues/5075#issuecomment-1878522808
`label_issue.py` automatically added labels {'parquet'} from #5154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: don't truncate for float16 [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #5075:
URL: https://github.com/apache/arrow-rs/issues/5075#issuecomment-1810095241
Potentially has relation to https://github.com/apache/arrow-rs/issues/5037?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: don't truncate min/max statistics for float16 and decimal when writing file [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5075: Parquet: don't truncate min/max statistics for float16 and decimal when writing file
URL: https://github.com/apache/arrow-rs/issues/5075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org