You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2023/06/17 05:06:55 UTC
[GitHub] [arrow] mapleFU opened a new issue, #36139: [C++][Parquet] Allow Truncate min-max Statistics
mapleFU opened a new issue, #36139:
URL: https://github.com/apache/arrow/issues/36139
### Describe the enhancement requested
Currently, in parquet c++, if the min-max is string / binary:
1. During writing values, min-max will be collect
2. When building statistics, there is apply truncate, which will discard min-max if they're longer than expected
Can we use a "truncate" for this:
1. For BINARY
1. If it's minimum, just truncate to length is ok
2. if it's maximum:
1. if the truncated binary would be 0xFF 0xFF ... 0xFF, we cannot truncate it
2. Otherwise, get the "next" valid truncated binary
2. For String
1. If it's minimum, just truncate and to **a valid utf8** is ok
2. If it's maximum, first truncate to a valid utf8, then try to advance it.
References:
1. https://github.com/apache/parquet-mr/pull/481
2. https://github.com/apache/arrow-rs/pull/4389
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #36139: [C++][Parquet] Allow Truncate min-max Statistics
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36139:
URL: https://github.com/apache/arrow/issues/36139#issuecomment-1601762024
Does truncate discard? Or does it write the truncated value and mark that it is truncated?
For example, if I know the minimum value is "Hello W..." then I can reject the row group if there is a filter "value == 'blue'" even if I don't know the complete minimum value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Allow Truncate min-max Statistics [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #36139:
URL: https://github.com/apache/arrow/issues/36139#issuecomment-3921758173
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Allow Truncate min-max Statistics [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36139:
URL: https://github.com/apache/arrow/issues/36139#issuecomment-1824241025
@westonpace Truncate is just Prefix truncate. Parquet-2.10 has released this:
1. Truncate means min might less than the exact mean, max might greater than exact maximum
2. Truncated value must be a valid value for this type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org