You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/04/22 16:41:10 UTC
[I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
paleolimbot opened a new issue, #46205:
URL: https://github.com/apache/arrow/issues/46205
### Describe the enhancement requested
The C++ Parquet implementation after adding variant and geometry will have several logical types with a sort order of UNKNOWN. The current implementation of statistics will not calculate a null count and add that statistic to the column metadata if the sort order is unknown, so this particular piece of information will be missing for geometry, geography, and variant. For geometry in particular, it will be needed to effectively push down a query rectangle (or else there is no mechanism to detect completely null row groups).
I'm not sure what the best way is to implement this...for geometry specifically we could keep track of the null count in the `GeoStatistics` but this wouldn't help with variant. I'm also not sure if the null count + statistics should be written at the page level for these types or not.
Noted by @wgtmac in https://github.com/apache/arrow/pull/45459
### Component(s)
Parquet, C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-3304192050
Apologies for abandoning the PR that got started on this! (I'm happy to review or unabandon it, although I am not sure I'll have time before the feature freeze to finish it since it seemed like it would require some reorganization).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-2843187570
No rush on my end (and no offense taken if either of you would rather take this on!), but I started #46275 to wrap my head around the issue. Happy to take pretty much any angle and run with it (or review if somebody else would like to take it on!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-3303571088
cc @raulcd who has been working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-2822845340
The current `parquet::Statistics` implementation is tied to `TypeDefinedOrder`. We will also add a `IEEE754TotalOrder` as proposed by https://github.com/apache/parquet-format/pull/221. Perhaps we can refactor the `parquet::Statistics` to be aware of column order? For `TypeDefinedOrder` and `IEEE754TotalOrder`, `parquet::Statistics` collects all fields. For `UndefinedOrder`, the stats writes empty `min` and `max` but keeps other fields including null_count.
cc @mapleFU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-3311775596
Hi @paleolimbot !
Just FYI, I didn't work on writing statistics when `SortOrder::UNKNOWN`, I only worked on keeping those statistics instead of throwing them away when reading, see:
- https://github.com/apache/arrow/pull/47466
But I do agree that the next step is following this up so we can also write those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-2823276918
> Perhaps we can refactor the `parquet::Statistics` to be aware of column order
`Statisitcs` has different part, `null_count` can not aware the order, but `min`, `max` and related should...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][C++] Logical types with sort order UNKNOWN are missing null_count statistics [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46205:
URL: https://github.com/apache/arrow/issues/46205#issuecomment-5741388715
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org