You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/11 11:22:14 UTC

[GitHub] [arrow-datafusion] alamb commented on issue #4169: Add ability to specify external sort information for ParquetExec

alamb commented on issue #4169:
URL: https://github.com/apache/arrow-datafusion/issues/4169#issuecomment-1311572149

   >  One alternative would be to encode the "sorted by" property into the parquet file itself. Sure that's more effort, but I kinda think that it would be nicer for the ecosystem. This metadata would be optional and solely help optimization (although if specified, it must be correct). This is very similar to statistics.
   
   Yes, I agree this would be nice if there was some standard way to do so. 
   
   I poked around in the format definition and it seems like there is a standard way to encode the sort order in each Row Group's metadata:
   
   There is a "SortingColumn" in the format
   https://github.com/apache/parquet-format/blob/54e53e5d7794d383529dd30746378f19a12afd58/src/main/thrift/parquet.thrift#L685-L698
   
   Which is then in the RowGroup metadata:
   https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L829-L832
   
   However, I did not find any code to read/write this in the parquet crate
   https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/apache/arrow-rs%24+SortingColumn&patternType=standard
   
   I will file some follow on tickets to properly support this in parquet and in datafusion. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org