[I] Faster string min/max statistics in the parquet-rs writer [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6919:
URL: https://github.com/apache/datafusion-comet/issues/6919

   ### What is the problem the feature request solves?
   
   For byte-array columns, parquet-rs folds every value into the page's minimum and maximum (`compute_min_max` in `arrow_writer/byte_array.rs`) with two `Ord` comparisons, each a `memcmp` call. In `CometIcebergEndToEndBenchmark` (#6910) that is about 5.6% of all CPU for nested types, whose string columns are short and repeat often (map keys, tags). It is separate from dictionary encoding, which hashes the same values again.
   
   ### Describe the potential solution
   
   Upstream in arrow-rs:
   
   - When the dictionary encoder is active, compute page statistics over the distinct dictionary entries the page uses rather than over every value.
   - Compare short values without a `memcmp` call, for example by an 8-byte prefix first.
   
   ### Additional context
   
   parquet 59.3.0 (the version Comet uses): `encode` and `compute_min_max` in `parquet/src/arrow/arrow_writer/byte_array.rs`.
   
   Part of #6911.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Faster string min/max statistics in the parquet-rs writer [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove commented on issue #6919:
URL: https://github.com/apache/datafusion-comet/issues/6919#issuecomment-6157093974

   Removing this from the 1.2.0 milestone and unassigning myself so that someone from the community can pick it up. Contributions welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org