[I] Make the native Iceberg writer's DictionaryChooser pre-pass cheaper [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6914:
URL: https://github.com/apache/datafusion-comet/issues/6914

   ### What is the problem the feature request solves?
   
   Before a partition's first rows reach the native Iceberg writer, `DictionaryChooser` replays parquet-mr's first-page dictionary decision for every column, so that native files use the encodings iceberg-java's would. It holds up to `write.parquet.page-row-limit` rows (20,000 by default) per partition and, for each column, inserts every value's bytes into a `HashSet<&[u8]>` through a per-value `value_bytes` closure, growing the set as it goes. The Parquet writer then dictionary-encodes the same values again.
   
   In `CometIcebergEndToEndBenchmark` (#6910), the pre-pass is 1.7% of all CPU for nested types and 2.4% for primitives, 4% to 8% of the writer. The cost is per partition, so a fanout write to many partitions pays it once for each.
   
   ### Describe the potential solution
   
   - Hash fixed-width values directly rather than as byte slices, and avoid the per-value dynamic dispatch through `value_bytes`.
   - Pre-size the set, which removes the rehashing visible in the profile.
   - Check whether the decision can be read from parquet-rs's own dictionary encoder state instead of a separate pass.
   
   ### Additional context
   
   `DictionaryChooser::choose` and `FirstPage::add` in `native/core/src/execution/operators/iceberg_dictionary.rs`; `PartitionFeed::release` in `native/core/src/execution/operators/iceberg_write.rs`. Measured on the fully native configuration of `CometIcebergEndToEndBenchmark`.
   
   Part of #6911.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Make the native Iceberg writer's DictionaryChooser pre-pass cheaper [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6914: Make the native Iceberg writer's DictionaryChooser pre-pass cheaper
URL: https://github.com/apache/datafusion-comet/issues/6914


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org