[I] Native Iceberg writes re-parse Hadoop's XML configuration in every task to re-read footers [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6912:
URL: https://github.com/apache/datafusion-comet/issues/6912

   ### What is the problem the feature request solves?
   
   After the native Iceberg writer finishes a task, `CometIcebergWriteExec` calls `IcebergReflection.rebuildDataFilesWithJavaMetrics`, which opens every data file the task wrote and reads its Parquet footer (`readParquetFooter`), so that the `DataFile` metrics come from iceberg-java's `ParquetUtil.footerMetrics`.
   
   In `CometIcebergEndToEndBenchmark` (#6910), 81% of that time is `Configuration.loadResources`. Opening the file builds Parquet read options from the table `FileIO`'s Hadoop configuration, and the configuration each task deserializes parses Hadoop's default XML resources again. Reading the footer itself is about 6%. Altogether it is 1.1% of all CPU for nested types and 3.1% for primitives, about 6 ms per task. On an object store, each data file also costs an extra open and footer read.
   
   ### Describe the potential solution
   
   - Reuse a configuration whose resources are already loaded, for example one per executor JVM, so the XML is parsed once.
   - Better, avoid the re-read altogether. The native writer already holds each file's footer when it closes the file, so passing the serialized footer back with the task's commit message would let the JVM build the same metrics without opening the file.
   
   ### Additional context
   
   `IcebergReflection.rebuildDataFilesWithJavaMetrics` and `readParquetFooter` in `spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala`, called from `CometIcebergWriteExec.doExecute`. Measured on the fully native configuration of `CometIcebergEndToEndBenchmark`.
   
   Part of #6911.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Native Iceberg writes re-parse Hadoop's XML configuration in every task to re-read footers [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6912: Native Iceberg writes re-parse Hadoop's XML configuration in every task to re-read footers
URL: https://github.com/apache/datafusion-comet/issues/6912


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org