[GitHub] aokolnychyi edited a comment on issue #63: Use Iceberg writers for Parquet data written from Spark.

[GitBox <gi...@apache.org>]

aokolnychyi edited a comment on issue #63: Use Iceberg writers for Parquet data written from Spark.
URL: https://github.com/apache/incubator-iceberg/pull/63#issuecomment-453615548
 
 
   I have a local benchmark that compares writing 5000000 records in Parquet format using gzip compression via Iceberg and via Spark native file source.
   
   Here are the results:
   ```
   Benchmark                                           Mode  Cnt  Score   Error  Units
   SparkParquetFlatDataWriteBenchmark.writeFileSource    ss    5  4.587 ± 0.226   s/op
   SparkParquetFlatDataWriteBenchmark.writeIceberg       ss    5  4.537 ± 0.090   s/op
   ```
   
   ```
   Benchmark                                             Mode  Cnt  Score   Error  Units
   SparkParquetNestedDataWriteBenchmark.writeFileSource    ss    5  8.798 ± 0.342   s/op
   SparkParquetNestedDataWriteBenchmark.writeIceberg       ss    5  8.457 ± 0.282   s/op
   ```
   
   This PR is also critical as Iceberg will collect proper metadata per file (e.g., min/max values). As for now, we only keep track of the number of rows during writes via Iceberg Spark data source.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org