[GitHub] [iceberg] atreyas commented on pull request #7318: [AWS] S3AsyncFileIO Client integration

["atreyas (via GitHub)" <gi...@apache.org>]

atreyas commented on PR #7318:
URL: https://github.com/apache/iceberg/pull/7318#issuecomment-1559586373

   I ran some tests against an aritificial dataset of really small files that originally lead me to dive into the Async client. I found a couple of interesting results.
   (The dataset is based on 30k small parquet files ~40KB each derived from [Amazon Book reviews dataset](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) )
   
   1. The async client consistently runs about 20% faster for compacting this dataset with default configuration
   2. The async client is also better at internally handling failures, so I see fewer task reruns, and the execution times are more stable/consistent than the older client.
   
   There is still some work required to run this against a standard datasets for validation, and potentially some tuning on the HTTP client choice/configurations. But at this point, it seems to overall take fewer system resources, and shows no downsides.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org