[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance

["Fokko (via GitHub)" <gi...@apache.org>]

Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1404692576

   Looking at the recent improvements by @rdblue 
   
   Before 887 requests, after the change 203 requests, which is a great improvement, but still more to be done!
   
   After:
   ```
   ➜  ~ time python3 /tmp/vo.py
   python3 /tmp/vo.py  2.10s user 2.02s system 77% cpu 5.294 total
   python3 /tmp/vo.py  2.56s user 1.27s system 68% cpu 5.558 total
   python3 /tmp/vo.py  1.96s user 1.94s system 68% cpu 5.729 total
   ```
   
   Before:
   ```
   python3 /tmp/vo.py  2.15s user 1.86s system 64% cpu 6.232 total
   python3 /tmp/vo.py  2.61s user 1.53s system 65% cpu 6.315 total
   python3 /tmp/vo.py  2.48s user 1.58s system 66% cpu 6.081 total
   ```
   
   Keep in mind that I'm testing against a local minio, so when using high-latency object storage such as s3 will make it much worse.
   
   Before: https://gist.github.com/Fokko/495dce8234148be74eb312d1af93e694
   After: https://gist.github.com/Fokko/a0cc4ac4ec6a7701a145a4a3da3a6bc6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org