[GitHub] [arrow] krfricke commented on issue #32372: [Python] Python hangs when use pyarrow.fs.copy_files with "used_threads=True"

["krfricke (via GitHub)" <gi...@apache.org>]

krfricke commented on issue #32372:
URL: https://github.com/apache/arrow/issues/32372#issuecomment-1421097792

   We experience the same issuein Ray, and it's easily reproducible. The issue comes up when requesting a recursive upload of more or equal files than CPUs are available.
   
   For instance, on my MacBook with 8 cores, I can upload a folder with 7 files, but not with 8 files:
   
   ```
   mkdir -p /tmp/pa-s3
   cd /tmp/pa-s3 
   for i in {1..7}; do touch $i.txt; done
   # This works
   python -c "import pyarrow.fs; pyarrow.fs.copy_files('/tmp/pa-s3', 's3://bucket/folder')"
   for i in {1..8}; do touch $i.txt; done  
   # This hangs forever
   python -c "import pyarrow.fs; pyarrow.fs.copy_files('/tmp/pa-s3', 's3://bucket/folder')"
   ```
   
   The problem comes up with pyarrow 6-11 and can be avoided with `use_threads=False`, but this obviously harms performance.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org