[I] [Python] to_table() call on dataset gets stuck and ignores timeout threshold [arrow]

["Heraldk (via GitHub)" <gi...@apache.org>]

Heraldk opened a new issue, #40019:
URL: https://github.com/apache/arrow/issues/40019

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   We're seeing an intermittent behaviour where a call to the `to_table()` method can get stuck when being called on a dataset stored in S3. The stuck call doesn't return before we give up after a couple of hours. But very often a subsequent request succeeds on the same dataset.
   
   We have a python stack trace pointing at that specific method as being where its getting stuck, but we haven't got a way to see where in the C++ code it is stuck.
   
   Our system has a couple of mechanisms to recover from processes that aren't making progress in the expected timeframe. The preferred and first attempt issues a sigalarm to interrupt the process. However the C-extension appears to be blocking this approach and isn't working. We revert then to an alternative approach that kills the container the process is running in.
   
   It's not totally clear where the problem here lies, although a likely candidate is attempting to fetch the dataset from S3. However if this is the cause of the problem, we are confused why the request to S3 isn't timing out after a few minutes.
   
   Version info:
   pyarrow version 14.0.2
   python 3.11
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org