[jira] [Created] (ARROW-18076) PyArrow cannot read from R2 (Cloudflare's S3)

["Vedant Roy (Jira)" <ji...@apache.org>]

Vedant Roy created ARROW-18076:
----------------------------------

             Summary: PyArrow cannot read from R2 (Cloudflare's S3)
                 Key: ARROW-18076
                 URL: https://issues.apache.org/jira/browse/ARROW-18076
             Project: Apache Arrow
          Issue Type: Bug
         Environment: Ubuntu 20
            Reporter: Vedant Roy


When using pyarrow to read parquet data (as part of the Ray project), I get the following stracktrace:

```
(_sample_piece pid=49818) Traceback (most recent call last):
(_sample_piece pid=49818)   File "python/ray/_raylet.pyx", line 859, in ray._raylet.execute_task
(_sample_piece pid=49818)   File "python/ray/_raylet.pyx", line 863, in ray._raylet.execute_task
(_sample_piece pid=49818)   File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/data/datasource/parquet_datasource.py", line 446, in _sample_piece
(_sample_piece pid=49818)     batch = next(batches)
(_sample_piece pid=49818)   File "pyarrow/_dataset.pyx", line 3202, in _iterator
(_sample_piece pid=49818)   File "pyarrow/_dataset.pyx", line 2891, in pyarrow._dataset.TaggedRecordBatchIterator.__next__
(_sample_piece pid=49818)   File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
(_sample_piece pid=49818)   File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
(_sample_piece pid=49818) OSError: AWS Error [code 99]: curlCode: 18, Transferred a partial file
```

I do not get this error when using Amazon S3 for the exact same data.
The error is coming from this line:
https://github.com/ray-project/ray/blob/6fb605379a726d889bd25cf0ee4ed335c74408ff/python/ray/data/datasource/parquet_datasource.py#L446



--
This message was sent by Atlassian Jira
(v8.20.10#820010)