You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Vadym Dytyniak (Jira)" <ji...@apache.org> on 2022/11/02 13:37:00 UTC

[jira] [Created] (ARROW-18228) AWS Error SLOW_DOWN during PutObject operation

Vadym Dytyniak created ARROW-18228:
--------------------------------------

             Summary: AWS Error SLOW_DOWN during PutObject operation
                 Key: ARROW-18228
                 URL: https://issues.apache.org/jira/browse/ARROW-18228
             Project: Apache Arrow
          Issue Type: Bug
    Affects Versions: 10.0.0
            Reporter: Vadym Dytyniak


We use Dask to parallelise read/write operations and pyarrow to write dataset from worker nodes.

After pyarrow released version 10.0.0, our data flows automatically switched to the latest version and some of them started to fail with the following error:
{code:java}
File "/usr/local/lib/python3.10/dist-packages/org/store/storage.py", line 768, in _write_partition
    ds.write_dataset(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/dataset.py", line 988, in write_dataset
    _filesystemdataset_write(
  File "pyarrow/_dataset.pyx", line 2859, in pyarrow._dataset._filesystemdataset_write
    check_status(CFileSystemDataset.Write(c_options, c_scanner))
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
    raise IOError(message)
OSError: When creating key 'equities.us.level2.by_security/' in bucket 'org-prod': AWS Error SLOW_DOWN during PutObject operation: Please reduce your request rate. {code}
Do you have any idea what was changed for dataset write between 9.0.0 and 10.0.0 to help us to fix the issue?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)