You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Vadym Dytyniak (Jira)" <ji...@apache.org> on 2022/11/02 13:37:00 UTC
[jira] [Created] (ARROW-18228) AWS Error SLOW_DOWN during PutObject operation
Vadym Dytyniak created ARROW-18228:
--------------------------------------
Summary: AWS Error SLOW_DOWN during PutObject operation
Key: ARROW-18228
URL: https://issues.apache.org/jira/browse/ARROW-18228
Project: Apache Arrow
Issue Type: Bug
Affects Versions: 10.0.0
Reporter: Vadym Dytyniak
We use Dask to parallelise read/write operations and pyarrow to write dataset from worker nodes.
After pyarrow released version 10.0.0, our data flows automatically switched to the latest version and some of them started to fail with the following error:
{code:java}
File "/usr/local/lib/python3.10/dist-packages/org/store/storage.py", line 768, in _write_partition
ds.write_dataset(
File "/usr/local/lib/python3.10/dist-packages/pyarrow/dataset.py", line 988, in write_dataset
_filesystemdataset_write(
File "pyarrow/_dataset.pyx", line 2859, in pyarrow._dataset._filesystemdataset_write
check_status(CFileSystemDataset.Write(c_options, c_scanner))
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
raise IOError(message)
OSError: When creating key 'equities.us.level2.by_security/' in bucket 'org-prod': AWS Error SLOW_DOWN during PutObject operation: Please reduce your request rate. {code}
Do you have any idea what was changed for dataset write between 9.0.0 and 10.0.0 to help us to fix the issue?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)