You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Caleb Overman (Jira)" <ji...@apache.org> on 2021/08/20 16:53:00 UTC
[jira] [Created] (ARROW-13685) [Python] Cannot write dataset to
S3FileSystem if bucket already exists
Caleb Overman created ARROW-13685:
-------------------------------------
Summary: [Python] Cannot write dataset to S3FileSystem if bucket already exists
Key: ARROW-13685
URL: https://issues.apache.org/jira/browse/ARROW-13685
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 5.0.0
Reporter: Caleb Overman
I'm trying to write a parquet file to an existing S3 bucket using the new S3FileSystem interface. However, this is failing with an AWS Access Denied error (I do have necessary access). It appears to be trying to recreate the bucket which already exists.
{code:java}
import numpy as np
import pyarrow as pa
from pyarrow import fs
import pyarrow.dataset as ds
s3 = fs.S3FileSystem(region="us-west-2")
table = pa.table({"a": range(10), "b": np.random.randn(10), "c": [1, 2] * 5})
ds.write_dataset(
table,
"my-bucket/test.parquet",
format="parquet",
filesystem=s3,
){code}
{code:java}
OSError: When creating bucket 'my-bucket': AWS Error [code 15]: Access Denied
{code}
I'm seeing the same behavior using `S3FileSystem.create_dir` when `recursive=True`.
{code:java}
s3.create_dir("my-bucket/test_dir/", recursive=True) # Fails
s3.create_dir("my-bucket/test_dir/", recursive=False) # Succeeds
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)