[jira] [Comment Edited] (ARROW-5072) [Python] write_table fails silently on S3 errors

["Krisztian Szucs (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931542#comment-16931542 ] 

Krisztian Szucs edited comment on ARROW-5072 at 9/17/19 2:58 PM:
-----------------------------------------------------------------

The root cause originates from s3fs. {{S3File}} uses buffering, and no S3 request happens until it is closed or flushed:

{code:python}
out = S3File(S3FileSystem(), 's3://some-bogus-bucket/df.parquet', mode='wb')
out.write(b'bbb')  # returns with 3
del out  # raises exception 
{code}

If we would call {{out.close()}} then it would be flushed, but the file itself is opened outside of arrow, so the caller should be responsible for closing it. We could fall {{flush()}} on file-like objects, although it works a bit differently with {{S3File}}:

{code:python}
out.flush()  # doesn't raise
out.flush(force=True)  # raises, but force is S3File - fsspec specific
{code}

When {{S3FileSystem}} is used, then the file object is opened and thus closed by arrow, so the exception propagates from {{pq.write_table()}}:

{code:python}
pq.write_table(table, s3_filepath, filesystem=S3FileSystem())
{code}

I'd consider this as an s3fs issue, because {{S3File().write()}} work on non-existing file (actually I get a non-authorised error, because I don't have S3 credentials set up), and the error is raised on object destruction, from {{S3File.__del__()}}. 

From arrow perspective the provided behaviour is the expected, although we can have a note for this s3fs case in the documentation.

cc [~wesmckinn] [~pitrou]




was (Author: kszucs):
The root cause originates from s3fs. {{S3File}} uses buffering, and no S3 request happens until it is closed or flushed:

{code:python}
out = S3File(S3FileSystem(), 's3://some-bogus-bucket/df.parquet', mode='wb')
out.write(b'bbb')  # returns with 3
del out  # raises exception 
{code}

If we would call {out.close()} then it would be flushed, but the file itself is opened outside of arrow, so the caller should be responsible for closing it. We could fall {flush()} on file-like objects, although it works a bit differently with {S3File}:

{code:python}
out.flush()  # doesn't raise
out.flush(force=True)  # raises, but force is S3File - fsspec specific
{code}

When {S3FileSystem} is used, then the file object is opened and thus closed by arrow, so the exception propagates from {pq.write_table()}:

{code:python}
pq.write_table(table, s3_filepath, filesystem=S3FileSystem())
{code}

I'd consider this as an s3fs issue, because {S3File().write()} work on non-existing file (actually I get a non-authorised error, because I don't have S3 credentials set up), and the error is raised on object destruction, from {S3File.__del__()}. 

From arrow perspective the provided behaviour is the expected, although we can have a note for this s3fs case in the documentation.

cc [~wesmckinn] [~pitrou]



> [Python] write_table fails silently on S3 errors
> ------------------------------------------------
>
>                 Key: ARROW-5072
>                 URL: https://issues.apache.org/jira/browse/ARROW-5072
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.12.1
>         Environment: Python 3.6.8
>            Reporter: Paul George
>            Priority: Minor
>              Labels: filesystem, parquet
>             Fix For: 0.15.0
>
>
> {{pyarrow==0.12.1}}
> *pyarrow.parquet.write_table* called with where=S3File(...) fails silently when encountering errors while writing to S3 (in the example below, boto3 is raising a NoSuchBucket exception). However, instead of using S3File(), calling write_table with where=_<filepath>_ and with filesystem=S3FileSystem() does *not* fail silently and raises, as is expected.
> h4. Code/Repro
>  
> {code:java}
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> from s3fs import S3File, S3FileSystem
> df = pd.DataFrame({'col0': []})
> s3_filepath = 's3://some-bogus-bucket/df.parquet'
> print('>> test 1')
> try:
> # use S3File --> fails silently
> pq.write_table(pa.Table.from_pandas(df.copy()),
> S3File(S3FileSystem(), s3_filepath, mode='wb'))
> except Exception:
> print('>>>> Exception raised!')
> else:
> print('>>>> Exception **NOT** raised!')
> print('>> test 2')
> try:
> # use filepath and S3FileSystem --> raises Exception, as expected
> pq.write_table(pa.Table.from_pandas(df.copy()),
> s3_filepath,
> filesystem=S3FileSystem())
> except Exception:
> print('>>>> Exception raised!')
> else:
> print('>>>> Exception **NOT** raised!'){code}
>  
> h4.  
> h4. Output
> {code:java}
> >> test 1
> Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>>
> Traceback (most recent call last):
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__
> self.close()
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close
> raise_from(IOError('Write failed: %s' % self.path), e)
> File "<string>", line 3, in raise_from
> OSError: Write failed: some-bogus-bucket/df.parquet
> >>>> Exception **NOT** raised!
> >> test 2
> >>>> Exception raised!
> Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>>
> Traceback (most recent call last):
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__
> self.close()
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close
> raise_from(IOError('Write failed: %s' % self.path), e)
> File "<string>", line 3, in raise_from
> OSError: Write failed: some-bogus-bucket/df.parquet
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)