[jira] [Work started] (AIRFLOW-1023) Upload file to S3 using S3 hook fails with "Connection reset by peer"

["Adrian Partl (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-1023 started by Adrian Partl.
---------------------------------------------
> Upload file to S3 using S3 hook fails with "Connection reset by peer"
> ---------------------------------------------------------------------
>
>                 Key: AIRFLOW-1023
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1023
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: Airflow 1.7.1
>            Reporter: Adrian Partl
>            Assignee: Adrian Partl
>
> Using the S3 hook to upload local files to an S3 bucket fails with 
> {noformat}
>   File "/usr/lib/python2.7/site-packages/airflow/hooks/S3_hook.py", line 364, in load_file
>     replace=replace)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1362, in set_contents_from_filename
>     encrypt_key=encrypt_key)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
>     chunked_transfer=chunked_transfer, size=size)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
>     chunked_transfer=chunked_transfer, size=size)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
>     query_args=query_args
>   File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
>     retry_handler=retry_handler
>   File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
>     retry_handler=retry_handler)
>   File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
>     raise ex
> error: [Errno 104] Connection reset by peer
> {noformat}
> This is a known issue with boto and only affects uploads to S3 buckets outside of the standard US location (in my case {{eu-west-1}}).
> The issue is reported on boto side as:
> https://github.com/boto/boto/issues/2207
> A work around is mentioned by user {{anna-buttfield-sirca}} which basically reconnects the boto S3 connection to the corresponding location.
> I will provide a PR implementing the work around, since a resolution of the issue on the boto side seems unlikely.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)