You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2019/01/23 07:46:00 UTC
[jira] [Commented] (AIRFLOW-2045) S3FileTransformOperator doesn't
adhere to current boto3 API
[ https://issues.apache.org/jira/browse/AIRFLOW-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749630#comment-16749630 ]
jack commented on AIRFLOW-2045:
-------------------------------
For s3 interaction the config requirement is 'boto3>=1.7.0, <1.8.0'
> S3FileTransformOperator doesn't adhere to current boto3 API
> -----------------------------------------------------------
>
> Key: AIRFLOW-2045
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2045
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws, boto3
> Affects Versions: 1.10.0
> Reporter: Zachary Rubenstein
> Priority: Major
>
> When using S3FileTransformOperator with boto3 >= 1.5.0, I get an error:
>
> Traceback (most recent call last):
> File "/home/airflow/data/airflow/.venv/bin/airflow", line 27, in <module>
> args.func(args)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 528, in test
> ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
> result = func(*args, **kwargs)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/models.py", line 1584, in run
> session=session)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
> result = func(*args, **kwargs)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/models.py", line 1493, in _run_raw_task
> result = task_copy.execute(context=context)
> File "/home/airflow/data/airflow/.venv/local/lib/python2.7/site-packages/airflow/operators/s3_file_transform_operator.py", line 86, in execute
> source_s3_key_object.get_contents_to_file(f_source)
> AttributeError: 's3.Object' object has no attribute 'get_contents_to_file'
> I believe the method has been renamed to download_fileobj (http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Object.download_fileobj)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)