[jira] [Commented] (AIRFLOW-3419) S3_hook.select_key is broken on Python3

["Kengo Seki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735353#comment-16735353 ] 

Kengo Seki commented on AIRFLOW-3419:
-------------------------------------

I think {{S3Hook.select_key}} should be fixed in the way [~vapira] proposed. As [~ashb] concerns, it introduces an incompatible change that revises the return type from str to unicode on Python2, but in addition to fixing the problem on Python3, it also makes {{S3Hook.select_key}}'s behaviour consistent with the other API such as {{S3Hook.read_key}}.
[~vapira], would you like to submit a PR? If not, I'll do that later.


> S3_hook.select_key is broken on Python3
> ---------------------------------------
>
>                 Key: AIRFLOW-3419
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3419
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: boto3, hooks
>    Affects Versions: 1.10.1
>            Reporter: Maria Rebelka
>            Priority: Major
>
> Hello,
> Using select_key throws an error:
> {quote}text = S3Hook('aws_conn').select_key(key='my_key',
>                                      bucket_name='my_bucket',
>                                      expression='SELECT * FROM S3Object s',
>                                      expression_type='SQL',
>                                      input_serialization={'JSON': \{'Type': 'DOCUMENT'}},
>                                      output_serialization={'JSON': {}}){quote}
> Traceback (most recent call last):
> {quote}   File "db.py", line 31, in <module>
>     output_serialization={'JSON': {}})
>   File "/usr/local/lib/python3.4/site-packages/airflow/hooks/S3_hook.py", line 262, in select_key
>     for event in response['Payload']
> TypeError: sequence item 0: expected str instance, bytes found{quote}
> Seems that the problem is in this line:
> S3_hook.py, line 262:  return ''.join(event['Records']['Payload']
> which probably should be return ''.join(event['Records']['Payload'].decode('utf-8')
> From example in Amazon blog:
> https://aws.amazon.com/blogs/aws/s3-glacier-select/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)