[jira] [Comment Edited] (AIRFLOW-1477) S3KeySensor throws 400 Error

["Michael Crawford (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182700#comment-16182700 ] 

Michael Crawford edited comment on AIRFLOW-1477 at 9/27/17 3:07 PM:
--------------------------------------------------------------------

I have been using the s3keysensor in 1.8.2 with the credentials in the extra field without issue.

I don't think Petros's comment applies as the sensor doesn't download files.


was (Author: dalupus):
I have been using the s3keysensor in 1.8.2 with the credentials in the extra field without issue.

I don't think [~petros]'s comment applies as the sensor doesn't download files.

> S3KeySensor throws 400 Error
> ----------------------------
>
>                 Key: AIRFLOW-1477
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1477
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.8.1
>         Environment: Ubuntu Linux 16.10
>            Reporter: bradford stephens
>
> Despite configuring an S3 connection, S3KeySensor throws an error:
> {code:java}
> [2017-07-31 20:06:34,090] {base_task_runner.py:95} INFO - Subtask: [2017-07-31 20:06:34,088] {models.py:1430} ERROR - S3ResponseError: 400 Bad Request
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1387, in run
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line 76, in execute
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     while not self.poke(context):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line 533, in poke
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return hook.check_for_key(self.bucket_key, self.bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 262, in check_for_key
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     bucket = self.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 222, in get_bucket
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return self.connection.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 506, in get_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     return self.head_bucket(bucket_name, headers=headers)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 553, in head_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     response.status, response.reason, body)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: S3ResponseError: S3ResponseError: 400 Bad Request
> {code}
> I have tried both with and without the host parameter in the connection.
> The DAG code includes:
> {code:python}
> inputsensor = S3KeySensor(
>    task_id='check_s3_for_file_in_s3',
>    bucket_key='test-input-temp.json',
>    wildcard_match=False,
>    bucket_name='ventera-staging-data',
>    s3_conn_id='S3Connection',
>    timeout=18*60*60,
>    poke_interval=120,
>    dag=dag)
> {code}
> The "extra" field in S3Connection has been set with the aws keys.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)