You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ash Berlin-Taylor (JIRA)" <ji...@apache.org> on 2019/04/05 17:58:00 UTC
[jira] [Closed] (AIRFLOW-1477) S3KeySensor throws 400 Error
[ https://issues.apache.org/jira/browse/AIRFLOW-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ash Berlin-Taylor closed AIRFLOW-1477.
--------------------------------------
Resolution: Fixed
The code has changed a lot since this ticket was opened. Please reopen/create a new one if this is still a problem for people.
> S3KeySensor throws 400 Error
> ----------------------------
>
> Key: AIRFLOW-1477
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1477
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
> Affects Versions: 1.8.1
> Environment: Ubuntu Linux 16.10
> Reporter: bradford stephens
> Priority: Major
>
> Despite configuring an S3 connection, S3KeySensor throws an error:
> {code:java}
> [2017-07-31 20:06:34,090] {base_task_runner.py:95} INFO - Subtask: [2017-07-31 20:06:34,088] {models.py:1430} ERROR - S3ResponseError: 400 Bad Request
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1387, in run
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: result = task_copy.execute(context=context)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line 76, in execute
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: while not self.poke(context):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line 533, in poke
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: return hook.check_for_key(self.bucket_key, self.bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 262, in check_for_key
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: bucket = self.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 222, in get_bucket
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: return self.connection.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 506, in get_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: return self.head_bucket(bucket_name, headers=headers)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 553, in head_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: response.status, response.reason, body)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: S3ResponseError: S3ResponseError: 400 Bad Request
> {code}
> I have tried both with and without the host parameter in the connection.
> The DAG code includes:
> {code:python}
> inputsensor = S3KeySensor(
> task_id='check_s3_for_file_in_s3',
> bucket_key='test-input-temp.json',
> wildcard_match=False,
> bucket_name='ventera-staging-data',
> s3_conn_id='S3Connection',
> timeout=18*60*60,
> poke_interval=120,
> dag=dag)
> {code}
> The "extra" field in S3Connection has been set with the aws keys.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)