[jira] [Created] (AIRFLOW-5115) S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables

["Dmitriy Synkov (JIRA)" <ji...@apache.org>]

Dmitriy Synkov created AIRFLOW-5115:
---------------------------------------

             Summary: S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables
                 Key: AIRFLOW-5115
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
             Project: Apache Airflow
          Issue Type: Bug
          Components: aws
    Affects Versions: 1.9.0
            Reporter: Dmitriy Synkov
            Assignee: Dmitriy Synkov
             Fix For: 1.10.4


In all Airflow operators (which inherit form {{BaseOperator}}) there is a {{template_fields}} attribute defined as ["which fields will get jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]). For the {{S3KeySensor}} op in specific, these are {{template_fields = ('bucket_key', 'bucket_name')}}. The {{bucket_key}} kwarg, however, has some input validation in that the {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this is exception is thrown by the [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74], which makes it impossible to use Jinja strings as an arg to {{bucket_key}}, since these don't get rendered in the scope of the DAG itself. Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:

Given the below DAG code, where "my_s3_key" is {{s3://bucket/prefix/object.txt:}}
{code:java}
dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))

s3_variable_sensor = S3KeySensor(
    task_id='s3_variable_sensor',
    bucket_key=Variable.get('my_s3_key'),
    dag=dag
)

s3_jinja_sensor = S3KeySensor(
    task_id='s3_jinja_sensor',
    bucket_key="{{ var.value.my_s3_key }}",
    dag=dag
)
{code}
Executing the first task will run just fine (or even just building the operator in the shell) while the next task will throw the following exception:
{code:java}
airflow.exceptions.AirflowException: Please provide a bucket_name.
{code}
This ticket is to propose a code change that will move input validation out of the constructor to allow for Jinja-templated strings to be passed into both {{bucket_name}} and {{bucket_key}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)