[jira] [Work started] (AIRFLOW-5115) S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables

["Dmitriy Synkov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-5115 started by Dmitriy Synkov.
-----------------------------------------------
> S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja variables
> ---------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5115
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: aws
>    Affects Versions: 1.9.0
>            Reporter: Dmitriy Synkov
>            Assignee: Dmitriy Synkov
>            Priority: Minor
>              Labels: easyfix, patch
>             Fix For: 1.10.4
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> In all Airflow operators (which inherit form {{BaseOperator}}) there is a {{template_fields}} attribute defined as ["which fields will get jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]). For the {{S3KeySensor}} op in specific, these are {{template_fields = ('bucket_key', 'bucket_name')}}.
> The {{bucket_key}} kwarg, however, has some input validation in that the {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this exception is thrown by the [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74], which makes it impossible to use Jinja strings as an arg to {{bucket_key}}, since these don't get rendered in the scope of the DAG {{*.py}} file itself. Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:
> Given the below DAG code, where "my_s3_key" is {{s3://bucket/prefix/object.txt:}}
> {code:java}
> dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))
> s3_variable_sensor = S3KeySensor(
>     task_id='s3_variable_sensor',
>     bucket_key=Variable.get('my_s3_key'),
>     dag=dag
> )
> s3_jinja_sensor = S3KeySensor(
>     task_id='s3_jinja_sensor',
>     bucket_key="{{ var.value.my_s3_key }}",
>     dag=dag
> )
> {code}
> Executing the first task will run just fine while the next task will throw the following exception:
> {code:java}
> airflow.exceptions.AirflowException: Please provide a bucket_name.
> {code}
> This ticket is to propose a code change that will move input validation out of the constructor to allow for Jinja-templated strings to be passed into both {{bucket_name}} and {{bucket_key}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)