[GitHub] [airflow] kaxil commented on issue #10451: Airflow tries running Jinja on any property name ending with .json

[GitBox <gi...@apache.org>]

kaxil commented on issue #10451:
URL: https://github.com/apache/airflow/issues/10451#issuecomment-678671658


   Airflow tried to renders all the values passed to template_fields. In your case as your are using `EmrAddStepsOperator`, it's template_fields are `['job_flow_id', 'job_flow_name', 'cluster_states', 'steps']` 
   
   
   
   **Source Code**: https://github.com/apache/airflow/blob/47c6657ce012f6db147fdcce3ca5e77f46a9e491/airflow/providers/amazon/aws/operators/emr_add_steps.py#L48
   
   This was added by https://github.com/apache/airflow/pull/8572 
   
   You can fix these in 2 ways:
   
   1) Bypass this by adding an extra space after `.json` example `"s3://dummy/spark/application.json "`. This works because Airflow looks for each element in the Iterable to find if the string ends in `.json`
   
   2) Subclass `EmrAddStepsOperator` and override the `template_ext` field. Example:
   
   ```python
   class FixedEmrAddStepsOperator(BaseOperator):
       template_ext = ()
   ```
   
   and then you can use this Operator:
   
   ```python
       proc_step = FixedEmrAddStepsOperator(
           task_id='process_data',
           job_flow_id="{{ task_instance.xcom_pull(task_ids='launch_emr_cluster', key='return_value') }}",
           aws_conn_id='aws_default',
           steps=STEPS,
       )
   ```
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org