[GitHub] [airflow] rsibanez89 commented on issue #10451: Airflow tries running Jinja on any property name ending with .json

[GitBox <gi...@apache.org>]

rsibanez89 commented on issue #10451:
URL: https://github.com/apache/airflow/issues/10451#issuecomment-1081354542


   > Airflow tried to renders all the values passed to template_fields. In your case as your are using `EmrAddStepsOperator`, it's template_fields are `['job_flow_id', 'job_flow_name', 'cluster_states', 'steps']`
   > 
   > **Source Code**:
   > 
   > https://github.com/apache/airflow/blob/47c6657ce012f6db147fdcce3ca5e77f46a9e491/airflow/providers/amazon/aws/operators/emr_add_steps.py#L48
   > 
   > This was added by #8572
   > 
   > You can fix these in 2 ways:
   > 
   > 1. Bypass this by adding an extra space after `.json` example `"s3://dummy/spark/application.json "`. This works because Airflow looks for each element in the Iterable to find if the string ends in `.json`
   > 2. Subclass `EmrAddStepsOperator` and override the `template_ext` field. Example:
   > 
   > ```python
   > class FixedEmrAddStepsOperator(EmrAddStepsOperator):
   >     template_ext = ()
   > ```
   > 
   > and then you can use this Operator:
   > 
   > ```python
   >     proc_step = FixedEmrAddStepsOperator(
   >         task_id='process_data',
   >         job_flow_id="{{ task_instance.xcom_pull(task_ids='launch_emr_cluster', key='return_value') }}",
   >         aws_conn_id='aws_default',
   >         steps=STEPS,
   >     )
   > ```
   
   @kaxil this solution works but seems like a hack to me. Is there any plan to solve this properly?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org