[GitHub] [airflow] josh-fell commented on issue #21765: Unable to pass parameters which aren’t strings while triggering external data in airflow 2.1.4

[GitBox <gi...@apache.org>]

josh-fell commented on issue #21765:
URL: https://github.com/apache/airflow/issues/21765#issuecomment-1049478442


   Starting with Airflow 2.1, there was a new parameter introduced called `render_template_as_native_obj` which will render templated values as native Python types rather than strings or stringified versions of native types. You can check out more info in the [Operator concepts documentation](https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects).
   
   Let's take the below DAGs as an example.
   ```python
   from pendulum import datetime
   from typing import Dict
   
   from airflow.decorators import dag, task
   from airflow.models import DAG
   from airflow.operators.trigger_dagrun import TriggerDagRunOperator
   
   # DAG performing the triggering of another.
   @dag(
       dag_id="parent_dag",
       start_date=datetime(2021, 1, 1),
       schedule_interval=None,
       render_template_as_native_obj=True
   )
   def parent_dag():
       @task
       def xcom_push() -> Dict:
           return {
               "gcs_bucket": "my_bucket",
               "gcs_objects_list": [1, 2, 3],
               "bq_auto_detect_schema": True,
               "bq_schema": None,
           }
   
       t = xcom_push()
   
       trigger_child_dag = TriggerDagRunOperator(
           task_id="trigger_child_dag",
           trigger_dag_id="child_dag",
           conf={
               "gcs_bucket": "{{ ti.xcom_pull(key='gcs_bucket', task_ids='xcom_push') }}",
               "gcs_objects_list": "{{ ti.xcom_pull(key='gcs_objects_list', task_ids='xcom_push') }}",
               "bq_auto_detect_schema": "{{ ti.xcom_pull(key='bq_auto_detect_schema', task_ids='xcom_push') }}",
               "bq_schema": "{{ ti.xcom_pull(key='bq_schema', task_ids='xcom_push') }}",
           },
           retries=1,
       )
   
       t >> trigger_child_dag
   
   
   _parent_dag = parent_dag()
   
   # DAG to be triggered.
   @dag(
       start_date=datetime(2021, 1, 1),
       schedule_interval=None,
   )
   def child_dag():
       @task
       def get_trigger_conf(dag_run=None):
           from pprint import pprint
   
           pprint(dag_run.conf)
   
       t = get_trigger_conf()
   
   
   _child_dag = child_dag()
   ```
   
   When the `child_dag` is triggered, the output printed from the "get_trigger_conf" task is:
   <img width="710" alt="image" src="https://user-images.githubusercontent.com/48934154/155456720-5a4bb9f0-c672-4529-82dd-2a9393318d17.png">
   
   Notice that the values for `gcs_bucket`, `gcs_objects_list`, `bq_auto_detect_schema`, and `bq_schema` are native Python types rather than the stringified versions that you are running into now.
   
   Can you try setting `render_template_as_native_obj=True` on the triggering DAG and see if this helps? I ran the above example on 2.1.4 and using the `main` branch with the same result so hopefully this works for you as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org