You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hanxdatadog (via GitHub)" <gi...@apache.org> on 2025/05/22 19:11:15 UTC

[I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

hanxdatadog opened a new issue, #50982:
URL: https://github.com/apache/airflow/issues/50982

   ### Apache Airflow version
   
   3.0.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When pulling xcom data from dynamic mapped tasks by doing
   ```
   ti.xcom_pull(task_ids="some_task")
   ```
   It used to return all mapped tasks' xcom data in airflow 2. It is still doing that when testing airflow dag using `airflow dags test my_dag`. But when running airflow in production/standalone, it returns None. 
   
   Reading the code, we are using RuntimeTaskInstance when actually deploying Airflow. The execution API XCom API cannot achieve the previous behavior of getting many XCom data from mapped tasks.
   https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L359-L375
   
   
   ### What you think should happen instead?
   
   1. It should restore the behavior from Airflow 2, which returns a list of XCom data from all dynamically mapped tasks. This is quite useful, as dynamically mapped tasks are often based on runtime data, making it difficult to know the number of mapped tasks in advance.
   
   2. If restoring the same behavior is difficult, could we provide a way to retrieve the mapped task count for a given upstream task? This would help in fetching all the XCom data from the mapped tasks later on.
   
   ### How to reproduce
   
   With the following DAG code
   ```
   with DAG(
       "my_dag",
       schedule=timedelta(days=1),
       start_date=datetime(2025, 1, 1),
       catchup=False,
   ) as dag:
   
       def push_data(x):
           return f"foo: {x}"
   
       push_task = PythonOperator.partial(
           task_id="push_task",
           python_callable=push_data,
       ).expand(op_kwargs=[{"x": "a"}, {"x": "b"}])
   
       def pull_data(**kwargs):
           ti = kwargs["ti"]
           print(f"ti_class:{ti.__class__}")
           print(f"ti.map_index:{ti.map_index}")
           data = ti.xcom_pull(task_ids="push_task")
           result = list(data) if data else data
           print(f"pulled data {result}")
           return result
   
       pull_task = PythonOperator(
           task_id="pull_data",
           python_callable=pull_data,
       )
   
       def pull_data_0(**kwargs):
           ti = kwargs["ti"]
           data = ti.xcom_pull(task_ids="push_task", map_indexes=0)
           result = list(data) if data else data
           print(f"pulled data 0 {result}")
           return result
   
       pull_task_explicit = PythonOperator(
           task_id="pull_task_explicit",
           python_callable=pull_data_0,
       )
       push_task >> pull_task >> pull_task_explicit
   ```
   If we run `airflow dags test my_dag`, in the `pull_data` task, it will print
   ```
   pulled data ['data: a', 'data: b']
   ```
   If we deploy airflow (either through k8s or standalone) and trigger the dag, the `pull_data` task will output
   ```
   pulled data None: chan="stdout": source="task"
   ```
   
   ### Operating System
   
   MacOS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

Posted by "hanxdatadog (via GitHub)" <gi...@apache.org>.
hanxdatadog commented on issue #50982:
URL: https://github.com/apache/airflow/issues/50982#issuecomment-3029099343

   Yes. Just confirmed that after pulling the latest main, it worked as expected. Thank you so much!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

Posted by "kevinhongzl (via GitHub)" <gi...@apache.org>.
kevinhongzl commented on issue #50982:
URL: https://github.com/apache/airflow/issues/50982#issuecomment-2971332386

   Hi @hanxdatadog. I couldn't reproduce the output and got `['data: a', 'data: b']` in the task `pull_data`.
   Does running the latest main help?
   
   ![Image](https://github.com/user-attachments/assets/760667e6-95c2-43be-a2d6-f316a3b59fcc)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb closed issue #50982: Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks
URL: https://github.com/apache/airflow/issues/50982


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50982:
URL: https://github.com/apache/airflow/issues/50982#issuecomment-2902288332

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Behavioral change and inconsistencies when performing xcom_pull on dynamically mapped tasks [airflow]

Posted by "kevinhongzl (via GitHub)" <gi...@apache.org>.
kevinhongzl commented on issue #50982:
URL: https://github.com/apache/airflow/issues/50982#issuecomment-3029653277

   @hanxdatadog Thank you! Then we can close this issue :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org