[GitHub] [airflow] paavanb opened a new issue #10571: Allow xcom_pull to return multiple results if task_ids is None

[GitBox <gi...@apache.org>]

paavanb opened a new issue #10571:
URL: https://github.com/apache/airflow/issues/10571


   **Description**
   
   `TaskInstance.xcom_pull` uses `XCom.get_many` in order to return multiple XComs for the given parameters. However, it proceeds to only return the first result if `task_ids` is not a list ([this line](https://github.com/apache/airflow/blob/master/airflow/models/taskinstance.py#L1733)). Unfortunately, this is the case when `task_ids` is unset or set to `None`. This seems to make it impossible to get all XComs for a given key without also knowing exactly which `task_ids` set them. Is this intentional behavior?
   
   Fortunately, I can work around this by writing my own function to call `XCom.get_many` directly, but it's not ideal.
   
   **Use case / motivation**
   
   Suppose there is a DAG in which tasks can declare artifacts that they output; these are all pushed as XCOMs with the same key (e.g., `"output-artifact"`). Future tasks attempting to use these artifacts can check for DAG consistency by pulling all XComs with that key and ensuring that a desired artifact was output by some previous task in the current DAG. But since these tasks are reusable/defined elsewhere, they don't know the `task_ids` of all preceding tasks. If `task_ids` is set to `None`, it seems that `xcom_pull` should return all XComs that match the parameters provided, but that's not true today.
   
   Appreciate any feedback!
   
   **Related Issues**
   
   https://github.com/apache/airflow/pull/6461
   https://issues.apache.org/jira/browse/AIRFLOW-5804


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10571: Allow xcom_pull to return multiple results if task_ids is None

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #10571:
URL: https://github.com/apache/airflow/issues/10571#issuecomment-680308153


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org