You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "atul-astronomer (via GitHub)" <gi...@apache.org> on 2025/02/12 07:23:01 UTC

[I] Not able to pull XCom when passing multiple map_indexes [airflow]

atul-astronomer opened a new issue, #46665:
URL: https://github.com/apache/airflow/issues/46665

   ### Apache Airflow version
   
   3.0.0a1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   While pulling XCom, if user passes multiple map_indexes in argument then task is failing.
   
   ```javascript
   {"timestamp":"2025-02-12T06:55:44.726594","level":"error","event":"Task failed with exception","logger":"task","error_detail":[{"exc_type":"NotImplementedError","exc_value":"Multiple map_indexes are not supported yet","syntax_error":null,"is_cause":false,"frames":[{"filename":"/opt/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":567,"name":"run"},
   {"filename":"/opt/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":671,"name":"_execute_task"},{"filename":"/opt/airflow/airflow/models/baseoperator.py","lineno":173,"name":"wrapper"},{"filename":"/opt/airflow/airflow/decorators/base.py","lineno":252,"name":"execute"},
   {"filename":"/opt/airflow/airflow/models/baseoperator.py","lineno":173,"name":"wrapper"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":196,"name":"execute"},
   {"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":222,"name":"execute_callable"},{"filename":"/opt/airflow/airflow/utils/operator_helpers.py","lineno":261,"name":"run"},{"filename":"/files/dags/taskmap/xcom_pull.py","lineno":49,"name":"xcom_pull"},
   {"filename":"/opt/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":277,"name":"xcom_pull"}]}]}
   {"timestamp":"2025-02-12T06:55:44.726849","level":"debug","event":"Sending request","json":"{\"state\":\"failed\",\"end_date\":\"2025-02-12T06:55:44.726813Z\",\"type\":\"TaskState\"}\n","logger":"task"}
   
   ``` 
   
   ### What you think should happen instead?
   
   DAG should pass as its passing in AF2.
   
   ### How to reproduce
   
   Run the below DAG code on AF3 tag 3.0.0a2 and notice error on task `xcom_pull`
   
   ```python
   from airflow import DAG
   from airflow.models.taskinstance import TaskInstance
   from datetime import datetime, timedelta
   
   with DAG(
       dag_id="taskmap_xcom_pull",
       start_date=datetime(1970, 1, 1),
       schedule=None,
   ) as dag:
   
       @dag.task
       def foo():
           return "foo"
   
       @dag.task
       def identity(thing):
           return thing
   
       @dag.task
       def uppercase(thing):
           return thing.upper()
   
       @dag.task(multiple_outputs=True)
       def two_parts(thing):
           return {"first": thing[0:3], "rest": thing[3:]}
   
       @dag.task
       def xcom_pull(**context):
           ti: TaskInstance = context["ti"]
   
           for x in [
               ti.xcom_pull(task_ids="identity"),
               # <airflow.models.taskinstance._LazyXComAccess object at 0x7f5d39f92430>
               ti.xcom_pull(task_ids="foo"),
               # foo
               ti.xcom_pull(task_ids="bar"),
               # []
               ti.xcom_pull(task_ids=["identity", "foo", "bar", "uppercase", "two_parts"]),
               # [2, 3, 1, 'foo', [], 'A', 'B', 'C', {'first': 'ips', 'rest': 'um'}, {'first': 'lor', 'rest': 'em'}, {'first': 'dol', 'rest': 'or'}]
               ti.xcom_pull(task_ids="identity", map_indexes=2),
               # 3
               ti.xcom_pull(task_ids="foo", map_indexes=0),
               # None
               ti.xcom_pull(task_ids="foo", map_indexes=2),
               # None
               ti.xcom_pull(task_ids=["identity", "plusfive"], map_indexes=2),
               # [3]
               ti.xcom_pull(task_ids=["identity", "foo", "plusfive"], map_indexes=[0, 2]),
               # [1, 3]
               ti.xcom_pull(task_ids=["identity", "plusfive", "two_parts"]),
           ]:
               print(x)
   
       # works
       (
           foo()
           >> identity.expand(thing=[1, 2, 3])
           >> uppercase.expand(thing=["a", "b", "c"])
           >> two_parts.expand(thing=["lorem", "ipsum", "dolor"])
           >> xcom_pull()
       )
   
   ``` 
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to pull XCom when passing multiple map_indexes [airflow]

Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46665:
URL: https://github.com/apache/airflow/issues/46665#issuecomment-2656087236

   I think it is part of AIP-72 https://github.com/orgs/apache/projects/405 
   
   I got the overall idea of the current XCOM pulling state and how the mechanism of multiple map_indexes requests can be implemented.
   
   A direct approach can be very inefficient cause it would lead to task_ids * map_indexes requests to the API. But it is the only supported request variant for map_indexes, we either can request a single map_index, or all of them.
   
   There is also drift in desired behavior from docs and functions arguments:
   ```
           map_indexes: int | Iterable[int] | None | ArgNotSet = NOTSET,
   ...
           When pulling one single task (``task_id`` is *None* or a str) without
           specifying ``map_indexes``, the return value is inferred from whether
           the specified task is mapped. If not, value from the one single task
           instance is returned. If the task to pull is mapped, an iterator (not a
           list) yielding XComs from mapped task instances is returned. In either
           case, ``default`` (*None* if not specified) is returned if no matching
           XComs are found.
   
           When pulling multiple tasks (i.e. either ``task_id`` or ``map_index`` is
           a non-str iterable), a list of matching XComs is returned. Elements in
           the list is ordered by item ordering in ``task_id`` and ``map_index``.
   ```
   
   And the API handler itself distinguishes between None map_index and -1.
   
   
   With whom can I discuss or where can I read about the desired behavior? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to pull XCom when passing multiple map_indexes [airflow]

Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46665:
URL: https://github.com/apache/airflow/issues/46665#issuecomment-2653075360

   I would like to work on this issue if possible!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to pull XCom when passing multiple map_indexes [airflow]

Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #46665:
URL: https://github.com/apache/airflow/issues/46665#issuecomment-2654063536

   Assigned to you @insomnes !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to pull XCom when passing multiple map_indexes [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed issue #46665: Not able to pull XCom when passing multiple map_indexes
URL: https://github.com/apache/airflow/issues/46665


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to pull XCom when passing multiple map_indexes [airflow]

Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46665:
URL: https://github.com/apache/airflow/issues/46665#issuecomment-2657424761

   I've created PR with implementation as I understood it:
   
   https://github.com/apache/airflow/pull/46728


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org