You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "DonHaul (via GitHub)" <gi...@apache.org> on 2025/10/21 14:10:09 UTC

[I] 'aiflow tasks tests' for mapped tasks with dependencies [airflow]

DonHaul opened a new issue, #56953:
URL: https://github.com/apache/airflow/issues/56953

   ### Apache Airflow version
   
   Other Airflow 2/3 version (please specify below)
   
   ### If "Other Airflow 2/3 version" selected, which one?
   
   3.0.6
   
   ### What happened?
   
   
   I have a simple dag where one map task depends on a single task:
   
   when running `airlfow task tests`:
   ```bash
   airflow tasks test problem_dag multiply_by_two  --map-index 0  -t '{"number":2}'
   ```
   I get an error saying that  the xcoms for `get_numbers` is not found so its defaulting to None, even thought I am testing `multiply_by_two` and provide params for it.
   ```bash
   2025-10-21 13:51:54 [error    ] XCom not found                 [airflow.sdk.api.client] dag_id=problem_dag detail={'detail': {'reason': 'not_found', 'message': "XCom with key='return_value' map_index=-1 not found for task 'get_numbers' in DAG run '__airflow_temporary_run_2025-10-21T13:51:52.789707+00:00__' of 'problem_dag'"}} key=return_value map_index=None run_id=__airflow_temporary_run_2025-10-21T13:51:52.789707+00:00__ status_code=404 task_id=get_numbers
   2025-10-21 13:51:54 [warning  ] No XCom value found; defaulting to None. [task] dag_id=problem_dag key=return_value map_index=None run_id=__airflow_temporary_run_2025-10-21T13:51:52.789707+00:00__ task_id=get_numbers
   ```
   which than fails 
   
   ```bash
   │ /home/airflow/.local/lib/python3.11/site-packages/airflow/sdk/definitions/xcom_arg.py:596 in _   │
   │                                                                                                  │
   │   593 │   if xcom_arg.operator.is_mapped:                                                        │
   │   594 │   │   # TODO: How to tell if all the upstream TIs finished?                              │
   │   595 │   │   pass                                                                               │
   │ ❱ 596 │   return (upstream_map_indexes.get(task_id) or 1) * len(resolved_val)     
   ```
   with
   ```
   TypeError: object of type 'NoneType' has no len()
   ```
   
   The full logs can be found here:
   [log.txt](https://github.com/user-attachments/files/23022615/log.txt)
   
   
   
   ### What you think should happen instead?
   
   When running
   
   ```bash
   airflow tasks test problem_dag multiply_by_two  --map-index 0  -t '{"number":2}'
   ```
   Since all the parameters required , it should either:
   - not try to fetch the xcoms from `get_numbers()`
   - run `get_numbers()` before it fetches its `xcoms`
   
   ### How to reproduce
   
   Run
   
   ```bash
   airflow tasks test problem_dag multiply_by_two  --map-index 0  -t '{"number":2}'
   ```
   
   for dag
   ```python
   from airflow.decorators import dag, task
   
   @dag
   def problem_dag():
       @task
       def get_numbers(**context):
           return [1,2]
   
       @task
       def multiply_by_two(number, **context):
           return number * 2
   
   
       numbers = get_numbers()
       multiplied_numbers = multiply_by_two.expand(number=numbers)
   
   problem_dag()
   ```
   
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 'airflow tasks tests' for mapped tasks with dependencies [airflow]

Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on issue #56953:
URL: https://github.com/apache/airflow/issues/56953#issuecomment-3806408995

   Wow, do you know which PR fixed it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 'airflow tasks tests' for mapped tasks with dependencies [airflow]

Posted by "DonHaul (via GitHub)" <gi...@apache.org>.
DonHaul commented on issue #56953:
URL: https://github.com/apache/airflow/issues/56953#issuecomment-3665382364

   Hey thanks for the workaround. Although I would say Imo we should have to add this
   ```
           if context["params"].get("number") is not None:
               number = context["params"]["number"]
   ```
   just so we can test it via the cli


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 'airflow tasks tests' for mapped tasks with dependencies [airflow]

Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on issue #56953:
URL: https://github.com/apache/airflow/issues/56953#issuecomment-3500712646

   Hi, I think you need to use `context` if you want to use `--task-params`
   
   e.g.
   ```python
   from airflow.decorators import dag, task
   
   @dag
   def problem_dag():
       @task
       def get_numbers(**context):
           return [1,2]
   
       @task
       def multiply_by_two(number, **context):
           if context["params"].get("number") is not None:
               number = context["params"]["number"]
           return number * 2
   
   
       numbers = get_numbers()
       multiplied_numbers = multiply_by_two.expand(number=numbers)
   
   problem_dag()
   ```
   
   result 1
   ```console
   root@9afb732f3cd7:/opt/airflow# airflow tasks test problem_dag multiply_by_two  --map-index 0  -t '{"number":2}'
   
   2025-11-07T04:51:08.515952Z [info     ] Done. Returned value was: 4    [airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator] loc=python.py:218
   ```
   
   result 2
   ```console
   root@9afb732f3cd7:/opt/airflow# airflow tasks test problem_dag multiply_by_two  --map-index 0  -t '{"number":1}'
   
   2025-11-07T04:52:28.369291Z [info     ] Done. Returned value was: 2    [airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator] loc=python.py:218
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 'airflow tasks tests' for mapped tasks with dependencies [airflow]

Posted by "DonHaul (via GitHub)" <gi...@apache.org>.
DonHaul commented on issue #56953:
URL: https://github.com/apache/airflow/issues/56953#issuecomment-3809743103

   Nevermind, I got it confused with another issue, indeed this one will most likely still occur


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 'airflow tasks tests' for mapped tasks with dependencies [airflow]

Posted by "DonHaul (via GitHub)" <gi...@apache.org>.
DonHaul commented on issue #56953:
URL: https://github.com/apache/airflow/issues/56953#issuecomment-3805202119

   Fixed in 3.2 To be checked once its released


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org