You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "linwanlong (via GitHub)" <gi...@apache.org> on 2025/07/18 14:20:36 UTC

[I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

linwanlong opened a new issue, #53498:
URL: https://github.com/apache/airflow/issues/53498

   ### Apache Airflow version
   
   3.0.3
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   This is my environmental information
   ```text
   Apache Airflow
   version                | 3.0.3                                                         
   executor               | LocalExecutor
   ```
   
   Here's my config information
   ```text
   [core]
   default_timezone = utc
   executor = LocalExecutor
   parallelism = 32
   max_active_tasks_per_dag = 16
   max_active_runs_per_dag = 16
   max_consecutive_failed_dag_runs_per_dag = 0
   ```
   
   Here's my dag file
   ```python
   from airflow.sdk import dag, task
   import time
   from airflow.providers.standard.operators.empty import EmptyOperator
   
   
   @dag(schedule=None, catchup=False, tags=["verification"], max_active_tasks=10)
   def busy_box():
   
       @task
       def task_a():
           print("start task A")
           time.sleep(10)
           print("end task A")
   
       @task
       def task_b():
           print("start task B")
           time.sleep(10)
           print("end task B")
   
       @task
       def task_c():
           print("start task c")
           time.sleep(10)
           print("end task c")
   
       start = EmptyOperator(task_id="start")
       a = task_a()
       b = task_b()
       c = task_c()
       end = EmptyOperator(task_id="end")
       # a >> b  # 串行
       # 或者改成
       start >> [a, b, c] >> end  # 并行
   
   
   busy_box_dag = busy_box()
   
   ```
   
   Present the results
   <img width="2235" height="603" alt="image" src="https://github.com/user-attachments/assets/e41ad052-35c5-49ac-a16a-f1ad83b48376" />
   Task C does not start until after the end of Task B. Sometimes the start time of task A and task B is different (task B starts after task A ends), but this problem is not encountered in Airflow2
   
   
   
   This is about the process of airflow3
   ```text
   root     2655981  0.0  0.2 128088 48592 pts/3    Sl+  21:28   0:00 /root/miniconda3/envs/licloud-node-airflow/bin/python /root/.vscode-server/extensions/ms-python.black-formatter-2025.2.0/bundled/tool/lsp_server.py --stdio
   root     2656190  0.1  0.9 190396 150140 pts/0   S+   21:28   0:02 airflow api_server -- host:0.0.0.0 port:8080
   root     2656199  0.0  0.0  16752 10360 pts/0    S+   21:28   0:00 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 -c from multiprocessing.resource_tracker import main;main(7)
   root     2656200  0.3  1.4 487120 230024 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=10) --multiprocessing-fork
   root     2656201  0.3  1.4 488744 232272 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=14) --multiprocessing-fork
   root     2656202  0.3  1.3 548744 221488 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=18) --multiprocessing-fork
   root     2656203  0.3  1.3 551072 223920 pts/0   Sl+  21:28   0:05 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=8, pipe_handle=24) --multiprocessing-fork
   root     2656706  3.1  1.0 212584 171700 pts/4   S+   21:31   0:46 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 /root/miniconda3/envs/licloud-node-airflow/bin/airflow scheduler
   systemd+ 2656711  1.0  0.1 226844 22560 ?        Ss   21:31   0:15 postgres: postgres airflow 172.20.0.1(55568) idle
   root     2656786  0.8  1.0 279012 166460 pts/6   Sl+  21:31   0:13 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 /root/miniconda3/envs/licloud-node-airflow/bin/airflow dag-processor
   root     2657018  0.0  0.9 204676 147136 pts/4   S+   21:31   0:00 airflow worker -- LocalExecutor: <idle>
   root     2657019  0.0  0.9 205468 147964 pts/4   S+   21:31   0:00 airflow worker -- LocalExecutor: <idle>
   root     2657511  0.7  0.9 200436 159288 pts/8   S+   21:32   0:09 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 /root/miniconda3/envs/licloud-node-airflow/bin/airflow triggerer
   root     2657555  0.1  0.8 198196 135840 pts/8   S+   21:32   0:02 /root/miniconda3/envs/licloud-node-airflow/bin/python3.10 /root/miniconda3/envs/licloud-node-airflow/bin/airflow triggerer
   
   ```
   
   ### What you think should happen instead?
   
   I think it could be an issue with the `airflow scheduler` and that this issue did not appear in airflow2 after migrating to airflow3
   
   ### How to reproduce
   
   Follow the official Quick Start command and reproduce using the dag file provided above
   
   ### Operating System
   
    uname_result(system='Linux', node='DESKTOP-QKK3UA9', release='5.15.167.4-microsoft-standard-WSL2', version='#1 SMP Tue Nov 5 00:21:55 UTC 2024', machine='x86_64')
   
   ### Versions of Apache Airflow Providers
   
   Providers info
   apache-airflow-providers-amazon        | 9.9.0 
   apache-airflow-providers-common-compat | 1.7.2 
   apache-airflow-providers-common-io     | 1.6.1 
   apache-airflow-providers-common-sql    | 1.27.3
   apache-airflow-providers-fab           | 2.3.0 
   apache-airflow-providers-http          | 5.3.2 
   apache-airflow-providers-smtp          | 2.1.1 
   apache-airflow-providers-standard      | 1.4.0
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #53498: Tasks in a DAG cannot be executed in parallel to find a solution
URL: https://github.com/apache/airflow/issues/53498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "yokwejuste (via GitHub)" <gi...@apache.org>.
yokwejuste commented on issue #53498:
URL: https://github.com/apache/airflow/issues/53498#issuecomment-3092687518

   Please kindly check your configurations on the `airflow.cfg` file to be sure of your parallelism configs.
   ```cfg
   parallelism = 32
   
   max_active_tasks_per_dag = 16
   
   dags_are_paused_at_creation = True
   
   max_active_runs_per_dag = 16
   
   max_consecutive_failed_dag_runs_per_dag = 0
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on issue #53498:
URL: https://github.com/apache/airflow/issues/53498#issuecomment-3092715886

   If you’d like to proceed please like this message and I will raise a PR for this issue.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #53498:
URL: https://github.com/apache/airflow/issues/53498#issuecomment-3094389229

   I tried to reproduce it with your DAG and could not.
   
   <img width="1246" height="574" alt="Image" src="https://github.com/user-attachments/assets/65afe2dd-f567-492b-baea-43c71a0d02f0" />
   
   You likely have a problem with configuration of your system. Maybe you have too little resources (memory/cpu) allocated on your WSL2 instance. Or maybe you have some other WSL2-specific limitations (for example filesystem limitations). Note that airflow only works on fully POSIX-compliant installations - including Operating System and filesystems it uses, so certain combination of those factors might have influenced behaviour of your installation as well.
   
   For example if you are using sqlite db and your filesystem is mounted from Windows, I guess this might heavily influence parallelism of database operations.
   
   My suggestion is to give plenty of resources to your WSL2, make absolutely sure that you install and use airflow installed only inside your linux WSL2 including all filesystem (reinstalling things from scratch might help). 
   
   Converting to a discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #53498: Tasks in a DAG cannot be executed in parallel to find a solution
URL: https://github.com/apache/airflow/issues/53498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Tasks in a DAG cannot be executed in parallel to find a solution [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53498:
URL: https://github.com/apache/airflow/issues/53498#issuecomment-3089640469

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org