You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "matrach (via GitHub)" <gi...@apache.org> on 2025/01/23 22:52:16 UTC
[I] Performance collapse with large mapped task instances [airflow]
matrach opened a new issue, #45991:
URL: https://github.com/apache/airflow/issues/45991
### Apache Airflow version
2.10.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Hi! I'm still evaluating if Airflow is suitable for our needs, and it seems it is experiencing hiccups when used with bigger mapped task instances.
Even for the minimized example (see below), that produces the following DAG structure, the performance of the system (effective parallelism) drops significantly when switching from 100x to 500x task Dynamic Task Mapping size:
Here, one `group` of tasks sleeps for 6s on average:
- 100x case takes ~70s -> effective parallelism is ~9
- 500x case takes ~560s -> total effective parallelism is ~5 (it catches up later in the run, so the value is overstated)
### What you think should happen instead?
Debug messages reveal, that the scheduler spends most time in evaluating task instance dependencies of the DagRun in
[TaskInstance.are_dependencies_met](https://github.com/apache/airflow/blob/2.10.4/airflow/models/taskinstance.py#L2603) (being CPU-bound, not by the DB), scheduling only one batch (16 by default) per 8s loop:
```
{scheduler_job_runner.py:2069} DEBUG - Finding 'running' jobs without a recent heartbeat
{event_scheduler.py:39} DEBUG - Calling <bound method SchedulerJobRunner._emit_pool_metrics of <airflow.jobs.scheduler_job_runner.SchedulerJobRunner object at 0x75ee2ad3d0a0>>
{scheduler_job_runner.py:1174} DEBUG - Next timed event is in 4.999933
{scheduler_job_runner.py:1176} DEBUG - Ran scheduling loop in 8.49 seconds
```
In particular, the 100x case has most of the task instances scheduled by the mini-scheduler, while initially none of the 500x run.
In that case, the mini-scheduler is skipped due to [locked rows](https://github.com/apache/airflow/blob/2.10.4/airflow/models/taskinstance.py#L3859), but it wasn't immediately clear to me where the lock happens in the scheduler.
Are there any architectural limitations preventing Airflow from managing thousands of fairly short-living tasks? The scheduler seems to spend excessively long time in evaluating the DagRun while holding database locks.
### How to reproduce
Use the docker-compose quick-start with the following DAG:
<details><summary>Source code</summary>
```python
from time import sleep
from airflow.models.dag import DAG
from airflow.decorators import task, task_group
import numpy as np
with DAG(
"minimal",
default_args={
# We want to maximize use of the limited hardware
'weight_rule': "upstream",
},
schedule=None,
start_date=None,
catchup=False,
tags=["bugs"],
) as dag:
@task
def generator():
return list(range(500))
@task
def part1(x):
rng = np.random.default_rng()
#assert rng.random() > 0.001
sleep(1 + rng.random())
return str(x)
@task
def part2():
sleep(0.5)
return "s"
@task
def combiner(part1, part2):
rng = np.random.default_rng()
#assert rng.random() > 0.001
sleep(1 + rng.random())
return part1 + part2
@task
def reporter(x):
print(x)
@task # pool = 'limited_hardware'
def evaluate(what):
#assert np.random.default_rng().random() > 0.001
sleep(3)
return what, len(what)
@task_group
def group(num, part2):
s1 = part1(num)
comb = combiner(s1, part2)
rep = reporter(comb)
res = evaluate(comb)
return res, comb, s1, rep
@task(trigger_rule="one_failed", priority_weight=1000,)
def skipper(ti):
assert False # this normally skips all unfinished tasks
@task(trigger_rule="all_done",)
def collect(ins):
print(list(ins))
t1 = generator()
s = skipper()
parts = group.partial(part2=part2()).expand(num=t1)
parts >> s
collect(ins=parts[0])
```
</details>
### Operating System
Ubuntu 24.04.1 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Vanilla docker-compose setup from docs. 20 core Xeon W-1290 CPU.
### Anything else?
I've noticed the mini-scheduler is already removed on the development branch, but it was the one saving performance in the 100x case...
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler performance with large number of mapped task instances [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #45991: Scheduler performance with large number of mapped task instances
URL: https://github.com/apache/airflow/issues/45991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler performance with large number of mapped task instances [airflow]
Posted by "matrach (via GitHub)" <gi...@apache.org>.
matrach commented on issue #45991:
URL: https://github.com/apache/airflow/issues/45991#issuecomment-2611334138
I've mentioned the mini-scheduler above, but if I understand correctly, it resolved the scheduling state for ALL task instances, but then ignored the ones not downstream...?
It appears that the scheduler for mapped tasks takes at least quadratic time in their quantity, as [here](https://github.com/apache/airflow/blob/main/airflow/ti_deps/deps/trigger_rule_dep.py#L255) it loops over all finished task instances, and does so for all schedulable task instances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Performance collapse with large mapped task instances [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #45991:
URL: https://github.com/apache/airflow/issues/45991#issuecomment-2611178226
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org