Pools: Resource Management: How does airflow do it?

[harish singh <ha...@gmail.com>]

Hi Guys,

I am facing this issue I think may be a serious one (or I may be just doing
something totally wrong)

So we use 'pool' in our pipeline.

We have a pool "cpu_pool" with 2 slots.
Now, currently, I see that both the slots are used.
And there are 18 tasks that are Queued for this pool.
We are using localExecutor.
What I am observing is: there is a task, which was queued yesterday. But
has not run yet.

Even currently, when there are 18 queued tasks,  for some reason, the order
in which the tasks are selected from the queue is very random. So one of my
tasks remains to be queued even after more than 24 hours, while similar
tasks that arrived late, go ahead and run.

Is there a fair scheduling strategy, while using localExecutor?

Is there a hook or some way to control the scheduling?

Thanks,
Harish