Is there a way to balance task scheduling across multiple DAGs?

[Reed Villanueva <rv...@ucera.org>]

Have multiple dags and airflow set to run with SequentialExecutor. Fine
with the fact that only one task will run at a given time with this
execution mode, but want dags that are scheduled at the same time to have
tasks scheduled in a balanced way (currently appears that dag A (which
starts 5min. earlier than dag B) gets more of its tasks scheduled while B
often just waiting around, even after both have been running for some time).

Is there a way to have airflow tasks explicitly scheduled in balanced way
across dags? Any other workaround to this situation (wouldn't want to
implement any concurrency
<https://airflow.apache.org/docs/stable/faq.html#how-can-my-airflow-dag-run-faster>
just
to see the same balancing problem magnified where one dag still gets all
the scheduling, just more of it)?

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Re: Is there a way to balance task scheduling across multiple DAGs?

[Ash Berlin-Taylor <as...@apache.org>]

Using the SequentialExecutor is very strongly discouraged for anything other than "playing around with my first DAG". 

LocalExecutor will run tasks in a pool of processes allowing more than one to run at the same time and is what I would recommend. Switching to this would let you very easily see if it behaves the same. I've not noticed this behaviour with LocalExec.

-ash

> On 17 Dec 2019, at 00:20, Reed Villanueva <rv...@ucera.org> wrote:
> 
> Have multiple dags and airflow set to run with SequentialExecutor. Fine with the fact that only one task will run at a given time with this execution mode, but want dags that are scheduled at the same time to have tasks scheduled in a balanced way (currently appears that dag A (which starts 5min. earlier than dag B) gets more of its tasks scheduled while B often just waiting around, even after both have been running for some time).
> 
> Is there a way to have airflow tasks explicitly scheduled in balanced way across dags? Any other workaround to this situation (wouldn't want to implement any concurrency <https://airflow.apache.org/docs/stable/faq.html#how-can-my-airflow-dag-run-faster> just to see the same balancing problem magnified where one dag still gets all the scheduling, just more of it)?
> 
> 
> This electronic message is intended only for the named 
> recipient, and may contain information that is confidential or 
> privileged. If you are not the intended recipient, you are 
> hereby notified that any disclosure, copying, distribution or 
> use of the contents of this message is strictly prohibited. If 
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the 
> sender at the electronic mail address noted above, and delete 
> and destroy all copies of this message. Thank you.