Scheduler High Availability & performances

[Nicolas Paris <ni...@riseup.net>]

hi

the HA RFC[1] had two objectives:

1. HA
2. performances scalability


Can anyone confirm that adding multiple schedulers can improve
performances in the case people having HUGE number of dags (> 500) ?


Our finding is that increasing the ressource on a standalone scheduler
performs better than adding scaling them horizontally (partly due to
database impact of locks)

Thanks


[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651

Re: Scheduler High Availability & performances

[Jarek Potiuk <ja...@potiuk.com>]

Yeah, I understand you'd like to save time for benchmarking. That's an
expectation I can sympathise with. I just think the number of
variables is much greater. Good Luck - with both getting the number
and trusting that the answers cover your case :).

On Sat, Nov 20, 2021 at 10:08 PM Nicolas Paris <ni...@riseup.net> wrote:
>
> Thanks for the interessting resources.
>
>
> > I believe it is impossible to get a straight answer from anyone,
>
> agreed real life testing is mandatory - like any informatic project
> isn't ? That being said, I bet there is a finite category on how using
> airflow, such :
>
> - high ressource consuming tasks (python computing operators...)
> - low ressource consuming tasks (extenal sql queries, api call...)
> - hybrid ressource consuming tasks
> - long running dags
> - short running dags
> - many dags / tasks
> - few dags / tasks
>
> For each of those category, general advice could be given, such :
>
> - kubernetes is great for few long running hybrid tasks
> - celery is great for many short running tasks
>
>
> In any cases, multiple schedulers are great for HA. I just wonder if
> anyone here has seen improvements by using multiple scheds in the case
> of :
>
> - high numbers of dags
> - low ressource consuming tasks
>
> This would save me benchmarking this, and BTW, if I eventually do
> benchmarks, I believe sharing the result will be useful for people.

Re: Scheduler High Availability & performances

[Nicolas Paris <ni...@riseup.net>]

Thanks for the interessting resources.


> I believe it is impossible to get a straight answer from anyone,

agreed real life testing is mandatory - like any informatic project
isn't ? That being said, I bet there is a finite category on how using
airflow, such :

- high ressource consuming tasks (python computing operators...)
- low ressource consuming tasks (extenal sql queries, api call...)
- hybrid ressource consuming tasks
- long running dags 
- short running dags
- many dags / tasks
- few dags / tasks

For each of those category, general advice could be given, such :

- kubernetes is great for few long running hybrid tasks
- celery is great for many short running tasks


In any cases, multiple schedulers are great for HA. I just wonder if
anyone here has seen improvements by using multiple scheds in the case
of :

- high numbers of dags
- low ressource consuming tasks

This would save me benchmarking this, and BTW, if I eventually do
benchmarks, I believe sharing the result will be useful for people.

Re: Scheduler High Availability & performances

[Jarek Potiuk <ja...@potiuk.com>]

It really depends on the scenario.

I believe it is impossible to get a straight answer from anyone,
simply because there isn't a "general" answer to that. You really have
to test it and see what works for you.

You can find for example this blog post describing how it behaved in
some scenarios: https://www.astronomer.io/blog/airflow-2-scheduler
And you can see this doc about fine-tuning the performance:
https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html#fine-tuning-your-scheduler-performance

As you will see in the docs - there are so many variables (starting
from filesystem, database choice, database performance and
optimisation, latency -  ending at the structure of your DAGs, how
many dag files vs. DAG you have but most of all the way they are
written and optimised) that the answer is pretty much always "it
depends".

You need to look at your case, verify, experiment and see what works
best for your case, identify bottlenecks you have,  look at the "fine
tuning" doc and knowing what your bottleneck you can fine-tune your
performance (but it will always be "your deployment" characteristics
which can be different from "others")  - the process of fine-tuning is
explained in the doc I linked.

J..

On Fri, Nov 19, 2021 at 5:27 PM Nicolas Paris <ni...@riseup.net> wrote:
>
> hi
>
> the HA RFC[1] had two objectives:
>
> 1. HA
> 2. performances scalability
>
>
> Can anyone confirm that adding multiple schedulers can improve
> performances in the case people having HUGE number of dags (> 500) ?
>
>
> Our finding is that increasing the ressource on a standalone scheduler
> performs better than adding scaling them horizontally (partly due to
> database impact of locks)
>
> Thanks
>
>
> [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651