You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Sahib Aulakh <sa...@liveearth.com> on 2023/01/24 19:09:28 UTC

Question on replicas for Web Server, Scheduler and Triggerer

We are running Airflow on Kubernetes.


   1. What is the advantage of setting the replicas property to a value
   greater than 1 for the web server, scheduler and triggerrer?
   2. If this value is set to > 1, then multiple scheduler pods get
   launched. Is this done for more system throughput? How do the schedulers
   coordinate with each other as to which dag to schedule, etc?
   3. Same question for triggerer.
   4. If running in Kubernetes and the replicas are each set to 1, then K8
   will restart the process whenever any pod goes down for any reason. Thus a
   value > 1 is *not *strictly needed for availability. Please confirm.

Thanks.

Re: Question on replicas for Web Server, Scheduler and Triggerer

Posted by Sahib Aulakh <sa...@liveearth.com>.
Many thanks.

On Tue, Jan 24, 2023 at 1:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> 1. scalability
> 2. You can watch this video - it answers all the questions about the
> scheduler you might have: https://www.youtube.com/watch?v=DYC4-xElccE.
> 3. Triggerrer follows the same principles as scheduler.
> 4. Depends what you mean by availability. But your observation is
> correct. However you can do rolling restarts if you want when replicas
> are > 1 (same as any other apps with replicas).
>
> J,
>
> On Tue, Jan 24, 2023 at 8:09 PM Sahib Aulakh <sa...@liveearth.com>
> wrote:
> >
> > We are running Airflow on Kubernetes.
> >
> > What is the advantage of setting the replicas property to a value
> greater than 1 for the web server, scheduler and triggerrer?
> > If this value is set to > 1, then multiple scheduler pods get launched.
> Is this done for more system throughput? How do the schedulers coordinate
> with each other as to which dag to schedule, etc?
> > Same question for triggerer.
> > If running in Kubernetes and the replicas are each set to 1, then K8
> will restart the process whenever any pod goes down for any reason. Thus a
> value > 1 is not strictly needed for availability. Please confirm.
> >
> > Thanks.
>

Re: Question on replicas for Web Server, Scheduler and Triggerer

Posted by Jarek Potiuk <ja...@potiuk.com>.
1. scalability
2. You can watch this video - it answers all the questions about the
scheduler you might have: https://www.youtube.com/watch?v=DYC4-xElccE.
3. Triggerrer follows the same principles as scheduler.
4. Depends what you mean by availability. But your observation is
correct. However you can do rolling restarts if you want when replicas
are > 1 (same as any other apps with replicas).

J,

On Tue, Jan 24, 2023 at 8:09 PM Sahib Aulakh <sa...@liveearth.com> wrote:
>
> We are running Airflow on Kubernetes.
>
> What is the advantage of setting the replicas property to a value greater than 1 for the web server, scheduler and triggerrer?
> If this value is set to > 1, then multiple scheduler pods get launched. Is this done for more system throughput? How do the schedulers coordinate with each other as to which dag to schedule, etc?
> Same question for triggerer.
> If running in Kubernetes and the replicas are each set to 1, then K8 will restart the process whenever any pod goes down for any reason. Thus a value > 1 is not strictly needed for availability. Please confirm.
>
> Thanks.