About airflow scheduler

[Jason Chen <ch...@gmail.com>]

Hi airflow team,

 We setup airflow as a upstart service using the
suggestion here
https://github.com/apache/incubator-airflow/blob/master/scripts/upstart/airflow-scheduler.conf#L33

We set SCHEDULER_RUNS=0 (unlimit).

We notice that using "ps aux | grep airflow", it indicates several
scheduler running as below...
I am thinking it's normal, because the scheduler executes for each run (and
there are multiple runs at the same time).
Is that right ?
Also, can you explain "Scheduler will restart after every ${SCHEDULER_RUNS}
runs" ?

Thanks.

Jason


......
ec2-user 25517  0.0  0.1 405372 56776 pts/0    Sl+  02:53   0:00
/usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
ec2-user 25518  0.0  0.1 405372 56544 pts/0    S+   02:53   0:00
/usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
ec2-user 25519  0.0  0.1 405372 56772 pts/0    Sl+  02:53   0:00
/usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
ec2-user 25520  0.0  0.1 405372 56772 pts/0    Sl+  02:53   0:00
/usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
ec2-user 25521  0.0  0.1 405372 56548 pts/0    S+   02:53   0:00
/usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
........

Re: About airflow scheduler

[Sumit Maheshwari <su...@gmail.com>]

Hi Jason,

See this from the conf file:

# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run. However airflow will never
# use more threads than the amount of cpu cores available.
max_threads = 2

So probably the no of processes you see are equal to no of processes your
machine have. About the "*num_runs*" things, scheduler process will
gracefully dies once it reaches to the said no of cycles. A cycle is
nothing but a loop of iteration over all the dags present in your dags
folder.


Thanks,
Sumit


On Sun, Sep 4, 2016 at 8:28 AM, Jason Chen <ch...@gmail.com>
wrote:

> Hi airflow team,
>
>  We setup airflow as a upstart service using the
> suggestion here
> https://github.com/apache/incubator-airflow/blob/master/
> scripts/upstart/airflow-scheduler.conf#L33
>
> We set SCHEDULER_RUNS=0 (unlimit).
>
> We notice that using "ps aux | grep airflow", it indicates several
> scheduler running as below...
> I am thinking it's normal, because the scheduler executes for each run (and
> there are multiple runs at the same time).
> Is that right ?
> Also, can you explain "Scheduler will restart after every ${SCHEDULER_RUNS}
> runs" ?
>
> Thanks.
>
> Jason
>
>
> ......
> ec2-user 25517  0.0  0.1 405372 56776 pts/0    Sl+  02:53   0:00
> /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
> ec2-user 25518  0.0  0.1 405372 56544 pts/0    S+   02:53   0:00
> /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
> ec2-user 25519  0.0  0.1 405372 56772 pts/0    Sl+  02:53   0:00
> /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
> ec2-user 25520  0.0  0.1 405372 56772 pts/0    Sl+  02:53   0:00
> /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
> ec2-user 25521  0.0  0.1 405372 56548 pts/0    S+   02:53   0:00
> /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0
> ........
>