You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/12/27 18:24:01 UTC
Kill airflow-scheduler without having to manually kill all scheduler PIDs?
Running airflow (v1.10.5) with LocalExecutor and finding that when wanting
to restart or otherwise kill the scheduler daemon (started via airflow
scheduler -D), need to *manually* kill -9 ... each scheduler process, eg...
[airflow@airflowetl airflow]$ ps -aux | grep scheduler
airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00
/usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00
airflow scheduler -- DagFileProcessorManager
[airflow@airflowetl airflow]$ kill -9 $(cat
$AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
Is there an easier way to kill the scheduler (and webserver) daemon(s)? Ie.
without having to kill all scheduler daemon instances manually (not just
the PID in airflow-scheduler.pid file and named PID).
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Right - seems that systemd default behaviour is to use "control-group" kill
mode which kills all the processes spawned by airflow:
https://www.freedesktop.org/software/systemd/man/systemd.kill.html# -
that's why it works nicely.
J.
On Sat, Jan 4, 2020 at 12:08 PM Kaxil Naik <ka...@gmail.com> wrote:
> I generally use systemd integration. It has various benefits like
> restarting on failure, start with certain sets of env vars and as a cetain
> user.
>
> You can use "systemctl stop airflow-webserver" or "systemctl start
> airflow-webserver"
>
> Regards,
> Kaxil
>
> On Sat, Jan 4, 2020, 11:03 Jarek Potiuk <Ja...@polidea.com> wrote:
>
>> I am not sure what the current behaviour is, but I brought the subject to
>> devlist
>> https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E .
>> I hope we can get some more insight from other committers/contributors and
>> we will describe/implement some good ways of killing the
>> webserver/scheduler (if there are no good ones currently).
>>
>> J
>>
>> On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com>
>> wrote:
>>
>>> Hi, Villanueva.
>>>
>>> I don’t find any out of box way to kill all webserver or scheduler.
>>> I alway use some bash script to kill all the webserver or scheduler,
>>> just like below
>>>
>>> ```
>>> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
>>> ```
>>>
>>> If you want to kill scheduler , just change webserver to scheduler
>>>
>>>
>>> Best Wish
>>> — Jiajie
>>>
>>>
>>>
>>> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org>
>>> wrote:
>>>
>>> Running airflow (v1.10.5) with LocalExecutor and finding that when
>>> wanting to restart or otherwise kill the scheduler daemon (started via airflow
>>> scheduler -D), need to *manually* kill -9 ... each scheduler process,
>>> eg...
>>>
>>> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
>>> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>>>
>>> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>>>
>>> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
>>> Ie. without having to kill all scheduler daemon instances manually (not
>>> just the PID in airflow-scheduler.pid file and named PID).
>>>
>>> This electronic message is intended only for the named
>>> recipient, and may contain information that is confidential or
>>> privileged. If you are not the intended recipient, you are
>>> hereby notified that any disclosure, copying, distribution or
>>> use of the contents of this message is strictly prohibited. If
>>> you have received this message in error or are not the named
>>> recipient, please notify us immediately by contacting the
>>> sender at the electronic mail address noted above, and delete
>>> and destroy all copies of this message. Thank you.
>>>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Kaxil Naik <ka...@gmail.com>.
I generally use systemd integration. It has various benefits like
restarting on failure, start with certain sets of env vars and as a cetain
user.
You can use "systemctl stop airflow-webserver" or "systemctl start
airflow-webserver"
Regards,
Kaxil
On Sat, Jan 4, 2020, 11:03 Jarek Potiuk <Ja...@polidea.com> wrote:
> I am not sure what the current behaviour is, but I brought the subject to
> devlist
> https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E .
> I hope we can get some more insight from other committers/contributors and
> we will describe/implement some good ways of killing the
> webserver/scheduler (if there are no good ones currently).
>
> J
>
> On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com>
> wrote:
>
>> Hi, Villanueva.
>>
>> I don’t find any out of box way to kill all webserver or scheduler.
>> I alway use some bash script to kill all the webserver or scheduler, just
>> like below
>>
>> ```
>> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
>> ```
>>
>> If you want to kill scheduler , just change webserver to scheduler
>>
>>
>> Best Wish
>> — Jiajie
>>
>>
>>
>> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>>
>> Running airflow (v1.10.5) with LocalExecutor and finding that when
>> wanting to restart or otherwise kill the scheduler daemon (started via airflow
>> scheduler -D), need to *manually* kill -9 ... each scheduler process,
>> eg...
>>
>> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
>> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>>
>> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>>
>> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
>> Ie. without having to kill all scheduler daemon instances manually (not
>> just the PID in airflow-scheduler.pid file and named PID).
>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Jarek Potiuk <Ja...@polidea.com>.
I am not sure what the current behaviour is, but I brought the subject to
devlist
https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E
.
I hope we can get some more insight from other committers/contributors and
we will describe/implement some good ways of killing the
webserver/scheduler (if there are no good ones currently).
J
On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com> wrote:
> Hi, Villanueva.
>
> I don’t find any out of box way to kill all webserver or scheduler.
> I alway use some bash script to kill all the webserver or scheduler, just
> like below
>
> ```
> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
> ```
>
> If you want to kill scheduler , just change webserver to scheduler
>
>
> Best Wish
> — Jiajie
>
>
>
> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>
> Running airflow (v1.10.5) with LocalExecutor and finding that when
> wanting to restart or otherwise kill the scheduler daemon (started via airflow
> scheduler -D), need to *manually* kill -9 ... each scheduler process,
> eg...
>
> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>
> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>
> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
> Ie. without having to kill all scheduler daemon instances manually (not
> just the PID in airflow-scheduler.pid file and named PID).
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by hotmail <zh...@hotmail.com>.
Hi, Villanueva.
I don’t find any out of box way to kill all webserver or scheduler.
I alway use some bash script to kill all the webserver or scheduler, just like below
```
ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
```
If you want to kill scheduler , just change webserver to scheduler
Best Wish
— Jiajie
> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>
> Running airflow (v1.10.5) with LocalExecutor and finding that when wanting to restart or otherwise kill the scheduler daemon (started via airflow scheduler -D), need to manually kill -9 ... each scheduler process, eg...
>
> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>
> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
> Is there an easier way to kill the scheduler (and webserver) daemon(s)? Ie. without having to kill all scheduler daemon instances manually (not just the PID in airflow-scheduler.pid file and named PID).
>
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.