You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by hotmail <zh...@hotmail.com> on 2020/01/04 09:31:10 UTC
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Hi, Villanueva.
I don’t find any out of box way to kill all webserver or scheduler.
I alway use some bash script to kill all the webserver or scheduler, just like below
```
ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
```
If you want to kill scheduler , just change webserver to scheduler
Best Wish
— Jiajie
> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>
> Running airflow (v1.10.5) with LocalExecutor and finding that when wanting to restart or otherwise kill the scheduler daemon (started via airflow scheduler -D), need to manually kill -9 ... each scheduler process, eg...
>
> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>
> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
> Is there an easier way to kill the scheduler (and webserver) daemon(s)? Ie. without having to kill all scheduler daemon instances manually (not just the PID in airflow-scheduler.pid file and named PID).
>
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Right - seems that systemd default behaviour is to use "control-group" kill
mode which kills all the processes spawned by airflow:
https://www.freedesktop.org/software/systemd/man/systemd.kill.html# -
that's why it works nicely.
J.
On Sat, Jan 4, 2020 at 12:08 PM Kaxil Naik <ka...@gmail.com> wrote:
> I generally use systemd integration. It has various benefits like
> restarting on failure, start with certain sets of env vars and as a cetain
> user.
>
> You can use "systemctl stop airflow-webserver" or "systemctl start
> airflow-webserver"
>
> Regards,
> Kaxil
>
> On Sat, Jan 4, 2020, 11:03 Jarek Potiuk <Ja...@polidea.com> wrote:
>
>> I am not sure what the current behaviour is, but I brought the subject to
>> devlist
>> https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E .
>> I hope we can get some more insight from other committers/contributors and
>> we will describe/implement some good ways of killing the
>> webserver/scheduler (if there are no good ones currently).
>>
>> J
>>
>> On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com>
>> wrote:
>>
>>> Hi, Villanueva.
>>>
>>> I don’t find any out of box way to kill all webserver or scheduler.
>>> I alway use some bash script to kill all the webserver or scheduler,
>>> just like below
>>>
>>> ```
>>> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
>>> ```
>>>
>>> If you want to kill scheduler , just change webserver to scheduler
>>>
>>>
>>> Best Wish
>>> — Jiajie
>>>
>>>
>>>
>>> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org>
>>> wrote:
>>>
>>> Running airflow (v1.10.5) with LocalExecutor and finding that when
>>> wanting to restart or otherwise kill the scheduler daemon (started via airflow
>>> scheduler -D), need to *manually* kill -9 ... each scheduler process,
>>> eg...
>>>
>>> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
>>> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>>> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>>>
>>> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>>>
>>> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
>>> Ie. without having to kill all scheduler daemon instances manually (not
>>> just the PID in airflow-scheduler.pid file and named PID).
>>>
>>> This electronic message is intended only for the named
>>> recipient, and may contain information that is confidential or
>>> privileged. If you are not the intended recipient, you are
>>> hereby notified that any disclosure, copying, distribution or
>>> use of the contents of this message is strictly prohibited. If
>>> you have received this message in error or are not the named
>>> recipient, please notify us immediately by contacting the
>>> sender at the electronic mail address noted above, and delete
>>> and destroy all copies of this message. Thank you.
>>>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Kaxil Naik <ka...@gmail.com>.
I generally use systemd integration. It has various benefits like
restarting on failure, start with certain sets of env vars and as a cetain
user.
You can use "systemctl stop airflow-webserver" or "systemctl start
airflow-webserver"
Regards,
Kaxil
On Sat, Jan 4, 2020, 11:03 Jarek Potiuk <Ja...@polidea.com> wrote:
> I am not sure what the current behaviour is, but I brought the subject to
> devlist
> https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E .
> I hope we can get some more insight from other committers/contributors and
> we will describe/implement some good ways of killing the
> webserver/scheduler (if there are no good ones currently).
>
> J
>
> On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com>
> wrote:
>
>> Hi, Villanueva.
>>
>> I don’t find any out of box way to kill all webserver or scheduler.
>> I alway use some bash script to kill all the webserver or scheduler, just
>> like below
>>
>> ```
>> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
>> ```
>>
>> If you want to kill scheduler , just change webserver to scheduler
>>
>>
>> Best Wish
>> — Jiajie
>>
>>
>>
>> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>>
>> Running airflow (v1.10.5) with LocalExecutor and finding that when
>> wanting to restart or otherwise kill the scheduler daemon (started via airflow
>> scheduler -D), need to *manually* kill -9 ... each scheduler process,
>> eg...
>>
>> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
>> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
>> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>>
>> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>>
>> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
>> Ie. without having to kill all scheduler daemon instances manually (not
>> just the PID in airflow-scheduler.pid file and named PID).
>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: Kill airflow-scheduler without having to manually kill all
scheduler PIDs?
Posted by Jarek Potiuk <Ja...@polidea.com>.
I am not sure what the current behaviour is, but I brought the subject to
devlist
https://lists.apache.org/thread.html/0b9881d2a70dfeacc7371f45b672ec1f1691868fe23dae1553002521%40%3Cdev.airflow.apache.org%3E
.
I hope we can get some more insight from other committers/contributors and
we will describe/implement some good ways of killing the
webserver/scheduler (if there are no good ones currently).
J
On Sat, Jan 4, 2020 at 10:31 AM hotmail <zh...@hotmail.com> wrote:
> Hi, Villanueva.
>
> I don’t find any out of box way to kill all webserver or scheduler.
> I alway use some bash script to kill all the webserver or scheduler, just
> like below
>
> ```
> ps aux | grep webserver | grep -v grep | awk '{print $2}' | xargs kill -9
> ```
>
> If you want to kill scheduler , just change webserver to scheduler
>
>
> Best Wish
> — Jiajie
>
>
>
> On Dec 28, 2019, at 02:24, Reed Villanueva <rv...@ucera.org> wrote:
>
> Running airflow (v1.10.5) with LocalExecutor and finding that when
> wanting to restart or otherwise kill the scheduler daemon (started via airflow
> scheduler -D), need to *manually* kill -9 ... each scheduler process,
> eg...
>
> [airflow@airflowetl airflow]$ ps -aux | grep scheduler
> airflow 9137 2.0 0.1 723196 72964 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9141 0.5 0.1 1503344 72724 ? Sl 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9149 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9150 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9154 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9157 0.0 0.1 722940 71408 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9162 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9166 0.0 0.1 722940 71416 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9171 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9175 0.0 0.1 722940 71412 ? S 11:06 0:00 /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D
> airflow 9181 0.7 0.1 723344 72808 ? S 11:06 0:00 airflow scheduler -- DagFileProcessorManager
>
> [airflow@airflowetl airflow]$ kill -9 $(cat $AIRFLOW_HOME/airflow-scheduler.pid) 9181 9175 ...
>
> Is there an easier way to kill the scheduler (and webserver) daemon(s)?
> Ie. without having to kill all scheduler daemon instances manually (not
> just the PID in airflow-scheduler.pid file and named PID).
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>