You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/30 13:58:47 UTC
[GitHub] [airflow] santosh-d3vpl3x opened a new pull request #17341: Add ExecStop to systemd service files.
santosh-d3vpl3x opened a new pull request #17341:
URL: https://github.com/apache/airflow/pull/17341
Add `ExecStop` to systemd service files and use `airflow celery stop` command to it.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-892574013
> To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
Why would you want to do this? You can have one worker listen on multiple queues if that is what you want to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-889913686
To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
<details>
<summary>airflow-worker-queue1.service</summary>
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8794
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue1 --pid /var/run/airflow/worker-queue1.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue1.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
</details>
<details>
<summary>airflow-worker-queue2.service</summary>
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8795
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue2 --pid /var/run/airflow/worker-queue2.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue2.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x closed pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x closed pull request #17341:
URL: https://github.com/apache/airflow/pull/17341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-893011330
> > To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
>
> Can you explain where doing this would be useful? You can have one worker listen on multiple queues if that is what you want to do
Indeed, celery provides different interactions with queues. The pattern mentioned is useful when one would want to keep lifecycle of workers consuming from different queues decoupled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-892574013
> To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
Can you explain where doing this would be useful? You can have one worker listen on multiple queues if that is what you want to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-893015051
> This shouldn't make any difference -- the default behaviour of systemd is to send SIGTERM, which `celery worker` will pick up and do a warm shut down already.
Yes! Systemd has default behavior of sending a SIGTERM. I agree, this PR is redundant. We are experiencing some issues with some hooks where zombies are left behind while shutting down the workers. This addition seemed to work at first and we got correct behavior. But after a longer observation, that isn't the case.
`systemctl stop` command is sending multiple sigterms to airflow for some reason, and that results in jobs being left behind + duplicate data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-892574013
> To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
Can you explain where doing this would be useful? You can have one worker listen on multiple queues if that is what you want to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x closed pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x closed pull request #17341:
URL: https://github.com/apache/airflow/pull/17341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x edited a comment on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x edited a comment on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-889913686
To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
**airflow-worker-queue1.service**
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8794
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue1 --pid /var/run/airflow/worker-queue1.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue1.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
**airflow-worker-queue2.service**
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8795
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue2 --pid /var/run/airflow/worker-queue2.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue2.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
One could also use `%i` but not so sure if it will benefit the example really.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-892574013
> To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
Why would you want to do this? You can have one worker listen on multiple queues if that is what you want to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x closed pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x closed pull request #17341:
URL: https://github.com/apache/airflow/pull/17341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x edited a comment on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x edited a comment on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-889913686
To add to this, would it be a good idea to demonstrate use of multiple worker on same machine?
**airflow-worker-queue1.service**
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8794
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue1 --pid /var/run/airflow/worker-queue1.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue1.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
**airflow-worker-queue2.service**
```
[Unit]
Description=Airflow celery worker daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
Environment=AIRFLOW__CELERY__WORKER_LOG_SERVER_PORT=8795
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow celery worker -q queue2 --pid /var/run/airflow/worker-queue2.pid
ExecStop=/bin/airflow celery stop --pid /var/run/airflow/worker-queue2.pid
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-893011330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17341: Add ExecStop to systemd service files.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17341:
URL: https://github.com/apache/airflow/pull/17341#issuecomment-890024490
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org