You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/07 20:50:01 UTC
[GitHub] [airflow] turbaszek opened a new pull request #11336: Refactor celery worker command
turbaszek opened a new pull request #11336:
URL: https://github.com/apache/airflow/pull/11336
This commit does small refactor of the way we star celery worker.
In this way it will be easier to migrate to Celery 5.0.
Running with currently used Celery version:
```
root@1e7421e00701:/opt/airflow# airflow celery worker
/usr/local/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
Starting flask
* Serving Flask app "airflow.utils.serve_logs" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
[2020-10-07 20:44:47,384: INFO/Process-1] * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
-------------- celery@1e7421e00701 v4.4.7 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.19.76-linuxkit-x86_64-with-debian-10.5 2020-10-07 20:44:47
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f401d0d2a10
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: postgresql://postgres:**@postgres/airflow
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
[2020-10-07 20:44:47,988: INFO/MainProcess] Connected to redis://redis:6379/0
[2020-10-07 20:44:47,998: INFO/MainProcess] mingle: searching for neighbors
[2020-10-07 20:44:49,024: INFO/MainProcess] mingle: all alone
[2020-10-07 20:44:49,045: INFO/MainProcess] celery@1e7421e00701 ready.
^C
worker: Hitting Ctrl+C again will terminate all running tasks!
worker: Warm shutdown (MainProcess)
```
Running with Celery 5.0:
```
root@5257c48ec87e:/opt/airflow# airflow celery worker
/usr/local/lib/python3.7/site-packages/celery/platforms.py:798: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
Starting flask
* Serving Flask app "airflow.utils.serve_logs" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
[2020-10-07 20:40:27,110: INFO/Process-1] * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
-------------- celery@5257c48ec87e v5.0.0 (singularity)
--- ***** -----
-- ******* ---- Linux-4.19.76-linuxkit-x86_64-with-debian-10.5 2020-10-07 20:40:27
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7fb3e65b5dd0
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: postgresql://postgres:**@postgres/airflow
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
[2020-10-07 20:40:27,701: INFO/MainProcess] Connected to redis://redis:6379/0
[2020-10-07 20:40:27,711: INFO/MainProcess] mingle: searching for neighbors
[2020-10-07 20:40:28,737: INFO/MainProcess] mingle: all alone
[2020-10-07 20:40:28,762: INFO/MainProcess] celery@5257c48ec87e ready.
^C
worker: Hitting Ctrl+C again will terminate all running tasks!
worker: Warm shutdown (MainProcess)
```
Related to #11301
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #11336: Refactor celery worker command
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #11336:
URL: https://github.com/apache/airflow/pull/11336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #11336: Refactor celery worker command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #11336:
URL: https://github.com/apache/airflow/pull/11336#issuecomment-705194815
@auvipy would you mind taking a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11336: Refactor celery worker command
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11336:
URL: https://github.com/apache/airflow/pull/11336#issuecomment-705208964
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/294270769) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11336: Refactor celery worker command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11336:
URL: https://github.com/apache/airflow/pull/11336#discussion_r501306757
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -130,17 +127,15 @@ def worker(args):
pass
# Setup Celery worker
- worker_instance = worker_bin.worker(app=celery_app)
options = {
'optimization': 'fair',
- 'O': 'fair',
Review comment:
Because according to worker implementation:
```
self.optimization = optimization or O
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org