You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/18 15:06:54 UTC
[GitHub] [airflow] nuclearpinguin opened a new pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
nuclearpinguin opened a new pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206
Now users can gracefully stop the Celery worker by sending a SIGTERM signal to Celery main process. To do that they have to call `airflow celery stop`.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368231509
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -93,9 +98,6 @@ def _serve_logs(skip_serve_logs: bool = False) -> Optional[Process]:
@cli_utils.action_logging
def worker(args):
"""Starts Airflow Celery worker"""
- env = os.environ.copy()
- env['AIRFLOW_HOME'] = settings.AIRFLOW_HOME
-
Review comment:
Unused stuff
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368232641
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -115,23 +127,19 @@ def worker(args):
'autoscale': autoscale,
'hostname': args.celery_hostname,
'loglevel': conf.get('logging', 'LOGGING_LEVEL'),
+ 'pidfile': pid_file_path,
}
if conf.has_option("celery", "pool"):
options["pool"] = conf.get("celery", "pool")
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker",
- args.pid,
- args.stdout,
- args.stderr,
- args.log_file)
+ # Run Celery worker as daemon
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
ctx = daemon.DaemonContext(
- pidfile=TimeoutPIDLockFile(pid, -1),
Review comment:
Also, it seems that -D flag doesn't work with this line. No error is propagated and I see no new celery processes. When I remove it, everything works as expected? WDYT? @potiuk @mik-laj @dimberman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.36%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.05% -0.37%
==========================================
Files 753 754 +1
Lines 39709 39842 +133
==========================================
- Hits 33918 33886 -32
- Misses 5791 5956 +165
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
| [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: |
| ... and [13 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...d25d950](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.34%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.06% -0.35%
==========================================
Files 753 791 +38
Lines 39709 40130 +421
==========================================
+ Hits 33918 34138 +220
- Misses 5791 5992 +201
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/api/client/local\_client.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L2xvY2FsX2NsaWVudC5weQ==) | `92% <0%> (-8%)` | :arrow_down: |
| ... and [93 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...eaa40d3](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dimberman commented on issue #7206: [AIRFLOW-6591] Add
cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575915167
@nuclearpinguin huge +1 on this one! Can you set up a unit test around it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.36%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.05% -0.37%
==========================================
Files 753 754 +1
Lines 39709 39842 +133
==========================================
- Hits 33918 33886 -32
- Misses 5791 5956 +165
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
| [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: |
| ... and [13 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...d25d950](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r369215821
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -23,8 +23,10 @@
from typing import Optional
import daemon
+import psutil
from celery.bin import worker as worker_bin
from daemon.pidfile import TimeoutPIDLockFile
Review comment:
Hm, and there's no pylint error?
In my opinion we should use Celery builtin mechanism:
https://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-pidfile
the main advantage is that in this way it works for both worker's modes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368941447
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -115,23 +127,19 @@ def worker(args):
'autoscale': autoscale,
'hostname': args.celery_hostname,
'loglevel': conf.get('logging', 'LOGGING_LEVEL'),
+ 'pidfile': pid_file_path,
}
if conf.has_option("celery", "pool"):
options["pool"] = conf.get("celery", "pool")
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker",
- args.pid,
- args.stdout,
- args.stderr,
- args.log_file)
+ # Run Celery worker as daemon
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
ctx = daemon.DaemonContext(
- pidfile=TimeoutPIDLockFile(pid, -1),
Review comment:
After all this works fine, the problem is when celery process exits unexpectedly and pid file is not removed. Then running worker with -D won't start up celery and user will not receive any information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.25% -0.16%
==========================================
Files 753 754 +1
Lines 39709 39842 +133
==========================================
+ Hits 33918 33968 +50
- Misses 5791 5874 +83
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
| [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/utils/log/json\_formatter.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvanNvbl9mb3JtYXR0ZXIucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...e8aad73](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.29%`.
> The diff coverage is `45.45%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
=========================================
- Coverage 85.41% 85.12% -0.3%
=========================================
Files 753 753
Lines 39709 39715 +6
=========================================
- Hits 33918 33807 -111
- Misses 5791 5908 +117
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `50% <45.45%> (-2.5%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...80acc9c](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368248914
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -93,9 +98,6 @@ def _serve_logs(skip_serve_logs: bool = False) -> Optional[Process]:
@cli_utils.action_logging
def worker(args):
"""Starts Airflow Celery worker"""
- env = os.environ.copy()
- env['AIRFLOW_HOME'] = settings.AIRFLOW_HOME
-
Review comment:
Looks like indeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r369223989
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -23,8 +23,10 @@
from typing import Optional
import daemon
+import psutil
from celery.bin import worker as worker_bin
from daemon.pidfile import TimeoutPIDLockFile
Review comment:
Agree. I looked at what celery uses and it's also PidFile under the hood + atexit removal of the lock, so I am perfectly OK.
https://github.com/celery/celery/blob/71d024446bbe3e0c9cdffe63fda2c83db42c7fc3/celery/platforms.py#L258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368249329
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -115,23 +127,19 @@ def worker(args):
'autoscale': autoscale,
'hostname': args.celery_hostname,
'loglevel': conf.get('logging', 'LOGGING_LEVEL'),
+ 'pidfile': pid_file_path,
}
if conf.has_option("celery", "pool"):
options["pool"] = conf.get("celery", "pool")
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker",
- args.pid,
- args.stdout,
- args.stderr,
- args.log_file)
+ # Run Celery worker as daemon
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
ctx = daemon.DaemonContext(
- pidfile=TimeoutPIDLockFile(pid, -1),
Review comment:
I'd have to discuss it in detail in-person, I am not 100% sure how PID handling should be done now :). Seems we have few places where it happens and I am not sure I understand all the paths.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7206: [AIRFLOW-6591] Add
cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr&el=desc) will **decrease** coverage by `0.25%`.
> The diff coverage is `53.84%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.37% 85.12% -0.26%
==========================================
Files 723 753 +30
Lines 39543 39717 +174
==========================================
+ Hits 33761 33809 +48
- Misses 5782 5908 +126
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `51.13% <53.84%> (-1.37%)` | :arrow_down: |
| [...rflow/contrib/sensors/sagemaker\_training\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvc2FnZW1ha2VyX3RyYWluaW5nX3NlbnNvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-22.33%)` | :arrow_down: |
| [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `85.9% <0%> (-0.31%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.23% <0%> (-0.12%)` | :arrow_down: |
| ... and [63 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [417feda...9e1af0c](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r369270713
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -23,8 +23,10 @@
from typing import Optional
import daemon
+import psutil
from celery.bin import worker as worker_bin
from daemon.pidfile import TimeoutPIDLockFile
Review comment:
Ouh, it's used in flower command 😺
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.29%`.
> The diff coverage is `45.45%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
=========================================
- Coverage 85.41% 85.12% -0.3%
=========================================
Files 753 753
Lines 39709 39715 +6
=========================================
- Hits 33918 33807 -111
- Misses 5791 5908 +117
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `50% <45.45%> (-2.5%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...80acc9c](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.34%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.06% -0.35%
==========================================
Files 753 791 +38
Lines 39709 40130 +421
==========================================
+ Hits 33918 34138 +220
- Misses 5791 5992 +201
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/api/client/local\_client.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L2xvY2FsX2NsaWVudC5weQ==) | `92% <0%> (-8%)` | :arrow_down: |
| ... and [93 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...eaa40d3](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368232641
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -115,23 +127,19 @@ def worker(args):
'autoscale': autoscale,
'hostname': args.celery_hostname,
'loglevel': conf.get('logging', 'LOGGING_LEVEL'),
+ 'pidfile': pid_file_path,
}
if conf.has_option("celery", "pool"):
options["pool"] = conf.get("celery", "pool")
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker",
- args.pid,
- args.stdout,
- args.stderr,
- args.log_file)
+ # Run Celery worker as daemon
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
ctx = daemon.DaemonContext(
- pidfile=TimeoutPIDLockFile(pid, -1),
Review comment:
Also, it seems that -D flag doesn't work with this line. No error is propagated and I see no new celery processes. When I remove it, everything works as expected? WDYT? @potiuk @mik-laj @dimberman
FYI, pidfile is now created by Celery.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.34%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.06% -0.35%
==========================================
Files 753 791 +38
Lines 39709 40130 +421
==========================================
+ Hits 33918 34138 +220
- Misses 5791 5992 +201
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/api/client/local\_client.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L2xvY2FsX2NsaWVudC5weQ==) | `92% <0%> (-8%)` | :arrow_down: |
| ... and [93 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...eaa40d3](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin removed a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin removed a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575908432
Also, it seems that -D flag doesn't work. No error is propagated and I see no new celery processes. Does anyone know how can I verify that deamonization works? @potiuk @mik-laj @dimberman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575910666
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=h1) Report
> Merging [#7206](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/37f026f66d0214449cad824d3d343999186d013e?src=pr&el=desc) will **decrease** coverage by `0.15%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7206 +/- ##
==========================================
- Coverage 85.41% 85.25% -0.16%
==========================================
Files 753 754 +1
Lines 39709 39842 +133
==========================================
+ Hits 33918 33968 +50
- Misses 5791 5874 +83
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `56.97% <100%> (+4.47%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
| [airflow/ti\_deps/deps/pool\_slots\_available\_dep.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcG9vbF9zbG90c19hdmFpbGFibGVfZGVwLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/utils/log/json\_formatter.py](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvanNvbl9mb3JtYXR0ZXIucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7206/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=footer). Last update [37f026f...e8aad73](https://codecov.io/gh/apache/airflow/pull/7206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin merged pull request #7206: [AIRFLOW-6591]
Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7206: [AIRFLOW-6591]
Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#issuecomment-575908432
Also, it seems that -D flag doesn't work. No error is propagated and I see no new celery processes. Does anyone know how can I verify that deamonization works? @potiuk @mik-laj @dimberman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7206: [AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368258541
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -115,23 +127,19 @@ def worker(args):
'autoscale': autoscale,
'hostname': args.celery_hostname,
'loglevel': conf.get('logging', 'LOGGING_LEVEL'),
+ 'pidfile': pid_file_path,
}
if conf.has_option("celery", "pool"):
options["pool"] = conf.get("celery", "pool")
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker",
- args.pid,
- args.stdout,
- args.stderr,
- args.log_file)
+ # Run Celery worker as daemon
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
ctx = daemon.DaemonContext(
- pidfile=TimeoutPIDLockFile(pid, -1),
Review comment:
After my changes pid file is handled by Celery builtin mechanism and standard lockfile / psutil methods in stop command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r368972555
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -23,8 +23,10 @@
from typing import Optional
import daemon
+import psutil
from celery.bin import worker as worker_bin
from daemon.pidfile import TimeoutPIDLockFile
Review comment:
Seems not used any more - can we still use it in our implementation ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7206:
[AIRFLOW-6591] Add cli option to stop celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
URL: https://github.com/apache/airflow/pull/7206#discussion_r369224282
##########
File path: airflow/cli/commands/celery_command.py
##########
@@ -23,8 +23,10 @@
from typing import Optional
import daemon
+import psutil
from celery.bin import worker as worker_bin
from daemon.pidfile import TimeoutPIDLockFile
Review comment:
And let's check why we have no pylint/flake warning :) ... Maybe the file is in the TO
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services