You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/20 20:32:44 UTC
[GitHub] [airflow] mjpieters opened a new issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
mjpieters opened a new issue #11697:
URL: https://github.com/apache/airflow/issues/11697
The recent [refactoring of the celery Worker process](https://github.com/apache/airflow/commit/02ce45cafec22a0a80257b2144d99ec8bb41c961) dropped the use of `celery.bin.worker.Worker()`, and switched to using the `celery.app.worker.Worker()` class more directly.
Unfortunately, it was the task of the former to format the [Celery hostname patterns](https://docs.celeryproject.org/en/latest/reference/cli.html?highlight=hostname#cmdoption-celery-worker-n).
Unfortunately this refactoring dropped support for hostname patterns (`%h`, `%n`, etc.). Can we re-introduce these? All that is needed is a few `celery.utils.nodenames` calls:
```python
from celery.utils.nodenames import default_nodename, host_format
# ...
'hostname': host_format(default_nodename(args.celery_hostname)),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-713128948
CC: @turbaszek @auvipy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-714393937
> If this is a problem in general, please file an issue on Celery's repository.
@mjpieters do you think it may be worth reporting it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
turbaszek closed issue #11697:
URL: https://github.com/apache/airflow/issues/11697
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thedrow commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
thedrow commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-716407819
I presumed that functionality is related to the CLI itself.
I can fix it or you can come up with a PR, if that's not the case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mjpieters commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
mjpieters commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-714636908
I _think_ it is a feature of the `celery worker` command, not necessarily of the `Worker` program?
The command-line for that command supports using `%h`, `%n` and `%i` placeholders, and by extension, so did `airflow worker`, and I was grateful for that support. Since Celery puts handling those placeholders in `celery.bin.worker` I'm presuming that they see this separate from `celery.apps.worker`. I haven't looked at how Celery 5 now handles this, since the whole `celery.apps` notion has been dropped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-713136307
To be honest, I think we will need to revert #11336 due to #1162, it's more feasible than upgrading to 5.0 as suggested in the latter issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-713128948
CC: @turbaszek @auvipy Can you look at it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mjpieters commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
mjpieters commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-714638691
I took a look at Celery 5, and there those parameters are very firmly placed in the 'command-line argument' camp, by defining a custom Click [argument type `Hostname`](https://docs.celeryproject.org/en/latest/_modules/celery/bin/worker.html#Hostname), that handles the placeholders when parsing command-line arguments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-713136307
To be honest, I think we will need to revert #11336 due to #11622 , it's more feasible than upgrading to 5.0 as suggested in the latter issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thedrow commented on issue #11697: Celery worker hostname no longer supports %h, %n & %d patterns
Posted by GitBox <gi...@apache.org>.
thedrow commented on issue #11697:
URL: https://github.com/apache/airflow/issues/11697#issuecomment-714360093
If this is a problem in general, please file an issue on Celery's repository.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org