You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/02 08:47:14 UTC
[GitHub] [airflow] georborodin opened a new issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
georborodin opened a new issue #17373:
URL: https://github.com/apache/airflow/issues/17373
**Description**
Allow custom executors subclassed from existing (CeleryExecutor, CeleryKubernetesExecutor, etc.) to use default CLI commands to start workers or flower monitoring.
**Use case / motivation**
Currently, users who decide to roll their own custom Celery-based executor cannot use default commands (i.e. `airflow celery worker`) even though it's built on top of existing CeleryExecutor. If they try to, they'll receive the following error: `airflow command error: argument GROUP_OR_COMMAND: celery subcommand works only with CeleryExecutor, your current executor: custom_package.CustomCeleryExecutor, see help above.`
One workaround for this is to create custom entrypoint script for worker/flower containers/processes that are still going to use the same Celery app as CeleryExecutor. This leads to unnecessary maintaining of this entrypoint script.
I'd suggest two ways of fixing that:
- Check if custom executor is subclassed from Celery executor (which might lead to errors, if custom executor is used to access other celery app, which might be a proper reason for rolling custom executor)
- Store `app` as attribute of Celery-based executors and match the one provided by custom executor with the default one
**Related Issues**
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] georborodin commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
georborodin commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-893122280
@potiuk would you mind taking a look at my PR to see if I got it right? I feel that code might be a bit 'hacky' 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-890974294
Assigned you then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17373:
URL: https://github.com/apache/airflow/issues/17373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-893223287
> @potiuk would you mind taking a look at my PR to see if I got it right? I feel that code might be a bit 'hacky'
Not really. It's good. Small Nit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] georborodin commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
georborodin commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-893122280
@potiuk would you mind taking a look at my PR to see if I got it right? I feel that code might be a bit 'hacky' 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] georborodin commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
georborodin commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-890960560
@potiuk would be my pleasure! Suppose I'd be able to provide a PR in a couple of days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-890846502
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-890934171
Would you like to provide a PR fixing it? Seems like the approach where executor could define what "type" of the executor it is a good approach. Sounds like an easy fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] santosh-d3vpl3x commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
santosh-d3vpl3x commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-894637591
We ran into this limitation while trying to extend celery executor to fix one of the issues found during airflow upgrade. It would be a good addition definitely :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17373: Allow using default celery commands for custom Celery executors subclassed from existing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17373:
URL: https://github.com/apache/airflow/issues/17373#issuecomment-893223287
> @potiuk would you mind taking a look at my PR to see if I got it right? I feel that code might be a bit 'hacky'
Not really. It's good. Small Nit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org