You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/02 06:31:50 UTC
[GitHub] [airflow] ac1997 opened a new issue #19959: Fully Configurable Queue based Executor
ac1997 opened a new issue #19959:
URL: https://github.com/apache/airflow/issues/19959
### Description
I would like to add a queue based executor to the main branch which we currently use to support wide variety of worker clusters we currently used for Airflow. The idea stems from `CeleryKubernetesExecutor` but with fully configurable routing logic to different executors.
I am not sure how much of this executor will help the community, hence raising an issue to see what the committee thinks about the idea. We can also remove `CeleryKubernetesExecutor` in the future as this new executor will be a general implementation of it.
### Use case/motivation
We have multiple different proprietary worker clusters available, each of them with different implementation and serves different purposes. As a workaround to support these worker clusters, we implemented a queue-based executor, very similar to `CeleryKubernetesExecutor` but with fully configurable routing logic.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ac1997 commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ac1997 commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-1007631289
Closing the issue per suggestion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] birallex commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
birallex commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-1003334507
This is a great idea, as it will make life much easier for those developers who use custom executors and still want to combine them. In my case, a modified `KubernetesExecutor` is used because of the platform where the airflow is used has limitations related to the creation of pods. It would be great to be able to combine it with another custom/built-in executor without any code rework, just using configuration. In the current version of aiflow using a custom `CeleryKubernetesExecutor` with my implementation of the patched `KubernetesExecutor` does not seem possible, but the proposed by @ac1997 solution can greatly simplify this process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ac1997 commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ac1997 commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-984993474
That's a good point. We didn't really think about those weird combination of executors. It might worth adding some sort of executor banned list to this multi-executor.
As for configuration, this is what we have currently. It certainly can be improved.
```
[core]
executor = path.to.queue_executor.QueueExecutor
[queue_executor]
count = 2
default = path.to.default_executor.Executor
executor_1 = queue_name1:path.to.executor_1.Executor_1
executor_2 = queue_name2:path.to.executor_2.Executor_2
```
My current implementation also allows additional configuration in queue name in the form of `queue:config`. Routing will be based on the first value in the queue name where as the rest of them will be passed to and consumed by downstream executor for executor level configuration such as cluster to execute on etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ac1997 commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ac1997 commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-1007631289
Closing the issue per suggestion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ac1997 closed issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ac1997 closed issue #19959:
URL: https://github.com/apache/airflow/issues/19959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-984332928
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-1004744051
Given that executors can already be used from anywhere I think this should start out as a third party module, and then if it proves both useful _and_ popular over time we can pull it in to core at a later date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ac1997 closed issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
ac1997 closed issue #19959:
URL: https://github.com/apache/airflow/issues/19959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-984942129
We discussed it at #19729. If you think you can implement full testing coverage and make it resilient to all kinds of configuration errors and uses (like joining SequentialExecutor with any other) and have a good and easy to use configuration, it might be worth making such PR
I am a bit sceptical, @ashb - you thought this might be a good idea, so if somoene has it already proven and working we could take a stab with it. Maybe if you start your PR we can discuss it and see if it's close to be ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19959: Fully Configurable Queue based Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19959:
URL: https://github.com/apache/airflow/issues/19959#issuecomment-1004766650
> Given that executors can already be used from anywhere I think this should start out as a third party module, and then if it proves both useful _and_ popular over time we can pull it in to core at a later date.
Yep. All for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org