You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/19 22:40:23 UTC
[GitHub] [airflow] tooptoop4 opened a new issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
tooptoop4 opened a new issue #8460:
URL: https://github.com/apache/airflow/issues/8460
is there a way to set executor per task? ie I want all tasks (except task named abc) to use celeryexecutor/daskexecutor but I want task named abc to use local executor (due to firewall only opened from master)
https://github.com/apache/airflow/pull/6725 has some relevant code, hoping it can be merged!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-619232568
I once added a POC for multiple executors based on queue - would you find it useful ?https://github.com/PolideaInternal/airflow/commit/6590f665d4d9aa2629052c1af2e6f4c36a9a4dd3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-619232568
I once added a POC for multiple executors based on queue - would you find it useful ?
https://github.com/PolideaInternal/airflow/commit/c87e87a5e7673e67e78d4ce39d5a15a8cc494e56
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joshowen commented on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
joshowen commented on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-666234499
Hi @potiuk I do think this is worth resurrecting. Having a path to migrate between executors is helpful with a larger deployments, especially now that the Kube executor is a thing. I assume I could build a custom executor to handle the executor instance routing, but making it easily configurable by dag/task is definitely preferable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-619336041
can u pls merge to apache airflow repo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-619232568
I once added a PR for multiple executors based on queue - would you find it useful ?https://github.com/PolideaInternal/airflow/commit/6590f665d4d9aa2629052c1af2e6f4c36a9a4dd3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #8460:
URL: https://github.com/apache/airflow/issues/8460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-750341992
We have CeleryK8SExecutor now and I think it is better to have this specialized executor rather than generic one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #8460: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8460:
URL: https://github.com/apache/airflow/issues/8460#issuecomment-619383691
Well. I could resurrect it yeah. But only after I finish some of the most urgent things with CI/Kubernetes/prod image - but If you would like to pick it up till then - feel free @tooptoop4 :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org