You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/08/13 14:30:00 UTC
[jira] [Commented] (AIRFLOW-4843) Allow orchestration of tasks with
Docker Swarm aka `SwarmOperator`
[ https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906255#comment-16906255 ]
ASF GitHub Bot commented on AIRFLOW-4843:
-----------------------------------------
mik-laj commented on pull request #5489: [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator)
URL: https://github.com/apache/airflow/pull/5489
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
> ------------------------------------------------------------------
>
> Key: AIRFLOW-4843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4843
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.3
> Reporter: Akshesh Doshi
> Assignee: Akshesh Doshi
> Priority: Major
> Labels: Docker, docker, orchestration, swarm
>
> Currently, Airflow supports spawning Docker containers for running tasks via the {color:#707070}_DockerOperator_{color} but these containers are run on the same node as the scheduler.
> It would be helpful for our use-case to be able to spawn these tasks wherever resources are available in our Docker Swarm cluster.
>
> This can be achieved by creating a Docker swarm service, waiting for its run and removing it after it has completed execution.
> This approach has been suggested/discussed at various places (and implemented in Golang for Swarm-cronjob):
> [https://blog.alexellis.io/containers-on-swarm/]
> [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3]
> [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b]
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)