You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/12/18 16:48:00 UTC
[jira] [Commented] (AIRFLOW-4843) Allow orchestration of tasks with
Docker Swarm aka `SwarmOperator`
[ https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999326#comment-16999326 ]
ASF subversion and git services commented on AIRFLOW-4843:
----------------------------------------------------------
Commit 29cd0babf42a200671bbcf6b5df627735417c2f9 in airflow's branch refs/heads/v1-10-test from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=29cd0ba ]
fixup! [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
> Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
> ------------------------------------------------------------------
>
> Key: AIRFLOW-4843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4843
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.3
> Reporter: Akshesh Doshi
> Assignee: Akshesh Doshi
> Priority: Major
> Labels: Docker, docker, orchestration, swarm
> Fix For: 1.10.7
>
>
> Currently, Airflow supports spawning Docker containers for running tasks via the {color:#707070}_DockerOperator_{color} but these containers are run on the same node as the scheduler.
> It would be helpful for our use-case to be able to spawn these tasks wherever resources are available in our Docker Swarm cluster.
>
> This can be achieved by creating a Docker swarm service, waiting for its run and removing it after it has completed execution.
> This approach has been suggested/discussed at various places (and implemented in Golang for Swarm-cronjob):
> [https://blog.alexellis.io/containers-on-swarm/]
> [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3]
> [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)