You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Yishan (Jira)" <ji...@apache.org> on 2019/10/10 17:55:00 UTC

[jira] [Created] (AIRFLOW-5633) Apache Airflow Integration with HashiCorp Nomad

Yishan created AIRFLOW-5633:
-------------------------------

             Summary: Apache Airflow Integration with HashiCorp Nomad
                 Key: AIRFLOW-5633
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5633
             Project: Apache Airflow
          Issue Type: New Feature
          Components: executors, operators
    Affects Versions: 1.10.5
            Reporter: Yishan


Nomad is an orchestrator, similar to Kubernetes and used in production by companies like PagerDuty, Target, Citadel, Trivago, Pandora, Roblox, etc.  Built by HashiCorp (company behind other tools such as Terraform, Vault, and Consul).

We've gotten many requests from our customers and open source community for an integration with Airflow.  There are a variety of community PRs from supportive members who are trying to build their own (linked notable ones below) - our approach would be to build one officially supported integration ourselves for all our users and customers.  

[https://github.com/apache/airflow/pull/2708]

[https://github.com/hashicorp/nomad/issues/419]

We'd love to understand how much interest and support we can gauge from the Airflow committee around an integration like this.  Ideally, we'd aim to avoid a situation where our integration is not accepted + Nomad users using Airflow are subject to an incomplete/less-than-ideal experience on a lagging HashiCorp-maintained-fork. 

This has unfortunately happened with our Spark integration, so we'd love to gauge the Airflow committee early-on in the process before we potentially build the integration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)