[jira] [Created] (AIRFLOW-5570) Allow to store DAGs on GIT

["Adam Trump (Jira)" <ji...@apache.org>]

Adam Trump created AIRFLOW-5570:
-----------------------------------

             Summary: Allow to store DAGs on GIT
                 Key: AIRFLOW-5570
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5570
             Project: Apache Airflow
          Issue Type: New Feature
          Components: core, DAG, scheduler
    Affects Versions: 1.10.5
            Reporter: Adam Trump


Today it is possible to store the DAGs only on a specific folder, which is configurable.

I'd like to suggest to allow storing the DAGs on GIT.

This feature comes from the fact that Airflow already have this functionallity - under the KubernetesExecutor.

Thus, it shouldn't take much time moving it out of that executor, to the Airflow scheduler.

When you run Airflow as a cluster on different machines, since all of the components (webserver, scheduler and worker) needs access to the DAGs file, there is always a need of accessable mount.

It can be very helpful if we could store the DAGs on GIT, and configure all components to pull from that GIT repo.

Just as today, the scheduler should pull every X seconds from that repository.

When you want to add new DAGs, you simply need to push it to that repo.

It will also allow us to have some version control over DAGs, which could be useful for keeping cleaner code and order.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)