You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Diogo Franco (JIRA)" <ji...@apache.org> on 2018/03/16 10:48:00 UTC

[jira] [Created] (AIRFLOW-2221) Fill up DagBad from remote locations

Diogo Franco created AIRFLOW-2221:
-------------------------------------

             Summary: Fill up DagBad from remote locations
                 Key: AIRFLOW-2221
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2221
             Project: Apache Airflow
          Issue Type: New Feature
          Components: configuration, core
    Affects Versions: Airflow 2.0
            Reporter: Diogo Franco
            Assignee: Diogo Franco
             Fix For: Airflow 2.0


The ability to fill up the DagBag from remote locations (HDFS, S3...) seems to be deemed useful, e.g. facilitating deployment processes.

This JIRA is to propose an implementation of a *DagFetcher* abstraction on the DagBag, where the collect_dags method can delegate the walking to a *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even *TarballInS3DagFetcher*.

This was briefly discussed in [this mailing list thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E]

I'm happy to start work on this and provide an initial implementation for review.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)