[jira] [Updated] (AIRFLOW-2221) Fill up DagBag from remote locations

["Diogo Franco (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Diogo Franco updated AIRFLOW-2221:
----------------------------------
    Summary: Fill up DagBag from remote locations  (was: Fill up DagBad from remote locations)

> Fill up DagBag from remote locations
> ------------------------------------
>
>                 Key: AIRFLOW-2221
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2221
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: configuration, core
>    Affects Versions: Airflow 2.0
>            Reporter: Diogo Franco
>            Assignee: Diogo Franco
>            Priority: Major
>             Fix For: Airflow 2.0
>
>
> The ability to fill up the DagBag from remote locations (HDFS, S3...) seems to be deemed useful, e.g. facilitating deployment processes.
> This JIRA is to propose an implementation of a *DagFetcher* abstraction on the DagBag, where the collect_dags method can delegate the walking to a *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even *TarballInS3DagFetcher*.
> This was briefly discussed in [this mailing list thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E]
> I'm happy to start work on this and provide an initial implementation for review.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)