You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/12/27 09:25:00 UTC
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote
locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729469#comment-16729469 ]
ASF GitHub Bot commented on AIRFLOW-2221:
-----------------------------------------
stale[bot] commented on pull request #3138: [AIRFLOW-2221] Create DagFetcher abstraction
URL: https://github.com/apache/incubator-airflow/pull/3138
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Fill up DagBag from remote locations
> ------------------------------------
>
> Key: AIRFLOW-2221
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2221
> Project: Apache Airflow
> Issue Type: New Feature
> Components: configuration, core
> Affects Versions: 2.0.0
> Reporter: Diogo Franco
> Assignee: Diogo Franco
> Priority: Major
> Fix For: 2.0.0
>
>
> The ability to fill up the DagBag from remote locations (HDFS, S3...) seems to be deemed useful, e.g. facilitating deployment processes.
> This JIRA is to propose an implementation of a *DagFetcher* abstraction on the DagBag, where the collect_dags method can delegate the walking to a *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even *TarballInS3DagFetcher*.
> This was briefly discussed in [this mailing list thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E]
> I'm happy to start work on this and provide an initial implementation for review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)