You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/30 23:53:35 UTC
[GitHub] [airflow] DmitryRusakovKodiak opened a new issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
DmitryRusakovKodiak opened a new issue #8657:
URL: https://github.com/apache/airflow/issues/8657
**Description**
Add a configuration option to use S3 as a DAGs storage/provider.
**Use case / motivation**
Currently Airflow assumes that all DAGs are stored in `/some/path/dags` folder which is present on a file system of web ui, scheduler and workers components. As a consequence DAGs are tight coupled with Airflow itself what makes the independent deployment of Airflow and DAGs quite complicated. It would be great to have a configuration option which enable Airflow to look for DAGs in a specified S3 bucket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ismailsimsek commented on issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
Posted by GitBox <gi...@apache.org>.
ismailsimsek commented on issue #8657:
URL: https://github.com/apache/airflow/issues/8657#issuecomment-623117678
#3138
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ismailsimsek edited a comment on issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
Posted by GitBox <gi...@apache.org>.
ismailsimsek edited a comment on issue #8657:
URL: https://github.com/apache/airflow/issues/8657#issuecomment-623117678
related to [AIRFLOW-2221] Create DagFetcher abstraction #3138
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #8657:
URL: https://github.com/apache/airflow/issues/8657
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8657:
URL: https://github.com/apache/airflow/issues/8657#issuecomment-671089031
Such a feature is not planned. It is recommended to set up a separate process (e.g. sidecar) that will be responsible for file synchronization. However, I would be happy if there was a guide in the documentation for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #8657: Add a configuration option to enable Airflow to look for DAGs in a specified S3 bucket.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8657:
URL: https://github.com/apache/airflow/issues/8657#issuecomment-671098446
Yep. There was an extensive discussion about it at the devlist https://lists.apache.org/thread.html/224d1e7d1b11e0b8314075f21b1b81708749f2899f4cce5af295e8a8%40%3Cdev.airflow.apache.org%3E and there is a long discussion in the wiki page: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher. I believe the current result of the discussion is that Airflow should not implement DAG fetcher for now and (as @mik-laj mentioned) it can be done with side-cars rather easily (and in the way that will be good for particular cases). Another option will be to add some options in the Helm Chart where we have side-cars already and git-sync is implemented as one. Adding and S3 side-car there might be a good idea.
@DmitryRusakovKodiak @ismailsimsek - since you are interested in it - please feel free to open an issue for Helm Chart extension (or even donate one) or re-open a discussion in the devlist, but for now I am closing this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org