You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/11 19:05:36 UTC
[GitHub] [airflow] mik-laj opened a new issue #14728: Synchronization of DAG files from object storage for Helm Chart
mik-laj opened a new issue #14728:
URL: https://github.com/apache/airflow/issues/14728
Hello,
A lot of production Apache Airflow environments deployed on a cloud platform - AWS, Google uses object storage for storing DAG files. This is due to the popularity of Cloud Composer or Amazon Managed Workflows for Apache Airflow (MWAA) services that have adopted this deployment model. For some users, these services didn't their expectations (e.g. due to a version shift) and are planning to migrate to Helm Chart.
I think we should help with these migrations and prepare a solution that will allow the use of Helm Chart together with object storages. I think adding two examples to the documentation describing how to use a side-car would be enough, but I am open to discussing another idea to support these syncrhonziations.
The current documentation describes only the following [DAG file management methods](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/helm-chart/latest/manage-dags-files.html):
- Bake DAGs in Docker image
- Mounting DAGs using Git-Sync sidecar with Persistence enabled
- Mounting DAGs using Git-Sync sidecar without Persistence
- Mounting DAGs from an externally populated PVC
Best regards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14728: Synchronization of DAG files from object storages (GCS/S3) for Helm Chart
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14728:
URL: https://github.com/apache/airflow/issues/14728#issuecomment-815674725
I would like to keep this a bit simpler and allow extra sideCar containers and add example of how someone can use GCS or S3 to sync instead of adding all cloud-specific containers here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Swalloow edited a comment on issue #14728: Synchronization of DAG files from object storages (GCS/S3) for Helm Chart
Posted by GitBox <gi...@apache.org>.
Swalloow edited a comment on issue #14728:
URL: https://github.com/apache/airflow/issues/14728#issuecomment-815566866
if no one assigned, I will try it.
We need a sidecar container image to use for our example. (s3-sync, gcs-sync)
Should images be included in `chart/dockerfiles`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Swalloow commented on issue #14728: Synchronization of DAG files from object storages (GCS/S3) for Helm Chart
Posted by GitBox <gi...@apache.org>.
Swalloow commented on issue #14728:
URL: https://github.com/apache/airflow/issues/14728#issuecomment-815566866
if no one assigned, I will try it.
We need a sidecar container image to use for our example. (s3-sync, gcs-sync)
Should images be included in chart/dockerfiles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14728: Synchronization of DAG files from object storages (GCS/S3) for Helm Chart
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14728:
URL: https://github.com/apache/airflow/issues/14728#issuecomment-815568711
It's a good idea to add them there, but it's not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org