Sync DAGs folder with Git

[Gabriele Di Bernardo <gd...@bol.com>]

Hello everyone,

I am relatively new to Airflow and I am not sure if this is the right place to ask questions.

We set up a simple Kubernetes pod running a Docker image that contains Airflow. We want the DAGs to be synced from a Git repository. The first idea we had was to define an extra dag responsible for syncing the dag folder with the remote repository. However, I saw that in the airflow.cfg file there are some entries related to Git:

# Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
git_repo =
git_branch =
git_subpath =
# Use git_user and git_password for user authentication or git_ssh_key_secret_name and git_ssh_key_secret_key
# for SSH authentication
git_user =
git_password =
git_sync_root = /git
git_sync_dest = repo
# Mount point of the volume if git-sync is being used.
# i.e. {AIRFLOW_HOME}/dags
git_dags_folder_mount_point =

So I was wondering what is the best way to keep in sync Airflow with a remote repository that contains DAGs and if this functionality is already integrated natively in Airflow!

Thank you so much!

Best,


Gabriele

Re: Sync DAGs folder with Git

[Chao-Han Tsai <mi...@gmail.com>]

If you are using KubernetesExecutor, you can configure a git-sync side car
that fetches files from remote git.

https://airflow.apache.org/kubernetes.html#kubernetes-executor

On Thu, Apr 11, 2019 at 9:00 AM Gabriele Di Bernardo <gd...@bol.com>
wrote:

> Hello everyone,
>
> I am relatively new to Airflow and I am not sure if this is the right
> place to ask questions.
>
> We set up a simple Kubernetes pod running a Docker image that contains
> Airflow. We want the DAGs to be synced from a Git repository. The first
> idea we had was to define an extra dag responsible for syncing the dag
> folder with the remote repository. However, I saw that in the airflow.cfg
> file there are some entries related to Git:
>
> # Git credentials and repository for DAGs mounted via Git (mutually
> exclusive with volume claim)
> git_repo =
> git_branch =
> git_subpath =
> # Use git_user and git_password for user authentication or
> git_ssh_key_secret_name and git_ssh_key_secret_key
> # for SSH authentication
> git_user =
> git_password =
> git_sync_root = /git
> git_sync_dest = repo
> # Mount point of the volume if git-sync is being used.
> # i.e. {AIRFLOW_HOME}/dags
> git_dags_folder_mount_point =
>
> So I was wondering what is the best way to keep in sync Airflow with a
> remote repository that contains DAGs and if this functionality is already
> integrated natively in Airflow!
>
> Thank you so much!
>
> Best,
>
>
> Gabriele
>


-- 

Chao-Han Tsai