You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/19 10:04:21 UTC
[GitHub] [airflow] Dr-Denzy opened a new pull request #15937: Improve Helm Chart Git-Sync documentation
Dr-Denzy opened a new pull request #15937:
URL: https://github.com/apache/airflow/pull/15937
Mounting DAGs from a private Github repo using Git-Sync sidecar is
quite complex because of the several steps required and the many
other moving parts.
The PR aims to ameliorate some of these pain points so that users can
have a smoother experience when mounting their DAGs from private repos
on Github.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15937: Improve Helm Chart Git-Sync documentation
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15937:
URL: https://github.com/apache/airflow/pull/15937#issuecomment-843959713
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #15937: Improve Helm Chart Git-Sync documentation
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #15937:
URL: https://github.com/apache/airflow/pull/15937#discussion_r635212429
##########
File path: docs/helm-chart/manage-dags-files.rst
##########
@@ -136,3 +137,65 @@ In this approach, Airflow will read the DAGs from a PVC which has ``ReadOnlyMany
--set dags.persistence.enabled=true \
--set dags.persistence.existingClaim=my-volume-claim
--set dags.gitSync.enabled=false
+
+Mounting DAGs from a private Github repo using Git-Sync sidecar
+---------------------------------------------------------------
+Create a private repo on Github if you have not created one already.
+
+Then create your ssh keys:
+
+.. code-block:: bash
+
+ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
+
+and add the public key to your private repo (under ``Settings > Deploy keys``).
+
+Now, you have to create a Kubernetes Secret object with which the Git-Sync sidecar will authenticate when
+fetching or syncing your DAGs from your private Github repo.
+
+You have to convert the private ssh key to a base64. You can convert the private ssh key file like so:
+
+.. code-block:: bash
+
+ base64 <my-private-ssh-key> -w 0 > temp.txt
+
+Then copy the string from the ``temp.txt`` file and add it to a yaml file to create your secret object.
+For example, ``my-ssh-secret.yaml`` should look like this:
+
+.. code-block:: yaml
+
+ apiVersion: v1
+ kind: Secret
+ metadata:
+ name: airflow-ssh-secret
+ data:
+ gitSshKey: '<base64-converted-ssh-private-key>'
+
+And from a terminal then run:
+
+.. code-block:: bash
+
+ kubectl create -f my-ssh-secret.yaml --namespace <your-airflow-namespace>
+
+You can easily create a yaml file to override values of interest in the ``values.yaml`` file. In this example, I will
+create a yaml file called ``override-values.yaml`` to override values in the ``values.yaml`` file.
+
+.. code-block:: yaml
+
+ dags:
+ gitSync:
+ enabled: true
+ repo: ssh://git@github.com/<username>/<private-repo-name>.git
+ branch: <branch-name>
+ subPath: ""
+ sshKeySecret: airflow-ssh-secret
+
+
+Finally, from the context of your Airflow Helm chart directory, you can install Airflow:
+
+.. code-block:: bash
+
+ helm install airflow --namespace <your-airflow-namespace> . -f override-values.yaml
Review comment:
This would likely change when the chart is released?.
Supposing the user added the released chart to helm with:
`helm repo add apache-airflow-chart path/to/chart`.
The install should be:
`helm install airflow --namespace <your-airflow-namespace> apache-airflow-chart/airflow -f override-values.yaml`
I don't know the best way to add it to this doc, maybe we should merge this and update the doc when the Chart is released? @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on pull request #15937: Improve Helm Chart Git-Sync documentation
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on pull request #15937:
URL: https://github.com/apache/airflow/pull/15937#issuecomment-844029696
> Love it. I think it's very much needed.
>
> But I think you need to fix the license indentation :)
Oh yeah! Thanks, Jarek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #15937: Improve Helm Chart Git-Sync documentation
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #15937:
URL: https://github.com/apache/airflow/pull/15937#discussion_r635231614
##########
File path: docs/helm-chart/manage-dags-files.rst
##########
@@ -136,3 +137,65 @@ In this approach, Airflow will read the DAGs from a PVC which has ``ReadOnlyMany
--set dags.persistence.enabled=true \
--set dags.persistence.existingClaim=my-volume-claim
--set dags.gitSync.enabled=false
+
+Mounting DAGs from a private Github repo using Git-Sync sidecar
+---------------------------------------------------------------
+Create a private repo on Github if you have not created one already.
+
+Then create your ssh keys:
+
+.. code-block:: bash
+
+ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
+
+and add the public key to your private repo (under ``Settings > Deploy keys``).
+
+Now, you have to create a Kubernetes Secret object with which the Git-Sync sidecar will authenticate when
+fetching or syncing your DAGs from your private Github repo.
+
+You have to convert the private ssh key to a base64. You can convert the private ssh key file like so:
+
+.. code-block:: bash
+
+ base64 <my-private-ssh-key> -w 0 > temp.txt
+
+Then copy the string from the ``temp.txt`` file and add it to a yaml file to create your secret object.
+For example, ``my-ssh-secret.yaml`` should look like this:
+
+.. code-block:: yaml
+
+ apiVersion: v1
+ kind: Secret
+ metadata:
+ name: airflow-ssh-secret
+ data:
+ gitSshKey: '<base64-converted-ssh-private-key>'
+
+And from a terminal then run:
+
+.. code-block:: bash
+
+ kubectl create -f my-ssh-secret.yaml --namespace <your-airflow-namespace>
+
+You can easily create a yaml file to override values of interest in the ``values.yaml`` file. In this example, I will
+create a yaml file called ``override-values.yaml`` to override values in the ``values.yaml`` file.
+
+.. code-block:: yaml
+
+ dags:
+ gitSync:
+ enabled: true
+ repo: ssh://git@github.com/<username>/<private-repo-name>.git
+ branch: <branch-name>
+ subPath: ""
+ sshKeySecret: airflow-ssh-secret
+
+
+Finally, from the context of your Airflow Helm chart directory, you can install Airflow:
+
+.. code-block:: bash
+
+ helm install airflow --namespace <your-airflow-namespace> . -f override-values.yaml
Review comment:
Good point, yes.
I will create a PR while releasing the chart to make sure we are consistent everywhere in the Helm Chart docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #15937: Improve Helm Chart Git-Sync documentation
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #15937:
URL: https://github.com/apache/airflow/pull/15937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org