You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/26 21:35:33 UTC
[GitHub] [airflow] zorzigio opened a new issue #15540: git-sync with persistent volume claim
zorzigio opened a new issue #15540:
URL: https://github.com/apache/airflow/issues/15540
**Apache Airflow version**: 2.0.2
**Kubernetes version**: Client Version: v1.17.4, Server Version: v1.18.14
**Cloud provider or hardware configuration**: Azure
**What happened**:
When using git-sync with persistence enabled, the scheduler pod fails to start.
The command
```bash
kubectl logs airflow2-scheduler-79d7f74f46-sgd6h -c git-sync
```
gives
```bash
INFO: detected pid 1, running init handler
I0426 20:32:38.953393 11 main.go:321] "level"=0 "msg"="starting up" "args"=["/git-sync"] "pid"=11
I0426 20:32:38.957866 11 main.go:574] "level"=0 "msg"="cloning repo" "origin"="git@github.com:Cybrik/airflow2-dags.git" "path"="/git"
E0426 20:32:40.633376 11 main.go:347] "msg"="failed to sync repo, aborting" "error"="error running command: exit status 128: { stdout: \"\", stderr: \"Cloning into '/git'...\\nerror: chmod on /git/.git/config.lock failed: Operation not permitted\\nfatal: could not set 'core.filemode' to 'false'\\n\" }"
```
**What you expected to happen**:
Airflow deploying correctly
**How to reproduce it**:
I am using the Helm chart from [`apache/airflow`](https://github.com/apache/airflow/tree/v2-0-stable/chart)
create `values.yaml` with the following content
```yaml
dags:
persistence:
enabled: true
existingClaim: dags
gitSync:
enabled: true
repo: "git@github.com:Company/repo.git"
branch: main
subPath: ""
sshKeySecret: airflow-ssh-secret
```
and a persistent volume claim as follows
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: dags
labels:
tier: airflow
release: airflow2
heritage: "Helm"
spec:
accessModes:
- "ReadWriteMany"
resources:
requests:
storage: "1Gi"
storageClassName: "azurefile"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy edited a comment on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
Dr-Denzy edited a comment on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-828610586
@zorzigio I could not reproduce this issue. For PV and PVC I used my local file system. I had my dags in a private repo on Github accessed via ssh. Changes made to the dags were picked by git-sync as expected.
Below is a portion of `values.yaml` configuration used.
```bash
dags:
persistence:
enabled: true
size: 3Gi
storageClassName: manual
accessMode: ReadWriteOnce
existingClaim:
gitSync:
enabled: true
excludeWebserver: false
repo: ssh://git@github.com:<company-name>/<repo-name>.git
branch: main
maxFailures: 0
subPath: ""
sshKeySecret: airflow-ssh-secret-name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-828714799
Thanks @Dr-Denzy for looking into this.
I guess the error I am seeing is related to the `azurefile` storageClass which seems to be the difference between my config and yours.
In Azure, the default available classes I see are (given by `kubectl get sc`)
```bash
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION
azurefile kubernetes.io/azure-file Delete Immediate true
azurefile-premium kubernetes.io/azure-file Delete Immediate true
default (default) kubernetes.io/azure-disk Delete Immediate true
managed kubernetes.io/azure-disk Delete WaitForFirstConsumer true
managed-premium kubernetes.io/azure-disk Delete Immediate true
```
and in particular the details of the one I was using are (`kubectl get sc azurefile -o yaml`)
```bash
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"azurefile"},"parameters":{"skuName":"Standard_LRS"},"provisioner":"kubernetes.io/azure-file"}
creationTimestamp: "2020-03-27T00:57:04Z"
labels:
kubernetes.io/cluster-service: "true"
name: azurefile
resourceVersion: "1265389"
selfLink: /apis/storage.k8s.io/v1/storageclasses/azurefile
uid: 412c26c7-3368-49f5-b06d-3c81bf4d7125
parameters:
skuName: Standard_LRS
provisioner: kubernetes.io/azure-file
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
what is the definition of the storageClass `manual` you are using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-827213032
cc @Dr-Denzy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-876818674
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-827438730
I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy edited a comment on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
Dr-Denzy edited a comment on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-828610586
@zorzigio I could not reproduce this issue. For PV and PVC I used my local file system. I had my dags in a private repo on Github accessed via ssh. Changes made to the dags were picked by git-sync as expected.
Below is a portion of `values.yaml` configuration used.
```yaml
dags:
persistence:
enabled: true
size: 3Gi
storageClassName: manual
accessMode: ReadWriteOnce
existingClaim:
gitSync:
enabled: true
excludeWebserver: false
repo: ssh://git@github.com:<company-name>/<repo-name>.git
branch: main
maxFailures: 0
subPath: ""
sshKeySecret: airflow-ssh-secret-name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-828610586
@zorzigio I could not reproduce this issue. For PV and PVC I used my local file system. I had my dags in a private repo on Github accessed via ssh. Changes made to the dags were picked by git-sync as expected.
Below is a portion of `values.yaml` configuration used.
```bash
dags:
persistence:
enabled: true
size: 3Gi
storageClassName: manual
accessMode: ReadWriteOnce
existingClaim:
gitSync:
enabled: true
excludeWebserver: false
repo: ssh://git@github.com:company-name/git-sync-dags.git
branch: main
maxFailures: 0
subPath: ""
sshKeySecret: airflow-ssh-secret-name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy edited a comment on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
Dr-Denzy edited a comment on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-828610586
@zorzigio I could not reproduce this issue. For PV and PVC I used my local file system. I had my dags in a private repo on Github accessed via ssh. Changes made to the dags were picked by git-sync as expected.
Below is a portion of `values.yaml` configuration used.
```bash
dags:
persistence:
enabled: true
size: 3Gi
storageClassName: manual
accessMode: ReadWriteOnce
existingClaim:
gitSync:
enabled: true
excludeWebserver: false
repo: ssh://git@github.com:<company-name>/git-sync-dags.git
branch: main
maxFailures: 0
subPath: ""
sshKeySecret: airflow-ssh-secret-name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #15540:
URL: https://github.com/apache/airflow/issues/15540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-881084539
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-883727318
@jedcunningham finally I got round to try this.
I created a custom storage class using
```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: storage-class-test
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Standard_LRS
```
and a persistent volume claim using
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: dags-test
labels:
tier: airflow
release: airflow
heritage: "Helm"
spec:
accessModes:
- "ReadWriteMany"
resources:
requests:
storage: "1Gi"
storageClassName: "storage-class-test"
```
Then in the `values.yaml`
```yaml
dags:
persistence:
enabled: true
existingClaim: dags-test
gitSync:
enabled: true
repo: "git@github.com:Company/dags.git"
branch: main
dest: "dags_repo"
subPath: ""
sshKeySecret: airflow-ssh-secret
```
I am still getting the same error
```bash
>kubectl logs airflow-scheduler-bfcc8668d-fh9kb -c git-sync
INFO: detected pid 1, running init handler
I0720 21:32:53.092295 10 main.go:495] "level"=0 "msg"="starting up" "args"=["/git-sync"] "pid"=10
I0720 21:32:53.100652 10 main.go:784] "level"=0 "msg"="cloning repo" "origin"="git@github.com:Company/dags.git" "path"="/git"
E0720 21:32:56.598086 10 main.go:168] "msg"="too many failures, aborting" "error"="Run(git clone --no-checkout -b main --depth 1 git@github.com:Company/dags.git /git): exit status 128: { stdout: \"\", stderr: \"Cloning into '/git'...\\nerror: chmod on /git/.git/config.lock failed: Operation not permitted\\nfatal: could not set 'core.filemode' to 'false'\\n\" }" "failCount"=0
```
Any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-832860602
@zorzigio, maybe creating a custom storage class and setting `dir_mode`, `file_mode`, and/or `uid`, `gid` will help with your issue?
https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv#create-a-storage-class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-883779031
Solved the issue adding `noperm` to the `StorageClass` `mountOptions`
```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: storage-class-test
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
- noperm
parameters:
skuName: Standard_LRS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-855715246
Thanks @jedcunningham, I will try this and update here on the result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-855152325
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15540: git-sync with persistent volume claim
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15540:
URL: https://github.com/apache/airflow/issues/15540#issuecomment-827161284
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org