You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/18 17:57:24 UTC
[GitHub] [airflow] volvocarsmatt opened a new issue, #27114: Problems mounting an externally populated file share for the DAGs with the Helm chart
volvocarsmatt opened a new issue, #27114:
URL: https://github.com/apache/airflow/issues/27114
### Official Helm Chart version
1.7.0 (latest released)
### Apache Airflow version
2.4.1
### Kubernetes Version
v1.23.8
### Helm Chart configuration
Fresh install for testing
```shell
helm install airflow apache-airflow/airflow --set dags.persistence.enabled=true \
--set dags.persistence.existingClaim=dag-static-file-system \
--set dags.gitSync.enabled=false
```
### Docker Image customisations
Default
### What happened
I tried to mount a static externally populated file share for the DAGs using the Helm chart, but for some reason I get strange error messages. I included my steps, maybe I misconfigured something?
### What you think should happen instead
The PV should have been successfully mounted and the pods should be running
### How to reproduce
This is what I tried mostly [based on this](https://learn.microsoft.com/en-us/azure/aks/azure-files-volume).
1. Create a new file share in Terraform
```tf
resource "azurerm_storage_share" "my_static_file_share" {
name = "dag-file-share-id"
storage_account_name = azurerm_storage_account.static_storage_acc.name
quota = 5
}
```
2. "Wrap" said file share in a new persistent volume to ensure we can point to specific static volume rather than dynamically allocating a generic volume.
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: dag-static-file-system
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
storageClassName: azurefile-csi
csi:
driver: file.csi.azure.com
readOnly: false
volumeHandle: dag-static-file-system # make sure this volumeid is unique in the cluster
volumeAttributes:
shareName: dag-file-share-id
nodeStageSecretRef:
name: azure-secret
namespace: airflow
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- nosharesock
- nobrl
```
3. "Wrap" the newly created persistent volume with a persistent volume claim
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dag-static-file-system
spec:
accessModes:
- ReadWriteMany
storageClassName: azurefile-csi
volumeName: dag-static-file-system. # same as previous
resources:
requests:
storage: 2Gi
```
4. Apply with `kubectl apply -f my_pv.yaml`
5. From the [Helm chart tutorial](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-an-externally-populated-pvc), you then install Helm chart with some additional parameters
```shell
helm install airflow apache-airflow/airflow --set dags.persistence.enabled=true \
--set dags.persistence.existingClaim=dag-static-file-system \
--set dags.gitSync.enabled=false
```
> "MountVolume.MountDevice failed for volume \"dag-static-file-system\" : rpc error: code = Internal desc = volume(dag-static-storage-volume) mount
> Warning: The state file either has no outputs defined, or all the defined outputs are empty. Please define an output in your configuration with the `output` keyword and run `terraform refresh` for it to become available. If you are using interpolation, please verify the interpolated value is not empty. You can use the `terraform console` command to assist. Mount failed: exit status 1
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] volvocarsmatt closed issue #27114: Problems mounting an externally populated file share for the DAGs with the Helm chart
Posted by GitBox <gi...@apache.org>.
volvocarsmatt closed issue #27114: Problems mounting an externally populated file share for the DAGs with the Helm chart
URL: https://github.com/apache/airflow/issues/27114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] volvocarsmatt commented on issue #27114: Problems mounting an externally populated file share for the DAGs with the Helm chart
Posted by GitBox <gi...@apache.org>.
volvocarsmatt commented on issue #27114:
URL: https://github.com/apache/airflow/issues/27114#issuecomment-1282853772
I don't know what happened. Perhaps I am going insane. But after upgrading to Kubernetes version `1.24.6`, it now works again as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27114: Problems mounting an externally populated file share for the DAGs with the Helm chart
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27114:
URL: https://github.com/apache/airflow/issues/27114#issuecomment-1282794151
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org