You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "warrenronsiek (via GitHub)" <gi...@apache.org> on 2023/03/08 04:20:22 UTC
[GitHub] [airflow] warrenronsiek opened a new issue, #29969: Helm chart not generating persistent volumes
warrenronsiek opened a new issue, #29969:
URL: https://github.com/apache/airflow/issues/29969
### Official Helm Chart version
1.8.0 (latest released)
### Apache Airflow version
2
### Kubernetes Version
1.25
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### What happened
Running the standard install on my kubernetes cluster doesn't create the required peristent volumes to bind to the persistent volume claims.
I ran:
```helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace```
Describing the resources yeilds:
```
> kubectl -n airflow get pods
NAME READY STATUS RESTARTS AGE
airflow-postgresql-0 0/1 Pending 0 8s
airflow-redis-0 0/1 Pending 0 8s
airflow-run-airflow-migrations-9cccj 0/1 ContainerCreating 0 8s
airflow-scheduler-8579dd79df-w2lq4 0/2 Init:0/1 0 8s
airflow-statsd-5b4964646f-887pt 0/1 Running 0 8s
airflow-triggerer-79677ff998-6x4tv 0/1 Init:0/1 0 8s
airflow-webserver-6c595fcb6d-2bjnm 0/1 Init:0/1 0 8s
airflow-worker-0 0/2 Pending 0 8s
> kubectl -n airflow get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-airflow-postgresql-0 Pending 18s
logs-airflow-worker-0 Pending 18s
redis-db-airflow-redis-0 Pending 18s
> kubectl -n airflow get pv
No resources found
```
The system will remain in this state, with redis, postgres, and workers failing to create with the status:
``` Warning FailedScheduling 15s default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.```
When I describe the persistent cliams I get:
```
> kubectl -n airflow describe pvc data-airflow-postgresql-0
Name: data-airflow-postgresql-0
Namespace: airflow
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=airflow
app.kubernetes.io/name=postgresql
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: airflow-postgresql-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 2s (x42 over 10m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
```
```
> kubectl -n airflow describe pvc logs-airflow-worker-0
Name: logs-airflow-worker-0
Namespace: airflow
StorageClass:
Status: Pending
Volume:
Labels: component=worker
release=airflow
tier=airflow
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: airflow-worker-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 76s (x42 over 11m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
```
```
> kubectl -n airflow describe pvc redis-db-airflow-redis-0
Name: redis-db-airflow-redis-0
Namespace: airflow
StorageClass:
Status: Pending
Volume:
Labels: component=redis
release=airflow
tier=airflow
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: airflow-redis-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 2m10s (x42 over 12m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
```
### What you think should happen instead
Airflow should create persistent volumes to bind to the appropriate claims.
### How to reproduce
The issue doesn't happen in Minikube, so not sure how to reproduce without access to my home Kubernetes. I have successfully provisioned persistent volumes and claims for other services, which run successfully in the same cluster.
Best I can do is to [install the Rancher kubernetes](https://docs.rke2.io/install/quickstart) and follow the same install steps from above.
### Anything else
This happens every time I try to install the chart. I've also tried hacking it by creating volumes for the claims to bind to, but the claims don't bind to them.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] CharlesFrank commented on issue #29969: Helm chart not generating persistent volumes
Posted by "CharlesFrank (via GitHub)" <gi...@apache.org>.
CharlesFrank commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1480553062
I'm seeing the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29969: Helm chart not generating persistent volumes
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1459416411
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] warrenronsiek commented on issue #29969: Helm chart not generating persistent volumes
Posted by "warrenronsiek (via GitHub)" <gi...@apache.org>.
warrenronsiek commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1463007381
They don't specify a storage class. An example that works:
```
apiVersion: v1
kind: PersistentVolume
metadata:
name: mlflow-pg-volume
labels:
app: mlflow
spec:
capacity:
storage: "3Gi"
accessModes:
- ReadWriteOnce
hostPath:
path: "{{.Values.storagePath}}/postgres"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mlflow-postgres
labels:
app: mlflow
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "3Gi"
selector:
matchLabels:
name: mlflow-pg-volume
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] samywang92 commented on issue #29969: Helm chart not generating persistent volumes
Posted by "samywang92 (via GitHub)" <gi...@apache.org>.
samywang92 commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1540973564
For anyone who might be stumbling across this issue in the future, I was running across the same issue upon upgrading from Kubernetes 1.22 due to EOS. I have tested on all versions after 1.23+. After failing on all the new versions, I reverted back to 1.22 and Airflow was able to successfully install and create the PV that is missing.
My resolution is exclusive to AWS environment but after looking up AWS EBS for 1.23 and onward, they state that we now have to manually install the EBS drivers on 1.23 and onward. This resolved my issue after following their documentation and creating the correct roles/installing the drivers.
References can be found here: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/945
and here: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
Cheers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29969: Helm chart not generating persistent volumes
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29969: Helm chart not generating persistent volumes
URL: https://github.com/apache/airflow/issues/29969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dnskr commented on issue #29969: Helm chart not generating persistent volumes
Posted by "dnskr (via GitHub)" <gi...@apache.org>.
dnskr commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1460537411
> I have successfully provisioned persistent volumes and claims for other services, which run successfully in the same cluster.
Which storage class is used for these PVCs/PVs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29969: Helm chart not generating persistent volumes
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29969:
URL: https://github.com/apache/airflow/issues/29969#issuecomment-1484215605
I guess you need to specify defailt storage class for your k8S.
https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/
You can also specify storage class name in values.
Closing unless there are other findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org