You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcampos87 (via GitHub)" <gi...@apache.org> on 2025/03/25 14:58:17 UTC
[I] Not able to deploy logs persistence [airflow]
rcampos87 opened a new issue, #48243:
URL: https://github.com/apache/airflow/issues/48243
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have Airflow running in EKS with CeleryKubernetesExecutor and Im unable to set logs persistence to use AWS EBS CSI driver. The PVC errors with
```
Normal Provisioning 48s (x9 over 2m57s) ebs.csi.aws.com_ebs-csi-controller-746b78965f-4tlrd_a43e92a0-9d72-4dfa-8a26-1bb48b74a19c External provisioner is provisioning volume for claim "airflow/airflow-logs"
Warning ProvisioningFailed 48s (x9 over 2m57s) ebs.csi.aws.com_ebs-csi-controller-746b78965f-4tlrd_a43e92a0-9d72-4dfa-8a26-1bb48b74a19c failed to provision volume with StorageClass "gp2": rpc error: code = InvalidArgument desc = Volume capabilities not supported
```
when trying to assign `gp2` as storage class.
Since logs is not able to be provisioned, it holds all the deployment and all other components remain as pending.
### What you think should happen instead?
Like for workers, redis, or triggerer persistence, a PV should automatically be provisioned for the PVC given the AWS EBS CSI driver is correctly setup with proper service account.
### How to reproduce
Setup airflow in EKS using helm charts, enable AWS EBS CSI driver with correctly service account, and enable logs persistence when deploying:
logs:
persistence:
enabled: true
storageClassName: "gp2"
size: 50Gi
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
workers:
persistence:
enabled: true
storageClassName: "gp2"
size: 50Gi
redis:
persistence:
enabled: true
storageClassName: "gp2"
size: 10Gi
triggerer:
persistence:
enabled: true
storageClassName: "gp2"
size: 10Gi
# Logs persistence does not work currently, it complains that volume capabilities are not supported.
logs:
persistence:
enabled: true
storageClassName: "gp2"
size: 50Gi
### Anything else?
I have experienced the same issue in SUSE rancher as well using the vsphere csi driver where workers, redis, and triggerer persistence works but logs persistence does not.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Not able to deploy logs persistence [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2751558022
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Not able to deploy logs persistence [airflow]
Posted by "y0zg (via GitHub)" <gi...@apache.org>.
y0zg commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2776044068
I use `storageClassName: efs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Not able to deploy logs persistence [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2764761881
Logs needs write-many capability,. Apparently gp2 does not have it. You should choose storage writer that have write many capatiblity - see for example here https://stackoverflow.com/questions/73220730/error-con-pods-in-azure-k8s-volume-capability-not-supported
Consult Amazon support if you do not know which storage driver to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Not able to deploy logs persistence [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #48243: Not able to deploy logs persistence
URL: https://github.com/apache/airflow/issues/48243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org