You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcampos87 (via GitHub)" <gi...@apache.org> on 2025/03/25 14:58:17 UTC

[I] Not able to deploy logs persistence [airflow]

rcampos87 opened a new issue, #48243:
URL: https://github.com/apache/airflow/issues/48243

   ### Apache Airflow version
   
   2.10.5
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I have Airflow running in EKS with CeleryKubernetesExecutor and Im unable to set logs persistence to use AWS EBS CSI driver. The PVC errors with 
   
   ```
     Normal   Provisioning          48s (x9 over 2m57s)  ebs.csi.aws.com_ebs-csi-controller-746b78965f-4tlrd_a43e92a0-9d72-4dfa-8a26-1bb48b74a19c  External provisioner is provisioning volume for claim "airflow/airflow-logs"
     Warning  ProvisioningFailed    48s (x9 over 2m57s)  ebs.csi.aws.com_ebs-csi-controller-746b78965f-4tlrd_a43e92a0-9d72-4dfa-8a26-1bb48b74a19c  failed to provision volume with StorageClass "gp2": rpc error: code = InvalidArgument desc = Volume capabilities not supported
   ```
   when trying to assign `gp2` as storage class.
   
   Since logs is not able to be provisioned, it holds all the deployment and all other components remain as pending.
   
   
   
   
   ### What you think should happen instead?
   
   Like for workers, redis, or triggerer persistence, a PV should automatically be provisioned for the PVC given the AWS EBS CSI driver is correctly setup with proper service account.
   
   ### How to reproduce
   
   Setup airflow in EKS using helm charts, enable AWS EBS CSI driver with correctly service account, and enable logs persistence when deploying:
   
   logs:
   
     persistence:
       enabled: true
       storageClassName: "gp2"
       size: 50Gi
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   
   workers:
     persistence:
       enabled: true
       storageClassName: "gp2"
       size: 50Gi
   
   redis:
     persistence:
       enabled: true
       storageClassName: "gp2"
       size: 10Gi
   
   triggerer:
     persistence:
       enabled: true
       storageClassName: "gp2"
       size: 10Gi
   
   # Logs persistence does not work currently, it complains that volume capabilities are not supported.
   logs:
     persistence:
       enabled: true
       storageClassName: "gp2"
       size: 50Gi
   
   ### Anything else?
   
   I have experienced the same issue in SUSE rancher as well using the vsphere csi driver where workers, redis, and triggerer persistence works but logs persistence does not.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to deploy logs persistence [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2751558022

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to deploy logs persistence [airflow]

Posted by "y0zg (via GitHub)" <gi...@apache.org>.
y0zg commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2776044068

   I use `storageClassName: efs`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to deploy logs persistence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #48243:
URL: https://github.com/apache/airflow/issues/48243#issuecomment-2764761881

   Logs needs write-many capability,. Apparently gp2 does not have it. You should choose storage writer that have write many capatiblity - see for example here https://stackoverflow.com/questions/73220730/error-con-pods-in-azure-k8s-volume-capability-not-supported
   
   Consult Amazon support if you do not know which storage driver to use.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Not able to deploy logs persistence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #48243: Not able to deploy logs persistence
URL: https://github.com/apache/airflow/issues/48243


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org