You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ktaborski (via GitHub)" <gi...@apache.org> on 2025/11/17 14:33:45 UTC

[PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

ktaborski opened a new pull request, #58398:
URL: https://github.com/apache/airflow/pull/58398

   Decouple StatefulSet deployment from PVC creation in worker configuration. Previously, workers.persistence.enabled controlled both StatefulSet usage and PVC creation, making it impossible to use StatefulSet without volumes.
   
   Changes:
   - Modify volume template logic to use logs.persistence.enabled for PVC
   - Allow StatefulSet workers with emptyDir when workers.persistence.enabled is true but logs.persistence.enabled is false
   - Maintain backward compatibility: default behavior unchanged
   
   This enables stable network identity for workers without persistent storage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

Posted by "ktaborski (via GitHub)" <gi...@apache.org>.
ktaborski commented on PR #58398:
URL: https://github.com/apache/airflow/pull/58398#issuecomment-3545797675

   Hi,
   advantage of using statefulset is, that I can have real-time logs in UI from workers (when they are running). When finished, logs are fetched from s3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

Posted by "ktaborski (via GitHub)" <gi...@apache.org>.
ktaborski commented on PR #58398:
URL: https://github.com/apache/airflow/pull/58398#issuecomment-3551641098

   @jscheffl works like a charm, thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

Posted by "ktaborski (via GitHub)" <gi...@apache.org>.
ktaborski commented on PR #58398:
URL: https://github.com/apache/airflow/pull/58398#issuecomment-3547182229

   It seems, that changes are not backward compatible, I need to rework them


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #58398:
URL: https://github.com/apache/airflow/pull/58398#issuecomment-3549524669

   > Hi, advantage of using statefulset is, that I can have real-time logs in UI from workers (when they are running). When finished, logs are fetched from s3.
   
   Yeah also had a bit of challenge, when deploying in K8s as "deployment" and not staefulset then you need to adjust one property tp fix nameserver lookup and deployment instances have no DNS.
   
   Personally I am not favoring statefulesets too much as a rolling release really takes long as K8s upgrades one-by-one. With a deployment all is in parallel with a defined volume.
   
   Would it maybe make sense to prevent such problems as alternative option to use config https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable setting it to `airflow.utils.net.get_host_ip_address`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Helm chart to support StatefulSet workers without PVC [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #58398:
URL: https://github.com/apache/airflow/pull/58398#issuecomment-3543831551

   Can you describe a reason why you want to have a statefulset?
   I assume (missing the legacy reason) there was an idea behind that if no persistence that a deployment is used and not a statefulset.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org