You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jedcunningham (via GitHub)" <gi...@apache.org> on 2026/07/30 15:53:12 UTC
[I] Chart scheduler persistence should not depend on celery configuration [airflow]
jedcunningham opened a new issue, #70784:
URL: https://github.com/apache/airflow/issues/70784
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
main
### What happened and how to reproduce it?
I noticed what the logic to determine if a scheduler is a deployment or a statefulset now looks at the `.Values.workers.celery.persistence.enabled `. This used to be the more general `.Values.workers.persistence.enabled`, but now this coupling does not make sense.
This was a miss when celery was split out of the general workers section.
### What you think should happen instead?
I haven't spent much time thinking about it, but maybe we just need a new `scheduler.persistence` to control it? Or `workers.local.persistence`?
Whoever picks this up should explore the possible options.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
main (development)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Chart scheduler persistence should not depend on celery configuration [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #70784:
URL: https://github.com/apache/airflow/issues/70784#issuecomment-5182099197
> I think scheduler.persistence would be cleaner than workers.local.persistence. While either could work functionally, the StatefulSet and PVC are part of the scheduler workload, so organizing the setting under scheduler seems more consistent with the resource hierarchy.
I'm not fully agreeing with this. The persistence thing is connected to the executor rather than the scheduler itself, even if the executor lives alongside the scheduler. Looking at that, I would say that `workers.local.persistence` is more correct, looking at the things underneath. `workers.celery` is `CeleryExecutor`, `workers.kubernetes` is `KubernetesExecutor` and `workers.local` would be e.g. `LocalExecutor`. `scheduler. persistence` abstracts executor type, which I think is important, and our user base here is mostly platform developers, DevOps people, etc. so the consistency within the values should be kept I think.
Happy to hear what others think about it.
> Should the new setting fall back to workers.celery.persistence for backward compatibility, or can we simply remove the existing coupling?
On `main` - no, on `chart/v1-2x-test `- yes. As this changes the parameters a bit, I think it should be backported to the 1.2x line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Chart scheduler persistence should not depend on celery configuration [airflow]
Posted by "daesung9511 (via GitHub)" <gi...@apache.org>.
daesung9511 commented on issue #70784:
URL: https://github.com/apache/airflow/issues/70784#issuecomment-5137305298
I'd like to work on this issue.
I think `scheduler.persistence` would be cleaner than `workers.local.persistence`. While either could work functionally, the StatefulSet and PVC are part of the scheduler workload, so organizing the setting under `scheduler` seems more consistent with the resource hierarchy.
Should the new setting fall back to `workers.celery.persistence` for backward compatibility, or can we simply remove the existing coupling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org