You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "RonaldGalea (via GitHub)" <gi...@apache.org> on 2023/09/08 16:25:28 UTC
[GitHub] [airflow] RonaldGalea opened a new issue, #34219: Deploy multiple (celery) workers with the Helm Chart
RonaldGalea opened a new issue, #34219:
URL: https://github.com/apache/airflow/issues/34219
### Description
Currently, the Helm [Chart](https://github.com/apache/airflow/blob/main/chart/values.yaml) supports a single Celery worker deployment. The CeleryExecutor however, supports multiple [queues](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html#queues). Ideally, the Helm Chart should allow deploying multiple Celery workers (different images, different resource allocations) to be able to use queues to send tasks to specialised workers (as already highlighted in the [docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html#queues))
### Use case/motivation
It is often the case that tasks might require different hardware resources/different environments. For instance, if I have task A, which requires 1GB of memory, and task B, which requires 10GB, it's inefficient to run them both on the same Celery worker deployment.
Having multiple deployments would allow using queues to distribute tasks with finer granularity.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "littlebtc (via GitHub)" <gi...@apache.org>.
littlebtc commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-2955471446
I am also thinking about assigning custom images per worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "VergeDX (via GitHub)" <gi...@apache.org>.
VergeDX commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-3036159365
You can use [HELM Docs - Post Rendering](https://helm.sh/docs/topics/advanced/#post-rendering) to add/copy/modify the Official Helm Chart's output, but this is still a hack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-3191460472
I think that for the simple case of same worker images / service account etc this should be a fairly simple change? It might be best to support at least this one.
There are use cases that can benefit from that. specifically the common use case of setting different parallelism per worker type like: 10 for small tasks worker and 3 for heavy tasks worker. So we just need to support multiple workers and queues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RonaldGalea commented on issue #34219: Deploy multiple (celery) workers with the Helm Chart
Posted by "RonaldGalea (via GitHub)" <gi...@apache.org>.
RonaldGalea commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-1712484775
> It would be great if we add this feature to the helm chart, but it won't be an easy task, because to implement it we will need to add a loop for each celery template and support two configuration sources: a new value of type list and the existing one for backward compatibility. We also need to update KEDA to do the same and update its queries to scale the different celery clusters based on tasks' queues.
>
> We can also use the existing value as default configurations for the different clusters, and override only the config provided in the list.
>
> cc: @jedcunningham wdyt?
>
> @RonaldGalea Would you like to work on it?
Haven't done templating before but I'd be happy to give it a try. I'll see what I can learn this weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "guhuajun (via GitHub)" <gi...@apache.org>.
guhuajun commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-2119648252
Greetings,
I am evaluating the features in Airflow project. I am still new to the Airflow project. But It's good to see the old friend Celery. It is the project I am familiar with since 2015.
After reading your words, maybe you can start with simple one, [Play Celery with django](https://docs.celeryq.dev/en/stable/django/index.html). It's easily start two workers with docker-compose and having some fun moments. Since we already have [DagRun](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dagrun/index.html#module-airflow.models.dagrun), once the dag files are synced/replicated to all workers, we should have a success.
Good luck! `airflow pools join` will be a major milestone for this project. :)
> I've had a look at how the helm chart is structured and my approach would be as follows:
>
> First, identify the minimum options that need to be configurable for the feature to be useful. The ones I'm seeing are:
>
> * worker image
>
> * command & args
>
> * replicas
>
> * resources
>
> * autoscaling (keda)
>
>
> Since setting the image specifically for workers is not yet configurable - that would be a first standalone PR.
>
> Next, I would leave the current `workers:` key exactly as it is (both for backward compatibility and for the fact that there will always be at least one worker type) and introduce an `additional-celery-workers:` key, where the aforementioned options can be specified and the rest will be the same as for the default worker type.
>
> I believe most worker-related components can be exactly the same for all workers (at least for now), so they should not require changes:
>
> * service account
>
> * service
>
> * network policy
>
> * DB connection setup for keda (e.g. pgbouncer network policy)
>
>
> What does need to change is Keda - we will have to create additional ScaledObjects to reflect the addition of Deployments/StatefulSets. What I'm a bit unsure of is the [query](https://github.com/apache/airflow/blob/main/chart/values.yaml#L585). Currently, it just lists everything from a table named `task_instance` - does this table contain queue-related info/can it be easily added?
>
> Let me know if the overall approach sounds reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #34219: Deploy multiple (celery) workers with the Helm Chart
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-1711931285
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #34219: Deploy multiple (celery) workers with the Helm Chart
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-1712116930
It would be great if we add this feature to the helm chart, but it won't be an easy task, because to implement it we will need to add a loop for each celery template and support two configuration sources: a new value of type list and the existing one for backward compatibility.
We also need to update KEDA to do the same and update its queries to scale the different celery clusters based on tasks' queues.
We can also use the existing value as default configurations for the different clusters, and override only the config provided in the list.
cc: @jedcunningham wdyt?
@RonaldGalea Would you like to work on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RonaldGalea commented on issue #34219: Deploy multiple (celery) workers with the Helm Chart
Posted by "RonaldGalea (via GitHub)" <gi...@apache.org>.
RonaldGalea commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-1712919682
I've had a look at how the helm chart is structured and my approach would be as follows:
First, identify the minimum options that need to be configurable for the feature to be useful. The ones I'm seeing are:
- worker image
- command & args
- replicas
- resources
- autoscaling (keda)
Since setting the image specifically for workers is not yet configurable - that would be a first standalone PR.
Next, I would leave the current `workers:` key exactly as it is (both for backward compatibility and for the fact that there will always be at least one worker type) and introduce an `additional-celery-workers:` key, where the aforementioned options can be specified and the rest will be the same as for the default worker type.
I believe most worker-related components can be exactly the same for all workers (at least for now), so they should not require changes:
- service account
- service
- network policy
- DB connection setup for keda (e.g. pgbouncer network policy)
What does need to change is Keda - we will have to create additional ScaledObjects to reflect the addition of Deployments/StatefulSets. What I'm a bit unsure of is the [query](https://github.com/apache/airflow/blob/main/chart/values.yaml#L585). Currently, it just lists everything from a table named `task_instance` - does this table contain queue-related info/can it be easily added?
Let me know if the overall approach sounds reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #34219: Deploy multiple (celery) workers with the Helm Chart
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-1712223766
At one point we had an issue (or maybe a PR) that talked about the complexities of doing exactly this, but I can't find it right now.
I don't see (or recall) any blockers. It's just not a trivial thing to add. It would be great to get it added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "MagicFrogSJTU (via GitHub)" <gi...@apache.org>.
MagicFrogSJTU commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-3077691097
Can I deploy multiple airflow helm release to enable multiple celery workers with helm chart?
* the first one fully deploy all componets
* the other ones only deploy workers
* all release use the same redis and postgresql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "ihnokim (via GitHub)" <gi...@apache.org>.
ihnokim commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-2814845828
I was wondering if there’s been any recent progress on this issue. This feature would be incredibly helpful for my use case. If things are currently delayed due to a busy schedule, would it be okay if I kindly offered to work on it and submit a pull request, perhaps based on @RonaldGalea ’s suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34219:
URL: https://github.com/apache/airflow/issues/34219#issuecomment-2815259126
> I was wondering if there’s been any recent progress on this issue. This feature would be incredibly helpful for my use case. If things are currently delayed due to a busy schedule, would it be okay if I kindly offered to work on it and submit a pull request, perhaps based on [@RonaldGalea](https://github.com/RonaldGalea) ’s suggestions?
Sure. that's how it works here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Deploy multiple (celery) workers with the Helm Chart [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed issue #34219: Deploy multiple (celery) workers with the Helm Chart
URL: https://github.com/apache/airflow/issues/34219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org