You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SKisContent (via GitHub)" <gi...@apache.org> on 2024/08/21 04:33:08 UTC

[PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

SKisContent opened a new pull request, #41628:
URL: https://github.com/apache/airflow/pull/41628

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #28880
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   This adds a kubernetes_resources setting to the pod template so that those using the CeleryKubernetesExecutor can specify different resource settings for the Celery workers and the Kubernetes workers. In practice the Celery workers will be few in number and probably need more CPU and memory since they will handle multiple tasks, while the Kubernetes workers will be many in number and need much less CPU and memory. However, with the current one-size-fits-both approach, Kubernetes worker pods have to wait to get scheduled because of high initial resource requests.
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #41628: Add kubernetes_resources to differentiate from Celery worker resources
URL: https://github.com/apache/airflow/pull/41628


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2427963831

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "SKisContent (via GitHub)" <gi...@apache.org>.
SKisContent commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2301208182

   @nevcohen I may not be understanding your question. The proposed change allows for a separate set of resource specifications for the Kubernetes worker pod. In my case, my values.yaml file under the workers section contains this:
   ```
     resources:
       limits:
         memory: 8500Mi
       requests:
         cpu: 2000m
         memory: 4500Mi
   
     # Additional resource requests for the KubernetesExecutor pods
     kubernetes_resources:
      limits:
       memory: 512Mi
      requests:
       cpu: 200m
       memory: 128Mi
   ```
   
   Thus the Celery worker pods are scheduled on nodes that have at least 2 vCPUs available, and the Kubernetes worker pods get scheduled on nodes that have 0.2 vCPUs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2306121185

   OK, I understand, agree with you for now.
   
   In the future, when more people use hybrid executor, we will need something like this:
   
   ```yaml
   worker:
       ...
       resurce:
           ...
       ...
   podTemplate:
       ...
       resurce:
           ...
       ...
   ```
   And the pod-template will look like this:
   ```yaml
       ...
         resources: {{- toYaml (or .Values.podTemplate.resources .Values.workers.resources) | nindent 8 }}
      ...
   ```
   
   And continue like this for most of the other values of the worker that should also be in the pod-template.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2301191858

   Now that there is a hybrid executor, we might want an entirely separate set of values ​​for the `pod-template`? Or it will just be duplication of values?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "SKisContent (via GitHub)" <gi...@apache.org>.
SKisContent commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2306073357

   @nevcohen  Are you asking why not use the existing values.podTemplate section? Writing out a whole template seems overkill, requires the user to spend a lot more time on the task, and if in the future the default template changes, the user would have to be aware of that and update the inline template.
   
   The solution in this PR seems like a quick fix for a problem that several people are facing (see related issue), I'm not aware of other parts of the manifest that people want to change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "SKisContent (via GitHub)" <gi...@apache.org>.
SKisContent commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2329185652

   Hi, @dstandish  @hussein-awala @jedcunningham
   Would it be possible to get a yea/nay on this PR so that my org can make a decision on how to go forward?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add kubernetes_resources to differentiate from Celery worker resources [airflow]

Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on PR #41628:
URL: https://github.com/apache/airflow/pull/41628#issuecomment-2301245615

   I mean why not add `values.podTemplate`? Guess in the future there will be more differences that won't be the same in the workers of the celery executor and in the pods of the k8s executor.
   
   I'm just asking, maybe I'm wrong..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org