You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/14 08:07:12 UTC
[GitHub] [airflow] ecerulm opened a new pull request #16425: Allow configuration of pod resources in helm chart
ecerulm opened a new pull request #16425:
URL: https://github.com/apache/airflow/pull/16425
Currently it's possible to configure some of the pod resources via the helm chart values like for example the scheduler pod main container but it's not possible for the `scheduler-log-groomer` container.
In deployments with `ResourceQuotas` it's desirable to be able to control the `limits` of each container specifically to avoid pods reserve way too much cpu and memory and eat up the quota unnecessarily. Even though the current helm chart allows to add a `LimitRange` that defines the default `limits` that is not enough.
This introduces the ability to provide a specific resource blocks for
* `pod_template_file`: for the pods started by the `KubernetesPodOperator` itself . ([pod_template_file doc](https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html#pod-template-file))
* pgbouncer's `metrics-exporter`
* scheduler's `scheduler-log-groomer`
* webserver's initContainer `wait-for-airflow-migrations`
* worker's `worker-log-groomer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16425: Allow configuration of pod resources in helm chart
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16425:
URL: https://github.com/apache/airflow/pull/16425#issuecomment-861883368
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on a change in pull request #16425: Allow configuration of pod resources in helm chart
Posted by GitBox <gi...@apache.org>.
ecerulm commented on a change in pull request #16425:
URL: https://github.com/apache/airflow/pull/16425#discussion_r651925189
##########
File path: chart/values.schema.json
##########
@@ -2798,6 +2875,24 @@
}
}
}
+ },
+ "initContainerResources": {
Review comment:
Yes it's probably a good idea just to use the main container resources for the initContainers. I'll change that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #16425: Allow configuration of pod resources in helm chart
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #16425:
URL: https://github.com/apache/airflow/pull/16425#discussion_r651331183
##########
File path: chart/files/pod-template-file.kubernetes-helm-yaml
##########
@@ -54,6 +54,8 @@ spec:
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
name: base
ports: []
+ resources:
+{{ toYaml .Values.podTemplateResources | indent 8 }}
Review comment:
```suggestion
{{ toYaml .Values.workers.resources | indent 8 }}
```
For other things we've used the `workers` parameters in the pod_template_file instead of duplicating them. Is there a reason not to in this specific case?
##########
File path: chart/values.schema.json
##########
@@ -2798,6 +2875,24 @@
}
}
}
+ },
+ "initContainerResources": {
Review comment:
I think we should be more specific with this naming. Maybe `waitForMigrationsInitContainerResources`? Open to other options.
That said, init containers may not need separate resources anyways? Can we just use the resources for the given component, e.g. `scheduler.resources` for the scheduler? Based on how [k8s handles pod resources](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resources), once the init containers finish you'll just need those same resources for the main component anyways (meaning, if setting lower init resources 'fixes' a resourcing issue you're flying too close to sun already)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org