You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/05 16:18:58 UTC
[GitHub] [airflow] dolfinus opened a new pull request, #28125: Set limits for XCOM pod
dolfinus opened a new pull request, #28125:
URL: https://github.com/apache/airflow/pull/28125
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
My k8s namespace has specified `ResourceQuota`, and thus every container should be created with both `requests` and `limits` resources set.
I've tried to run simple dag like this:
```python
import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s
with DAG(
dag_id="example_k8s_operator",
schedule=None,
start_date=datetime.datetime(2021, 1, 1),
catchup=False,
tags=["example"],
) as dag:
task = KubernetesPodOperator(
namespace="onetl-demo-user-namespace",
name="hello-dry-run",
image="debian",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
task_id="dry_run_demo",
container_resources=k8s.V1ResourceRequirements(
requests={"memory": "250M", "cpu": "100m"},
limits={"memory": "250M", "cpu": "100m"},
),
do_xcom_push=True,
)
```
but got an exception:
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 135, in run_pod_async
body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7469, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
_preload_content, _request_timeout, _host)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 281, in POST
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"hello-dry-run-8863acf43ec04fa0b293f81ca981d701\" is forbidden: failed quota: portalquota: must specify limits.cpu,limits.memory","reason":"Forbidden","details":{"name":"hello-dry-run-8863acf43ec04fa0b293f81ca981d701","kind":"pods"},"code":403}
```
Full pod spec from logs:
```json
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {},
"labels": {
"foo": "bar",
"dag_id": "example_k8s_operator",
"task_id": "dry_run_demo",
"run_id": "manual__2022-12-05T151152.3955870000-b2ac559bc",
"kubernetes_pod_operator": "True",
"try_number": "1",
"airflow_version": "2.4.1",
"airflow_kpo_in_cluster": "True"
},
"name": "hello-dry-run-8863acf43ec04fa0b293f81ca981d701",
"namespace": "onetl-demo-user-namespace"
},
"spec": {
"affinity": {},
"containers": [
{
"args": [
"echo",
"10"
],
"command": [
"bash",
"-cx"
],
"env": [],
"envFrom": [],
"image": "debian",
"name": "base",
"ports": [],
"resources": {
"limits": {
"memory": "250M",
"cpu": "100m"
},
"requests": {
"memory": "250M",
"cpu": "100m"
}
},
"volumeMounts": [
{
"mountPath": "/airflow/xcom",
"name": "xcom"
}
]
},
{
"command": [
"sh",
"-c",
"trap \"exit 0\" INT; while true; do sleep 1; done;"
],
"image": "alpine",
"name": "airflow-xcom-sidecar",
"resources": {
"requests": {
"cpu": "1m"
}
},
"volumeMounts": [
{
"mountPath": "/airflow/xcom",
"name": "xcom"
}
]
}
],
"hostNetwork": false,
"imagePullSecrets": [],
"initContainers": [],
"nodeSelector": {},
"restartPolicy": "Never",
"securityContext": {},
"tolerations": [],
"volumes": [
{
"emptyDir": {},
"name": "xcom"
}
]
}
}
```
Container with pod command has the same resources I've set using `KubernetesPodOperator.container_resources`, but sidecar container with XCOM fetcher is created without limits, causing this exception. As a result I cannot get XCOM of any task created using this operator.
Here I've set limits as `1m` for cpu and `10m` for memory. IMHO, `sleep 1` command does not need more resources, even `10m` maybe too much.
I haven't add any options to change these values (like in #26766), it seems that currently there is no way to change command executed by this container.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus commented on pull request #28125: Set limits for XCOM pod
Posted by "dolfinus (via GitHub)" <gi...@apache.org>.
dolfinus commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1482726600
Ok, got your point.
I've added `xcom_sidecar_container_resources` field to `extras` field of k8s connection, just like `xcom_sidecar_container_image`. User can set here resources in JSON syntax, and they will be passed to sidecar container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1383250971
Well you removed two reviewers that likely would have been much better to review it (not sure why) so you limited your chances to get it reviewed ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1347226880
Surely - you can download the logs from the job - just click on the "Summary" and you will be able to downlad the logs from failed jobs - it is an artifact that is attached to the job (this is a complet export of all logs from failed kind).
Also you can very easily reproduce what the tests are doing locally by following our documentation: https://github.com/apache/airflow/blob/main/TESTING.rst#typical-testing-pattern-for-kubernetes-tests is a step-by-step wizard like guide and above that you will find explanation of what happens in each step. Also breeze has `k9s` integration included so that you can explore the logs and everything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
dolfinus commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1383100276
Any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1383251255
I am not sure we want to set arbitrary limits for the pods @dstandish @jedcunningham WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1347229516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28125: Set limits for XCOM pod
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #28125:
URL: https://github.com/apache/airflow/pull/28125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1436006357
Now. There are some confflicts to resolve - It's been some time it was out (and there is some backlog of issues/PRs so this takes longer than usual) - can you please rebase and fix the conflicts though.
I have not looked in detail and we've not heard from @jedcunningham and @dstandish, but I think the numbers here are too magical and forcing them on everyone is a bit too much to solve your case @dolfinus. You can add a feature to optionally add resource quota for those PODS with default being "not set" - that is the best way of handling it. I hardly imagine we will merge a change that we have a fixed memory used for the XCom POD (and it's backwards incompatible too).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
dolfinus commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1346472920
Is there any way to get failing pod log or something like that? I don't understand why container is starting using one executor but not other ones, and these executors are changing all the time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus closed pull request #28125: Set limits for XCOM pod
Posted by "dolfinus (via GitHub)" <gi...@apache.org>.
dolfinus closed pull request #28125: Set limits for XCOM pod
URL: https://github.com/apache/airflow/pull/28125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1338923797
Tests are failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1383263572
> > Well you removed two reviewers
>
> There was a "Requested changes" status, and I've clicked on it to inform that changes were made. I don't have permissions to change list of reviewers, maybe this is some kind of Github bug.
Maybe. I've added them back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
dolfinus commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1383261388
> Well you removed two reviewers
There was a "Requested changes" status, and I've clicked on it to inform that changes were made. I don't have permissions to change list of reviewers, maybe this is some kind of Github bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dolfinus commented on pull request #28125: Set limits for XCOM pod
Posted by GitBox <gi...@apache.org>.
dolfinus commented on PR #28125:
URL: https://github.com/apache/airflow/pull/28125#issuecomment-1353066725
XCOM sidecar container was killed because of OOM, even on memory limit 10Mi and 20Mi. I didn't expect that bash + sleep requires more RAM than 1Mi. Increased limit to 50Mi, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org