You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Evgeniy Lyutikov <eb...@avito.ru> on 2023/03/13 09:55:53 UTC
Kubernetes operator set container resources and limits
Hi all
Is there any way to specify different values for resources and limits for a jobmanager container?
The problem is that sometimes kubernetes kills the jobmanager container because it exceeds the memory consumption.
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Tue, 07 Mar 2023 18:06:01 +0700
Finished: Fri, 10 Mar 2023 23:20:54 +0700
What I tried to do:
1. added the 'jobmanager.memory.process.size' parameter to flinkConfiguration with a value less than the allocated resources for the container, but after launch, the value of this parameter is set to the amount of memory allocated to the container.
2. I tried to set resources and limits through the jobmanager pod template, but for the running container, the values again return to the same values from jobManager.resource FlinkDeployment manifest parameter
Kubernetes operator 1.2.0 and Flink 1.14.4
________________________________
"This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом."
Re: Kubernetes operator set container resources and limits
Posted by Evgeniy Lyutikov <eb...@avito.ru>.
I want to set different values for resources and limits.
________________________________
От: Andrew Otto <ot...@wikimedia.org>
Отправлено: 13 марта 2023 г. 17:50
Кому: Evgeniy Lyutikov
Копия: user@flink.apache.org
Тема: Re: Kubernetes operator set container resources and limits
Hi,
> return to the same values from jobManager.resource FlinkDeployment manifest parameter
I believe this is the correct way; using jobManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23jobmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C7fbac6d919574aa2b24208db23b0c679%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143014385143446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7P4y2mYvJ%2FgQb7fnFUg71FNq6X%2BGDAcvgXQkfFB7zv8%3D&reserved=0> and taskManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23taskmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C7fbac6d919574aa2b24208db23b0c679%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143014385143446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BDwtRhhMM%2FVDWq7KtGlbx4fl%2F901CGrnqus9LUVozko%3D&reserved=0> in the FlinkDeployment<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23flinkdeploymentspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C7fbac6d919574aa2b24208db23b0c679%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143014385143446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SMEzhiCBAYbZ%2BSWKocyPy51hsew2dFW6vFpE%2BqhXVXg%3D&reserved=0>.
Is there a reason you can't change the resources values there? I don't think you should need to do so with podTemplate.
On Mon, Mar 13, 2023 at 9:56 AM Evgeniy Lyutikov <eb...@avito.ru>> wrote:
Hi all
Is there any way to specify different values for resources and limits for a jobmanager container?
The problem is that sometimes kubernetes kills the jobmanager container because it exceeds the memory consumption.
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Tue, 07 Mar 2023 18:06:01 +0700
Finished: Fri, 10 Mar 2023 23:20:54 +0700
What I tried to do:
1. added the 'jobmanager.memory.process.size' parameter to flinkConfiguration with a value less than the allocated resources for the container, but after launch, the value of this parameter is set to the amount of memory allocated to the container.
2. I tried to set resources and limits through the jobmanager pod template, but for the running container, the values again return to the same values from jobManager.resource FlinkDeployment manifest parameter
Kubernetes operator 1.2.0 and Flink 1.14.4
________________________________
“This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”
Re: [EXT] Re: Kubernetes operator set container resources and limits
Posted by Evgeniy Lyutikov <eb...@avito.ru>.
Thanks, this looks like what I need
kubernetes.jobmanager.memory.limit-factor: '1.5'
resources:
limits:
cpu: "1"
memory: 4608Mi
requests:
cpu: "1"
memory: 3Gi
________________________________
От: Jasmin Redzepovic <ja...@happening.xyz>
Отправлено: 13 марта 2023 г. 19:31:07
Кому: Andrew Otto; Evgeniy Lyutikov
Копия: user@flink.apache.org
Тема: Re: [EXT] Re: Kubernetes operator set container resources and limits
Hi,
You can set the following properties in the flinkConfiguration inside your .yaml file:
* kubernetes.jobmanager.cpu.limit-factor
- kubernetes.jobmanager.memory.limit-factor
- kubernetes.taskmanager.cpu.limit-factor
- kubernetes.taskmanager.memory.limit-factor
Example:
jobManager:
replicas: 1
resource:
memory: "1000m"
cpu: 0.2
taskManager:
resource:
memory: "1000m"
cpu: 0.3
flinkConfiguration:
taskmanager.numberOfTaskSlots: "1"
kubernetes.jobmanager.cpu.limit-factor: "2"
kubernetes.jobmanager.memory.limit-factor: "1.2"
kubernetes.taskmanager.cpu.limit-factor: "2"
kubernetes.taskmanager.memory.limit-factor: "1.5"
Limits for jobmanager would then be:
* cpu: 0.4
* memory: 1200
Hope this helps,
Jasmin
From: Andrew Otto <ot...@wikimedia.org>
Date: Monday, 13 March 2023 at 11:51
To: Evgeniy Lyutikov <eb...@avito.ru>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: [EXT] Re: Kubernetes operator set container resources and limits
[CAUTION] This email comes from an external organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi,
> return to the same values from jobManager.resource FlinkDeployment manifest parameter
I believe this is the correct way; using jobManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23jobmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074737299342%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HZa0bLWRea6U4cxaKQVgit0qzEYQH8K8JzumAxt0f14%3D&reserved=0> and taskManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23taskmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074737455578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=slC1Bv5QD%2Bx%2FVCoMJFoeXcXUyHZMm9RMzEUgLhHkoXg%3D&reserved=0> in the FlinkDeployment<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23flinkdeploymentspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074738080447%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dXnL8FofG7ZE3%2BLWDJ9VbVc1iqqBhTo5TX7M%2BNQg%2FhU%3D&reserved=0>.
Is there a reason you can't change the resources values there? I don't think you should need to do so with podTemplate.
On Mon, Mar 13, 2023 at 9:56 AM Evgeniy Lyutikov <eb...@avito.ru>> wrote:
Hi all
Is there any way to specify different values for resources and limits for a jobmanager container?
The problem is that sometimes kubernetes kills the jobmanager container because it exceeds the memory consumption.
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Tue, 07 Mar 2023 18:06:01 +0700
Finished: Fri, 10 Mar 2023 23:20:54 +0700
What I tried to do:
1. added the 'jobmanager.memory.process.size' parameter to flinkConfiguration with a value less than the allocated resources for the container, but after launch, the value of this parameter is set to the amount of memory allocated to the container.
2. I tried to set resources and limits through the jobmanager pod template, but for the running container, the values again return to the same values from jobManager.resource FlinkDeployment manifest parameter
Kubernetes operator 1.2.0 and Flink 1.14.4
________________________________
“This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”
Re: Kubernetes operator set container resources and limits
Posted by Andrew Otto <ot...@wikimedia.org>.
Hi,
> return to the same values from jobManager.resource
FlinkDeployment manifest parameter
I believe this is the correct way; using jobManager.resources
<https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobmanagerspec>
and taskManager.resources
<https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#taskmanagerspec>
in
the FlinkDeployment
<https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeploymentspec>
.
Is there a reason you can't change the resources values there? I don't
think you should need to do so with podTemplate.
On Mon, Mar 13, 2023 at 9:56 AM Evgeniy Lyutikov <eb...@avito.ru>
wrote:
> Hi all
> Is there any way to specify different values for resources and limits for
> a jobmanager container?
> The problem is that sometimes kubernetes kills the jobmanager container
> because it exceeds the memory consumption.
>
>
> Last State: Terminated
> Reason: OOMKilled
> Exit Code: 137
> Started: Tue, 07 Mar 2023 18:06:01 +0700
> Finished: Fri, 10 Mar 2023 23:20:54 +0700
>
> What I tried to do:
> 1. added the 'jobmanager.memory.process.size' parameter to
> flinkConfiguration with a value less than the allocated resources for the
> container, but after launch, the value of this parameter is set to the
> amount of memory allocated to the container.
> 2. I tried to set resources and limits through the jobmanager pod template,
> but for the running container, the values again return to the same values
> from jobManager.resource FlinkDeployment manifest parameter
>
> Kubernetes operator 1.2.0 and Flink 1.14.4
>
>
>
> * ------------------------------ *“This message contains confidential
> information/commercial secret. If you are not the intended addressee of
> this message you may not copy, save, print or forward it to any third party
> and you are kindly requested to destroy this message and notify the sender
> thereof by email.
> Данное сообщение содержит конфиденциальную информацию/информацию,
> являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом
> данного сообщения, Вы не вправе копировать, сохранять, печатать или
> пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и
> уведомить об этом отправителя электронным письмом.”
>