Re: [EXT] Re: Kubernetes operator set container resources and limits

[Evgeniy Lyutikov <eb...@avito.ru>]

Thanks, this looks like what I need


kubernetes.jobmanager.memory.limit-factor: '1.5'

    resources:
      limits:
        cpu: "1"
        memory: 4608Mi
      requests:
        cpu: "1"
        memory: 3Gi



________________________________
От: Jasmin Redzepovic <ja...@happening.xyz>
Отправлено: 13 марта 2023 г. 19:31:07
Кому: Andrew Otto; Evgeniy Lyutikov
Копия: user@flink.apache.org
Тема: Re: [EXT] Re: Kubernetes operator set container resources and limits

Hi,

You can set the following properties in the flinkConfiguration inside your .yaml file:

  *   kubernetes.jobmanager.cpu.limit-factor

-          kubernetes.jobmanager.memory.limit-factor
-          kubernetes.taskmanager.cpu.limit-factor
-          kubernetes.taskmanager.memory.limit-factor
Example:
jobManager:
    replicas: 1
    resource:
      memory: "1000m"
      cpu: 0.2
  taskManager:
    resource:
      memory: "1000m"
      cpu: 0.3

flinkConfiguration:

    taskmanager.numberOfTaskSlots: "1"

    kubernetes.jobmanager.cpu.limit-factor: "2"

    kubernetes.jobmanager.memory.limit-factor: "1.2"

    kubernetes.taskmanager.cpu.limit-factor: "2"

    kubernetes.taskmanager.memory.limit-factor: "1.5"

Limits for jobmanager would then be:

  *   cpu: 0.4
  *   memory: 1200

Hope this helps,
Jasmin


From: Andrew Otto <ot...@wikimedia.org>
Date: Monday, 13 March 2023 at 11:51
To: Evgeniy Lyutikov <eb...@avito.ru>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: [EXT] Re: Kubernetes operator set container resources and limits

[CAUTION] This email comes from an external organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi,

> return to the same values from jobManager.resource FlinkDeployment manifest parameter
I believe this is the correct way; using jobManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23jobmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074737299342%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HZa0bLWRea6U4cxaKQVgit0qzEYQH8K8JzumAxt0f14%3D&reserved=0> and taskManager.resources<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23taskmanagerspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074737455578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=slC1Bv5QD%2Bx%2FVCoMJFoeXcXUyHZMm9RMzEUgLhHkoXg%3D&reserved=0> in the FlinkDeployment<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2Fdocs%2Fcustom-resource%2Freference%2F%23flinkdeploymentspec&data=05%7C01%7Ceblyutikov%40avito.ru%7C8c0623ac6abf4190fcb308db23bed329%7Caf0e07b3b90b472392e63fab11dd5396%7C0%7C0%7C638143074738080447%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dXnL8FofG7ZE3%2BLWDJ9VbVc1iqqBhTo5TX7M%2BNQg%2FhU%3D&reserved=0>.

Is there a reason you can't change the resources values there?  I don't think you should need to do so with podTemplate.



On Mon, Mar 13, 2023 at 9:56 AM Evgeniy Lyutikov <eb...@avito.ru>> wrote:
Hi all
Is there any way to specify different values for resources and limits for a jobmanager container?
The problem is that sometimes kubernetes kills the jobmanager container because it exceeds the memory consumption.


Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Tue, 07 Mar 2023 18:06:01 +0700
  Finished:     Fri, 10 Mar 2023 23:20:54 +0700

What I tried to do:
1. added the 'jobmanager.memory.process.size' parameter to flinkConfiguration with a value less than the allocated resources for the container, but after launch, the value of this parameter is set to the amount of memory allocated to the container.
2. I tried to set resources and limits through the jobmanager pod template, but for the running container, the values again return to the same values from jobManager.resource FlinkDeployment manifest parameter

Kubernetes operator 1.2.0 and Flink 1.14.4



________________________________
“This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”