You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Lu Niu <qq...@gmail.com> on 2019/11/06 05:29:13 UTC
Limit max cpu usage per TaskManager
Hi,
When run flink application in yarn mode, is there a way to limit maximum
cpu usage per TaskManager?
I tried this application with just source and sink operator. parallelism of
source is 60 and parallelism of sink is 1. When running in default config,
there are 60 TaskManager assigned. I notice one TaskManager process cpu
usage could be 200% white the rest below 50%.
When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down to
30. and one TaskManger process cpu usage could be 600% while the rest below
50%.
Tried to set yarn.containers.vcores = 2, all tasks are in start state
forever, application is not able to turn to running state.
Best
Lu
Re: Limit max cpu usage per TaskManager
Posted by Rong Rong <wa...@gmail.com>.
Hi Lu,
Yang is right. enabling cgroup isolation is probably the one you are
looking for to control how Flink utilize the CPU resources.
One more idea is to enable DominantResourceCalculator[1] which I think
you've probably done so already.
Found an interesting read[2] about this if you would like to dig deeper.
Thanks,
Rong
[1]
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.html
[2] https://developer.ibm.com/hadoop/2017/06/30/deep-dive-yarn-cgroups/
--
Rong
On Fri, Nov 8, 2019 at 3:51 AM Yang Wang <da...@gmail.com> wrote:
> Hi Lu Niu,
>
> Yes, you could use `yarn.containers.vcores` to set the vcores of
> taskmanager. However, it could not
> guarantee that the application do not affect each other. By default, the
> yarn cluster are using cgroup
> share. That means a taskmanager could use more cpu than it allocated. When
> the machine is heavy,
> linux kernel will use cpu share as weight to control different processes.
>
> If you want to limit the taskmanager could only use as it allocated, the
> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
> is the only way. Yarn
> nodemanager will set cpu quota for each taskmanager.
>
>
>
>
> Best,
> Yang
>
> Lu Niu <qq...@gmail.com> 于2019年11月7日周四 上午1:15写道：
>
>> Hi,
>>
>> Thanks for replying! Basically I want to limit cpu usage so that
>> different application don't affect each other. What's current best
>> practice? Looks
>> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
>> is one way. How to set how many cpu resources to use? is it
>> "yarn.containers.vcores" ?
>>
>> it should be -ys not -yn in original post, sorry for the typo.
>>
>> Best
>> Lu
>>
>> On Wed, Nov 6, 2019 at 1:41 AM Yang Wang <da...@gmail.com> wrote:
>>
>>> If you want to limit the TaskManager container cpu usage, it is based on
>>> your yarn cluster configuration.
>>> By default, yarn only uses cpu share. You need to set
>>> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
>>> in yarn-site.xml of all yarn node managers.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Victor Wong <ji...@outlook.com> 于2019年11月6日周三 下午5:02写道：
>>>
>>>> Hi Lu,
>>>>
>>>>
>>>>
>>>> You can check out which operator thread causes the high CPU usage, and
>>>> set a unique slot sharing group name [1] to it to prevent too many operator
>>>> threads running in the same TM.
>>>>
>>>> Hope this will be helpful😊
>>>>
>>>>
>>>>
>>>> [1].
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>> Victor
>>>>
>>>>
>>>>
>>>> *From: *Vino Yang <ya...@gmail.com>
>>>> *Date: *Wednesday, 6 November 2019 at 4:26 PM
>>>> *To: *Lu Niu <qq...@gmail.com>
>>>> *Cc: *user <us...@flink.apache.org>
>>>> *Subject: *Re: Limit max cpu usage per TaskManager
>>>>
>>>>
>>>>
>>>> Hi Lu,
>>>>
>>>>
>>>>
>>>> When using Flink on YARN, it will rely on YARN's resource management
>>>> capabilities, and Flink cannot currently limit CPU usage.
>>>>
>>>> Also, what version of Flink do you use? As far as I know, since Flink
>>>> 1.8, the -yn parameter will not work.
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>> Vino
>>>>
>>>>
>>>>
>>>> Lu Niu <qq...@gmail.com> 于2019年11月6日周三 下午1:29写道：
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> When run flink application in yarn mode, is there a way to limit
>>>> maximum cpu usage per TaskManager?
>>>>
>>>>
>>>>
>>>> I tried this application with just source and sink operator.
>>>> parallelism of source is 60 and parallelism of sink is 1. When running in
>>>> default config, there are 60 TaskManager assigned. I notice one TaskManager
>>>> process cpu usage could be 200% white the rest below 50%.
>>>>
>>>>
>>>>
>>>> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped
>>>> down to 30. and one TaskManger process cpu usage could be 600% while the
>>>> rest below 50%.
>>>>
>>>>
>>>>
>>>> Tried to set yarn.containers.vcores = 2, all tasks are in start state
>>>> forever, application is not able to turn to running state.
>>>>
>>>>
>>>>
>>>> Best
>>>>
>>>> Lu
>>>>
>>>>
Re: Limit max cpu usage per TaskManager
Posted by Yang Wang <da...@gmail.com>.
Hi Lu Niu,
Yes, you could use `yarn.containers.vcores` to set the vcores of
taskmanager. However, it could not
guarantee that the application do not affect each other. By default, the
yarn cluster are using cgroup
share. That means a taskmanager could use more cpu than it allocated. When
the machine is heavy,
linux kernel will use cpu share as weight to control different processes.
If you want to limit the taskmanager could only use as it allocated, the
`yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
is the only way. Yarn
nodemanager will set cpu quota for each taskmanager.
Best,
Yang
Lu Niu <qq...@gmail.com> 于2019年11月7日周四 上午1:15写道：
> Hi,
>
> Thanks for replying! Basically I want to limit cpu usage so that different
> application don't affect each other. What's current best practice? Looks
> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
> is one way. How to set how many cpu resources to use? is it
> "yarn.containers.vcores" ?
>
> it should be -ys not -yn in original post, sorry for the typo.
>
> Best
> Lu
>
> On Wed, Nov 6, 2019 at 1:41 AM Yang Wang <da...@gmail.com> wrote:
>
>> If you want to limit the TaskManager container cpu usage, it is based on
>> your yarn cluster configuration.
>> By default, yarn only uses cpu share. You need to set
>> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
>> in yarn-site.xml of all yarn node managers.
>>
>>
>> Best,
>> Yang
>>
>> Victor Wong <ji...@outlook.com> 于2019年11月6日周三 下午5:02写道：
>>
>>> Hi Lu,
>>>
>>>
>>>
>>> You can check out which operator thread causes the high CPU usage, and
>>> set a unique slot sharing group name [1] to it to prevent too many operator
>>> threads running in the same TM.
>>>
>>> Hope this will be helpful😊
>>>
>>>
>>>
>>> [1].
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups
>>>
>>>
>>>
>>> Best,
>>>
>>> Victor
>>>
>>>
>>>
>>> *From: *Vino Yang <ya...@gmail.com>
>>> *Date: *Wednesday, 6 November 2019 at 4:26 PM
>>> *To: *Lu Niu <qq...@gmail.com>
>>> *Cc: *user <us...@flink.apache.org>
>>> *Subject: *Re: Limit max cpu usage per TaskManager
>>>
>>>
>>>
>>> Hi Lu,
>>>
>>>
>>>
>>> When using Flink on YARN, it will rely on YARN's resource management
>>> capabilities, and Flink cannot currently limit CPU usage.
>>>
>>> Also, what version of Flink do you use? As far as I know, since Flink
>>> 1.8, the -yn parameter will not work.
>>>
>>>
>>>
>>> Best,
>>>
>>> Vino
>>>
>>>
>>>
>>> Lu Niu <qq...@gmail.com> 于2019年11月6日周三 下午1:29写道：
>>>
>>> Hi,
>>>
>>>
>>>
>>> When run flink application in yarn mode, is there a way to limit maximum
>>> cpu usage per TaskManager?
>>>
>>>
>>>
>>> I tried this application with just source and sink operator.
>>> parallelism of source is 60 and parallelism of sink is 1. When running in
>>> default config, there are 60 TaskManager assigned. I notice one TaskManager
>>> process cpu usage could be 200% white the rest below 50%.
>>>
>>>
>>>
>>> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down
>>> to 30. and one TaskManger process cpu usage could be 600% while the rest
>>> below 50%.
>>>
>>>
>>>
>>> Tried to set yarn.containers.vcores = 2, all tasks are in start state
>>> forever, application is not able to turn to running state.
>>>
>>>
>>>
>>> Best
>>>
>>> Lu
>>>
>>>
Re: Limit max cpu usage per TaskManager
Posted by Lu Niu <qq...@gmail.com>.
Hi,
Thanks for replying! Basically I want to limit cpu usage so that different
application don't affect each other. What's current best practice? Looks
`yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
is one way. How to set how many cpu resources to use? is it
"yarn.containers.vcores" ?
it should be -ys not -yn in original post, sorry for the typo.
Best
Lu
On Wed, Nov 6, 2019 at 1:41 AM Yang Wang <da...@gmail.com> wrote:
> If you want to limit the TaskManager container cpu usage, it is based on
> your yarn cluster configuration.
> By default, yarn only uses cpu share. You need to set
> `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
> in yarn-site.xml of all yarn node managers.
>
>
> Best,
> Yang
>
> Victor Wong <ji...@outlook.com> 于2019年11月6日周三 下午5:02写道：
>
>> Hi Lu,
>>
>>
>>
>> You can check out which operator thread causes the high CPU usage, and
>> set a unique slot sharing group name [1] to it to prevent too many operator
>> threads running in the same TM.
>>
>> Hope this will be helpful😊
>>
>>
>>
>> [1].
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups
>>
>>
>>
>> Best,
>>
>> Victor
>>
>>
>>
>> *From: *Vino Yang <ya...@gmail.com>
>> *Date: *Wednesday, 6 November 2019 at 4:26 PM
>> *To: *Lu Niu <qq...@gmail.com>
>> *Cc: *user <us...@flink.apache.org>
>> *Subject: *Re: Limit max cpu usage per TaskManager
>>
>>
>>
>> Hi Lu,
>>
>>
>>
>> When using Flink on YARN, it will rely on YARN's resource management
>> capabilities, and Flink cannot currently limit CPU usage.
>>
>> Also, what version of Flink do you use? As far as I know, since Flink
>> 1.8, the -yn parameter will not work.
>>
>>
>>
>> Best,
>>
>> Vino
>>
>>
>>
>> Lu Niu <qq...@gmail.com> 于2019年11月6日周三 下午1:29写道：
>>
>> Hi,
>>
>>
>>
>> When run flink application in yarn mode, is there a way to limit maximum
>> cpu usage per TaskManager?
>>
>>
>>
>> I tried this application with just source and sink operator.
>> parallelism of source is 60 and parallelism of sink is 1. When running in
>> default config, there are 60 TaskManager assigned. I notice one TaskManager
>> process cpu usage could be 200% white the rest below 50%.
>>
>>
>>
>> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down
>> to 30. and one TaskManger process cpu usage could be 600% while the rest
>> below 50%.
>>
>>
>>
>> Tried to set yarn.containers.vcores = 2, all tasks are in start state
>> forever, application is not able to turn to running state.
>>
>>
>>
>> Best
>>
>> Lu
>>
>>
Re: Limit max cpu usage per TaskManager
Posted by Yang Wang <da...@gmail.com>.
If you want to limit the TaskManager container cpu usage, it is based on
your yarn cluster configuration.
By default, yarn only uses cpu share. You need to set
`yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true`
in yarn-site.xml of all yarn node managers.
Best,
Yang
Victor Wong <ji...@outlook.com> 于2019年11月6日周三 下午5:02写道：
> Hi Lu,
>
>
>
> You can check out which operator thread causes the high CPU usage, and set
> a unique slot sharing group name [1] to it to prevent too many operator
> threads running in the same TM.
>
> Hope this will be helpful😊
>
>
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups
>
>
>
> Best,
>
> Victor
>
>
>
> *From: *Vino Yang <ya...@gmail.com>
> *Date: *Wednesday, 6 November 2019 at 4:26 PM
> *To: *Lu Niu <qq...@gmail.com>
> *Cc: *user <us...@flink.apache.org>
> *Subject: *Re: Limit max cpu usage per TaskManager
>
>
>
> Hi Lu,
>
>
>
> When using Flink on YARN, it will rely on YARN's resource management
> capabilities, and Flink cannot currently limit CPU usage.
>
> Also, what version of Flink do you use? As far as I know, since Flink 1.8,
> the -yn parameter will not work.
>
>
>
> Best,
>
> Vino
>
>
>
> Lu Niu <qq...@gmail.com> 于2019年11月6日周三 下午1:29写道：
>
> Hi,
>
>
>
> When run flink application in yarn mode, is there a way to limit maximum
> cpu usage per TaskManager?
>
>
>
> I tried this application with just source and sink operator.
> parallelism of source is 60 and parallelism of sink is 1. When running in
> default config, there are 60 TaskManager assigned. I notice one TaskManager
> process cpu usage could be 200% white the rest below 50%.
>
>
>
> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down
> to 30. and one TaskManger process cpu usage could be 600% while the rest
> below 50%.
>
>
>
> Tried to set yarn.containers.vcores = 2, all tasks are in start state
> forever, application is not able to turn to running state.
>
>
>
> Best
>
> Lu
>
>
Re: Limit max cpu usage per TaskManager
Posted by Victor Wong <ji...@outlook.com>.
Hi Lu,
You can check out which operator thread causes the high CPU usage, and set a unique slot sharing group name [1] to it to prevent too many operator threads running in the same TM.
Hope this will be helpful😊
[1]. https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups
Best,
Victor
From: Vino Yang <ya...@gmail.com>
Date: Wednesday, 6 November 2019 at 4:26 PM
To: Lu Niu <qq...@gmail.com>
Cc: user <us...@flink.apache.org>
Subject: Re: Limit max cpu usage per TaskManager
Hi Lu,
When using Flink on YARN, it will rely on YARN's resource management capabilities, and Flink cannot currently limit CPU usage.
Also, what version of Flink do you use? As far as I know, since Flink 1.8, the -yn parameter will not work.
Best,
Vino
Lu Niu <qq...@gmail.com>> 于2019年11月6日周三 下午1:29写道：
Hi,
When run flink application in yarn mode, is there a way to limit maximum cpu usage per TaskManager?
I tried this application with just source and sink operator. parallelism of source is 60 and parallelism of sink is 1. When running in default config, there are 60 TaskManager assigned. I notice one TaskManager process cpu usage could be 200% white the rest below 50%.
When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down to 30. and one TaskManger process cpu usage could be 600% while the rest below 50%.
Tried to set yarn.containers.vcores = 2, all tasks are in start state forever, application is not able to turn to running state.
Best
Lu
Re: Limit max cpu usage per TaskManager
Posted by vino yang <ya...@gmail.com>.
Hi Lu,
When using Flink on YARN, it will rely on YARN's resource management
capabilities, and Flink cannot currently limit CPU usage.
Also, what version of Flink do you use? As far as I know, since Flink 1.8,
the -yn parameter will not work.
Best,
Vino
Lu Niu <qq...@gmail.com> 于2019年11月6日周三 下午1:29写道：
> Hi,
>
> When run flink application in yarn mode, is there a way to limit maximum
> cpu usage per TaskManager?
>
> I tried this application with just source and sink operator.
> parallelism of source is 60 and parallelism of sink is 1. When running in
> default config, there are 60 TaskManager assigned. I notice one TaskManager
> process cpu usage could be 200% white the rest below 50%.
>
> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down
> to 30. and one TaskManger process cpu usage could be 600% while the rest
> below 50%.
>
> Tried to set yarn.containers.vcores = 2, all tasks are in start state
> forever, application is not able to turn to running state.
>
> Best
> Lu
>