[broker]loadBalancerMemoryResourceWeight config default value optimiz

[jiangxinwei <ji...@huya.com>]

https://github.com/apache/pulsar/pull/13464

 

Pasted below for quoting convenience. -------

 

Motivation

memory.percentUsage() value is depends on the gc trigger moment, this value
has a random affect on ThresholdShedder loadBalancer strategy. it is a
meaningless dimension

the usage data from one broker

 
<https://user-images.githubusercontent.com/5210343/147305803-45f42c19-ec9c-4
89d-9f98-b44205a43f8c.png> 

solution

set loadBalancerMemoryResourceWeight=0.0



-- 
Avast 防毒软件已对此电子邮件执行病毒检查。
https://www.avast.com/antivirus

Re: [broker]loadBalancerMemoryResourceWeight config default value optimiz

[Haiting Jiang <ji...@apache.org>]

Agreed on the default settings would cause some unnecessary bundle shedding.
But I believe that disable it by default is not the best option here.

I would like share some preliminary thoughts on this,

1) About memory usage definition, we can try to use the memory usage just after last GC 
instead of instant memory usage,  like com.sun.management.GcInfo#usageAfterGc [1], 
although it depends on JVM platform.

2) About the default value. Normally, we have an expected max heap memory usage for a
Java Application. For example, In pulsar we enables UseG1GC by default, and with G1GC
we have InitiatingHeapOccupancyPercent = 45 by default. It will start a GC when heap 
usage is greater than 45%. If heap usage is over 45%, then it means it's under heavy load 
and we should do some load shedding. So We can set the loadBalancerMemoryResourceWeight
to 0.45 or leave some margin like 0.5.

[1] https://docs.oracle.com/en/java/javase/12/docs/api/jdk.management/com/sun/management/GcInfo.html
[2] https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc.html

Thanks,
Haiting Jiang


On 2021/12/24 01:57:37 jiangxinwei wrote:
> https://github.com/apache/pulsar/pull/13464
> 
>  
> 
> Pasted below for quoting convenience. -------
> 
>  
> 
> Motivation
> 
> memory.percentUsage() value is depends on the gc trigger moment, this value
> has a random affect on ThresholdShedder loadBalancer strategy. it is a
> meaningless dimension
> 
> the usage data from one broker
> 
>  
> <https://user-images.githubusercontent.com/5210343/147305803-45f42c19-ec9c-4
> 89d-9f98-b44205a43f8c.png> 
> 
> solution
> 
> set loadBalancerMemoryResourceWeight=0.0
> 
> 
> 
> -- 
> Avast 防毒软件已对此电子邮件执行病毒检查。
> https://www.avast.com/antivirus
>