You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexis Sarda-Espinosa <sa...@gmail.com> on 2023/02/20 16:14:30 UTC
Calculation of UI's maximum non-heap memory
Hello,
I have configured a job manager with the following settings (Flink 1.16.1):
jobmanager.memory.process.size: 1024m
jobmanager.memory.jvm-metaspace.size: 150m
jobmanager.memory.off-heap.size: 64m
jobmanager.memory.jvm-overhead.min: 168m
jobmanager.memory.jvm-overhead.max: 168m
jobmanager.memory.enable-jvm-direct-memory-limit: "true"
However, when I look at the job manager dashboard in the UI, I see that the
value of Non-Heap Maximum is reported as 532 MB. Could someone clarify how
this value is calculated?
In case it's relevant, the effective configuration for JVM Heap is reported
as 642 MB, with the reported maximum being 621 MB.
Regards,
Alexis.
Re: Calculation of UI's maximum non-heap memory
Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
Very useful, thanks a lot.
Regards,
Alexis.
Am Di., 21. Feb. 2023 um 12:04 Uhr schrieb Weihua Hu <huweihua.ckl@gmail.com
>:
> Hi Alexis,
>
> The maximum Non-Heap is the sum of the memory pool (which is non-hep) max
> size. There are 3 memory pools(based on jdk11):
> 1. Metaspace, we can control the size with JVM parameter -XX:MaxMetaspaceSize
> or Flink configuration: jobmanager.memory.jvm-metaspace.size. For your
> job, this pool size is 150m
> 2. Compressed Class Space, this is controlled by JVM parameter
> -XX:CompressedClassSpaceSize. The default value is 1G or MetaspaceSize - 2
> * InitialBootClassLoaderMetaspaceSize(default is 4194304). For your job,
> this pool size is 142m
> 3. CodeCache(codeHeap profiled nmethods/non-profiled
> nmethods/non-nmethods), this is controlled by JVM parameter
> -XX:ReservedCodeCacheSize. The default value is 240m
>
> So, the maximum non-heap is 150+142+240 = 532m.
>
>
> Best,
> Weihua
>
>
> On Tue, Feb 21, 2023 at 2:33 PM Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> wrote:
>
>> Hi Weihua,
>>
>> Thanks for your response, I am familiar with those calculations, the one
>> I don't understand is the Maximum Non-Heap value.
>>
>> Regards,
>> Alexis.
>>
>> On Tue, 21 Feb 2023, 04:45 Weihua Hu, <hu...@gmail.com> wrote:
>>
>>> Hi, Alexis
>>>
>>> 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx
>>> to 673185792(642m),-XX:MaxDirectMemorySize to 67108864(64m),-XX:MaxMetaspaceSize
>>> to 157286400(150m), you can find more information from [1]
>>> 2. As the hint in Flink UI: "The maximum heap displayed might differ
>>> from the configured values depending on the used GC algorithm for this
>>> process.", This[2] shows how JVM calculate the max heap memory from
>>> configured -Xms/-Xmx
>>>
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/
>>> [2]
>>> https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa <
>>> sarda.espinosa@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have configured a job manager with the following settings (Flink
>>>> 1.16.1):
>>>>
>>>> jobmanager.memory.process.size: 1024m
>>>> jobmanager.memory.jvm-metaspace.size: 150m
>>>> jobmanager.memory.off-heap.size: 64m
>>>> jobmanager.memory.jvm-overhead.min: 168m
>>>> jobmanager.memory.jvm-overhead.max: 168m
>>>> jobmanager.memory.enable-jvm-direct-memory-limit: "true"
>>>>
>>>> However, when I look at the job manager dashboard in the UI, I see that
>>>> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify
>>>> how this value is calculated?
>>>>
>>>> In case it's relevant, the effective configuration for JVM Heap is
>>>> reported as 642 MB, with the reported maximum being 621 MB.
>>>>
>>>> Regards,
>>>> Alexis.
>>>>
>>>>
Re: Calculation of UI's maximum non-heap memory
Posted by Weihua Hu <hu...@gmail.com>.
Hi Alexis,
The maximum Non-Heap is the sum of the memory pool (which is non-hep) max
size. There are 3 memory pools(based on jdk11):
1. Metaspace, we can control the size with JVM parameter -XX:MaxMetaspaceSize
or Flink configuration: jobmanager.memory.jvm-metaspace.size. For your job,
this pool size is 150m
2. Compressed Class Space, this is controlled by JVM parameter
-XX:CompressedClassSpaceSize. The default value is 1G or MetaspaceSize - 2
* InitialBootClassLoaderMetaspaceSize(default is 4194304). For your job,
this pool size is 142m
3. CodeCache(codeHeap profiled nmethods/non-profiled
nmethods/non-nmethods), this is controlled by JVM parameter
-XX:ReservedCodeCacheSize. The default value is 240m
So, the maximum non-heap is 150+142+240 = 532m.
Best,
Weihua
On Tue, Feb 21, 2023 at 2:33 PM Alexis Sarda-Espinosa <
sarda.espinosa@gmail.com> wrote:
> Hi Weihua,
>
> Thanks for your response, I am familiar with those calculations, the one I
> don't understand is the Maximum Non-Heap value.
>
> Regards,
> Alexis.
>
> On Tue, 21 Feb 2023, 04:45 Weihua Hu, <hu...@gmail.com> wrote:
>
>> Hi, Alexis
>>
>> 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx
>> to 673185792(642m),-XX:MaxDirectMemorySize to 67108864(64m),-XX:MaxMetaspaceSize
>> to 157286400(150m), you can find more information from [1]
>> 2. As the hint in Flink UI: "The maximum heap displayed might differ
>> from the configured values depending on the used GC algorithm for this
>> process.", This[2] shows how JVM calculate the max heap memory from
>> configured -Xms/-Xmx
>>
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/
>> [2]
>> https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method
>>
>> Best,
>> Weihua
>>
>>
>> On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have configured a job manager with the following settings (Flink
>>> 1.16.1):
>>>
>>> jobmanager.memory.process.size: 1024m
>>> jobmanager.memory.jvm-metaspace.size: 150m
>>> jobmanager.memory.off-heap.size: 64m
>>> jobmanager.memory.jvm-overhead.min: 168m
>>> jobmanager.memory.jvm-overhead.max: 168m
>>> jobmanager.memory.enable-jvm-direct-memory-limit: "true"
>>>
>>> However, when I look at the job manager dashboard in the UI, I see that
>>> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify
>>> how this value is calculated?
>>>
>>> In case it's relevant, the effective configuration for JVM Heap is
>>> reported as 642 MB, with the reported maximum being 621 MB.
>>>
>>> Regards,
>>> Alexis.
>>>
>>>
Re: Calculation of UI's maximum non-heap memory
Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
Hi Weihua,
Thanks for your response, I am familiar with those calculations, the one I
don't understand is the Maximum Non-Heap value.
Regards,
Alexis.
On Tue, 21 Feb 2023, 04:45 Weihua Hu, <hu...@gmail.com> wrote:
> Hi, Alexis
>
> 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx
> to 673185792(642m),-XX:MaxDirectMemorySize to 67108864(64m),-XX:MaxMetaspaceSize
> to 157286400(150m), you can find more information from [1]
> 2. As the hint in Flink UI: "The maximum heap displayed might differ from
> the configured values depending on the used GC algorithm for this
> process.", This[2] shows how JVM calculate the max heap memory from
> configured -Xms/-Xmx
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/
> [2]
> https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method
>
> Best,
> Weihua
>
>
> On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> wrote:
>
>> Hello,
>>
>> I have configured a job manager with the following settings (Flink
>> 1.16.1):
>>
>> jobmanager.memory.process.size: 1024m
>> jobmanager.memory.jvm-metaspace.size: 150m
>> jobmanager.memory.off-heap.size: 64m
>> jobmanager.memory.jvm-overhead.min: 168m
>> jobmanager.memory.jvm-overhead.max: 168m
>> jobmanager.memory.enable-jvm-direct-memory-limit: "true"
>>
>> However, when I look at the job manager dashboard in the UI, I see that
>> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify
>> how this value is calculated?
>>
>> In case it's relevant, the effective configuration for JVM Heap is
>> reported as 642 MB, with the reported maximum being 621 MB.
>>
>> Regards,
>> Alexis.
>>
>>
Re: Calculation of UI's maximum non-heap memory
Posted by Weihua Hu <hu...@gmail.com>.
Hi, Alexis
1. With those configuration, Flink will set JVM parameters -Xms and -Xmx to
673185792(642m),-XX:MaxDirectMemorySize to 67108864(64m),-XX:MaxMetaspaceSize
to 157286400(150m), you can find more information from [1]
2. As the hint in Flink UI: "The maximum heap displayed might differ from
the configured values depending on the used GC algorithm for this
process.", This[2] shows how JVM calculate the max heap memory from
configured -Xms/-Xmx
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/
[2]
https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method
Best,
Weihua
On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa <
sarda.espinosa@gmail.com> wrote:
> Hello,
>
> I have configured a job manager with the following settings (Flink 1.16.1):
>
> jobmanager.memory.process.size: 1024m
> jobmanager.memory.jvm-metaspace.size: 150m
> jobmanager.memory.off-heap.size: 64m
> jobmanager.memory.jvm-overhead.min: 168m
> jobmanager.memory.jvm-overhead.max: 168m
> jobmanager.memory.enable-jvm-direct-memory-limit: "true"
>
> However, when I look at the job manager dashboard in the UI, I see that
> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify
> how this value is calculated?
>
> In case it's relevant, the effective configuration for JVM Heap is
> reported as 642 MB, with the reported maximum being 621 MB.
>
> Regards,
> Alexis.
>
>