Best way/tool to debug memory leaks in HiveServer2

[rajan sthapit <ra...@gmail.com>]

Hi,

I am currently trying to debug a memory leak in a HiveServer2 java process.
The heap size for the jvm is set to 16GB. When I look at the heap usage it
is well under 16 GB, However, if I look at the total memory consumed by the
process(top command)  it's more than 30 GB.

Currently, I am trying to use pmap to see the memory consumption by the
process. Here is the result of the pmap

https://gist.github.com/anonymous/977b2ba8e7484d02dcec00a10439e516


It also shows that the process is consuming more than 30GB. However, it is
not clear what is causing the process to consume more than 30GB.


Any suggestion, on how I can debug this issue. The JVM heap memory usage
seems to be fine. It is well under 16GB. However, the total process memory
consumed by the java process is more than 30GB.


Could you please recommend any tools/methods that I could use to debug this
issue?


Thanks

Re: Best way/tool to debug memory leaks in HiveServer2

[rajan sthapit <ra...@gmail.com>]

I tried getting the histo as you suggested


jmap -histo:live ${PID}



However, it only shows the histo of the live objects
https://gist.github.com/anonymous/669eae8128c9a11f72fcdc1f82270460. Also
the total memory from the histogram shows to be around 5GB. However, if I
look at the total process memory for the java process using the top command
it is more than 30 GB.


On Tue, Mar 13, 2018 at 6:33 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:

> > It also shows that the process is consuming more than 30GB. However, it
> is not clear what is causing the process to consume more than 30GB.
>
> The Xmx only applies to the heap size, there's another factor that is
> usually ignored which are the network buffers and compression buffers used
> by Java.
>
> Most of these are only collected on a full GC pause (or OS memory
> pressure, I think).
>
> Try running a jmap -histo:live and see what happens to the extra buffers
> on that list.
>
> Cheers,
> Gopal
>
>
>

Re: Best way/tool to debug memory leaks in HiveServer2

[Gopal Vijayaraghavan <go...@apache.org>]

> It also shows that the process is consuming more than 30GB. However, it is not clear what is causing the process to consume more than 30GB.

The Xmx only applies to the heap size, there's another factor that is usually ignored which are the network buffers and compression buffers used by Java.

Most of these are only collected on a full GC pause (or OS memory pressure, I think).

Try running a jmap -histo:live and see what happens to the extra buffers on that list.

Cheers,
Gopal

Re: Best way/tool to debug memory leaks in HiveServer2

[Prasanth Jayachandran <pj...@hortonworks.com>]

One option to check for is if THP is enabled on the node running HS2. If enable try disabling THP and see if it helps

# Disable THP echo never | sudo tee -a /sys/kernel/mm/transparent_hugepage/enabled echo never | sudo tee -a /sys/kernel/mm/transparent_hugepage/defrag Thanks Prasanth

On Mar 13, 2018, at 5:32 PM, rajan sthapit <ra...@gmail.com>> wrote:

Hi,

I am currently trying to debug a memory leak in a HiveServer2 java process. The heap size for the jvm is set to 16GB. When I look at the heap usage it is well under 16 GB, However, if I look at the total memory consumed by the process(top command)  it's more than 30 GB.

Currently, I am trying to use pmap to see the memory consumption by the process. Here is the result of the pmap

https://gist.github.com/anonymous/977b2ba8e7484d02dcec00a10439e516

It also shows that the process is consuming more than 30GB. However, it is not clear what is causing the process to consume more than 30GB.

Any suggestion, on how I can debug this issue. The JVM heap memory usage seems to be fine. It is well under 16GB. However, the total process memory consumed by the java process is more than 30GB.

Could you please recommend any tools/methods that I could use to debug this issue?

Thanks