You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Alexander Pivovarov <ap...@gmail.com> on 2015/02/05 03:03:56 UTC
Hiveserver2 memory / thread leak v 0.13.1 (hdp-2.1.5)
I'm using hive-0.13.1 (hdp-2.1.5)
Hiveserver2 is running for 6 days
nobody is connected to the server now.
last connection was 30 min ago.
Used memory: 688MB (after GC)
Live threads: 297
perm: 99.6% used
Two days ago numbers were
Used memory: 259MB (after GC)
Live threads: 108
perm: 98% used
Looks like a memory/thread leak.
Today and 2 days ago VisualVM screenshots are attached
Re: Hiveserver2 memory / thread leak v 0.13.1 (hdp-2.1.5)
Posted by Alexander Pivovarov <ap...@gmail.com>.
Is HIVE-7353 relates to embedded MetaStore only?
We use MySQL for metadata.
HS2 OOM yesterday
Current HS2 instance uses just 77MB for now (39 live threads).
I found another issue fixes HS2 leaks
https://issues.apache.org/jira/browse/HIVE-9234
JVM Arguments:
-Xmx1024m
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9010
-Dhadoop.log.dir=/var/log/hadoop/svc-hive
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/lib/hadoop
-Dhadoop.id.str=svc-hive
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml
-Djava.net.preferIPv4Stack=true
-Xmx1024m
-Xmx4096m
-Dhadoop.security.logger=INFO,NullAppender
On Fri, Feb 6, 2015 at 1:48 AM, Vaibhav Gumashta <vg...@hortonworks.com>
wrote:
> Alexander,
>
> Can you share the jmap histo (or even better, a heapdump)? What are the
> top consumers? What are the heap+permgen sizes that the JVM is configured
> to use?
>
> FYI, we fixed this memory leak in Hive-14:
> https://issues.apache.org/jira/browse/HIVE-7353.
>
> Thanks,
> —Vaibhav
>
> From: Alexander Pivovarov <ap...@gmail.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Wednesday, February 4, 2015 at 6:03 PM
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Subject: Hiveserver2 memory / thread leak v 0.13.1 (hdp-2.1.5)
>
> I'm using hive-0.13.1 (hdp-2.1.5)
>
> Hiveserver2 is running for 6 days
>
> nobody is connected to the server now.
> last connection was 30 min ago.
>
> Used memory: 688MB (after GC)
> Live threads: 297
> perm: 99.6% used
>
> Two days ago numbers were
> Used memory: 259MB (after GC)
> Live threads: 108
> perm: 98% used
>
> Looks like a memory/thread leak.
>
> Today and 2 days ago VisualVM screenshots are attached
>
>
Re: Hiveserver2 memory / thread leak v 0.13.1 (hdp-2.1.5)
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
Alexander,
Can you share the jmap histo (or even better, a heapdump)? What are the top consumers? What are the heap+permgen sizes that the JVM is configured to use?
FYI, we fixed this memory leak in Hive-14: https://issues.apache.org/jira/browse/HIVE-7353.
Thanks,
-Vaibhav
From: Alexander Pivovarov <ap...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Wednesday, February 4, 2015 at 6:03 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Hiveserver2 memory / thread leak v 0.13.1 (hdp-2.1.5)
I'm using hive-0.13.1 (hdp-2.1.5)
Hiveserver2 is running for 6 days
nobody is connected to the server now.
last connection was 30 min ago.
Used memory: 688MB (after GC)
Live threads: 297
perm: 99.6% used
Two days ago numbers were
Used memory: 259MB (after GC)
Live threads: 108
perm: 98% used
Looks like a memory/thread leak.
Today and 2 days ago VisualVM screenshots are attached