You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/11/13 17:25:00 UTC
[jira] [Work logged] (HIVE-24179) Memory leak in HS2 DbTxnManager
when compiling SHOW LOCKS statement
[ https://issues.apache.org/jira/browse/HIVE-24179?focusedWorklogId=511446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-511446 ]
ASF GitHub Bot logged work on HIVE-24179:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Nov/20 17:24
Start Date: 13/Nov/20 17:24
Worklog Time Spent: 10m
Work Description: jcamachor merged pull request #1509:
URL: https://github.com/apache/hive/pull/1509
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 511446)
Time Spent: 2h 10m (was: 2h)
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a few minutes before the server crashes with OutOfMemory error such as the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive configurations referenced by {{DbTxnManager}}.
> The latter are not eligible for garbage collection since at [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212] time they are passed implicitly in a callback stored inside ShutdownHookManager.
> When the {{DbTxnManager}} is closed properly the leak is not present since the callback is [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882] from ShutdownHookManager.
> {{SHOW LOCKS}} statements create ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52], [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72]) a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)