You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jesus Camacho Rodriguez (Jira)" <ji...@apache.org> on 2020/11/13 17:26:00 UTC
[jira] [Resolved] (HIVE-24179) Memory leak in HS2 DbTxnManager when
compiling SHOW LOCKS statement
[ https://issues.apache.org/jira/browse/HIVE-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jesus Camacho Rodriguez resolved HIVE-24179.
--------------------------------------------
Resolution: Fixed
Pushed to master, thanks [~zabetak]!
> Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
> -------------------------------------------------------------------
>
> Key: HIVE-24179
> URL: https://issues.apache.org/jira/browse/HIVE-24179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: summary.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The problem can be reproduced by executing repeatedly a SHOW LOCK statement and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a few minutes before the server crashes with OutOfMemory error such as the one shown below.
> {noformat}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
> at org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
> at org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
> at org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
> at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
> at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
> at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
> at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
> at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
> at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
> at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
> at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
> at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
> at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
> at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
> at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
> at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
> at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
> at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
> at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
> at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
> {noformat}
> The heap dump shows (summary.png) that most of the memory is consumed by {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive configurations referenced by {{DbTxnManager}}.
> The latter are not eligible for garbage collection since at [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212] time they are passed implicitly in a callback stored inside ShutdownHookManager.
> When the {{DbTxnManager}} is closed properly the leak is not present since the callback is [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882] from ShutdownHookManager.
> {{SHOW LOCKS}} statements create ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52], [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72]) a new {{TxnManager}} and they never close it leading to the memory leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)