[jira] [Created] (HIVE-24179) Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement

["Stamatis Zampetakis (Jira)" <ji...@apache.org>]

Stamatis Zampetakis created HIVE-24179:
------------------------------------------

             Summary: Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement
                 Key: HIVE-24179
                 URL: https://issues.apache.org/jira/browse/HIVE-24179
             Project: Hive
          Issue Type: Bug
          Components: HiveServer2
            Reporter: Stamatis Zampetakis
            Assignee: Stamatis Zampetakis
             Fix For: 4.0.0
         Attachments: summary.png

The problem can be reproduced by executing repeatedly a SHOW LOCK statement and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a few minutes before the server crashes with OutOfMemory error such as the one shown below.

{noformat}
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
        at org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
        at org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
        at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
        at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
        at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
        at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
        at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
        at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
        at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
        at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
        at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
        at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
        at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
        at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
        at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
        at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
{noformat}

The heap dump shows (summary.png) that most of the memory is consumed by {{Hashtable$Entry}} and {{ConcurrentHashMap$Node}} objects coming from Hive configurations referenced by {{DbTxnManager}}. 

The latter are not eligible for garbage collection since at [construction|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L212] time they are passed implicitly in a callback  stored inside ShutdownHookManager.  

When the {{DbTxnManager}} is closed properly the leak is not present since the callback is [removed|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java#L882] from ShutdownHookManager. 

{{SHOW LOCKS}} statements create ([ShowDbLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java#L52], [ShowLocksAnalyzer|https://github.com/apache/hive/blob/975c832b6d069559c5b406a4aa8def3180fe4e75/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java#L72]) a new {{TxnManager}} and they never close it leading to the memory leak.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)