[jira] [Created] (IMPALA-11753) CatalogD OOMkilled due to natively allocated memory

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-11753:
------------------------------------------

             Summary: CatalogD OOMkilled due to natively allocated memory
                 Key: IMPALA-11753
                 URL: https://issues.apache.org/jira/browse/IMPALA-11753
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Zoltán Borók-Nagy


We can hit this bug in several installations: https://bugs.openjdk.org/browse/JDK-8257032
(another nice description of the issue can be found here: https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)

The problem is that we creating our own Deflater object and pass it to the constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47

This means that Java's DeflaterOutputStream won't assume ownership on the Deflater, and won't invoke its end() method:
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247

The Deflater's methods are implemented in C and allocate native memory. This means that until the GC doesn't destroy the unreachable Deflater objects they can consume quite much native memory. In some scenarios it can even result in OOMKills by the kernel.

We should solve this issue by either
* override close() of DeflaterOutputStream (like mentioned in the above blog post)
* directly invoke end() on the Deflater object in a finally clause



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org