You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/11/29 16:17:00 UTC
[jira] [Commented] (IMPALA-11753) CatalogD OOMkilled due to natively allocated memory
[ https://issues.apache.org/jira/browse/IMPALA-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640811#comment-17640811 ]
ASF subversion and git services commented on IMPALA-11753:
----------------------------------------------------------
Commit b816b6e3980d41d49644dc8909a9b4c845bed31a in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b816b6e39 ]
IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
CatalogD can be OOMKilled due to too much natively allocated memory.
The bug is due to a misuse of a Java compression API:
https://bugs.openjdk.org/browse/JDK-8257032
The problem is that we create our own Deflater object and pass it
to the constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on
the Deflater, and won't invoke its end() method:
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory.
This means that until the GC doesn't destroy the unreachable Deflater
objects they can consume quite much native memory. In some scenarios
it can even result in OOMKills by the kernel.
The fix is to override the DeflaterOutputStream's close() method so
it invokes end() on the Deflater object.
Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Reviewed-on: http://gerrit.cloudera.org:8080/19282
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> CatalogD OOMkilled due to natively allocated memory
> ---------------------------------------------------
>
> Key: IMPALA-11753
> URL: https://issues.apache.org/jira/browse/IMPALA-11753
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: OOM
>
> We can hit this bug in several installations: https://bugs.openjdk.org/browse/JDK-8257032
> (another nice description of the issue can be found here: https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)
> The problem is that we are creating our own Deflater object and pass it to the constructor of DeflaterOutputStream:
> https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
> This means that Java's DeflaterOutputStream won't assume ownership on the Deflater, and won't invoke its end() method:
> * https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
> * https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
> The Deflater's methods are implemented in C and allocate native memory. This means that until the GC doesn't destroy the unreachable Deflater objects they can consume quite much native memory. In some scenarios it can even result in OOMKills by the kernel.
> We should solve this issue by either
> * override close() of DeflaterOutputStream (like mentioned in the above blog post)
> * directly invoke end() on the Deflater object in a finally clause
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org