[impala] 03/17: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch branch-4.1.2
in repository https://gitbox.apache.org/repos/asf/impala.git

commit d9cbfa8733f25af0f6b41f650eac8d7feb2a52e1
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Mon Nov 28 18:19:10 2022 +0100

    IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
    
    CatalogD can be OOMKilled due to too much natively allocated memory.
    The bug is due to a misuse of a Java compression API:
    https://bugs.openjdk.org/browse/JDK-8257032
    
    The problem is that we create our own Deflater object and pass it
    to the constructor of DeflaterOutputStream:
    https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
    
    This means that Java's DeflaterOutputStream won't assume ownership on
    the Deflater, and won't invoke its end() method:
    
    * https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
    * https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
    
    The Deflater's methods are implemented in C and allocate native memory.
    This means that until the GC doesn't destroy the unreachable Deflater
    objects they can consume quite much native memory. In some scenarios
    it can even result in OOMKills by the kernel.
    
    The fix is to override the DeflaterOutputStream's close() method so
    it invokes end() on the Deflater object.
    
    Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
    Reviewed-on: http://gerrit.cloudera.org:8080/19282
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../main/java/org/apache/impala/util/CompressionUtil.java   | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
index f6e11a110..cfa39c914 100644
--- a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
@@ -44,7 +44,18 @@ public class CompressionUtil {
     // Deflater with 'BEST_SPEED' level provided reasonable compression ratios at much
     // faster speeds compared to other modes like BEST_COMPRESSION/DEFAULT_COMPRESSION.
     DeflaterOutputStream stream =
-        new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED));
+        new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED)) {
+          // IMPALA-11753: to avoid CatalogD OOM we invoke def.end() which frees
+          // the natively allocated memory by the Deflater. See details in Jira.
+          @Override
+          public void close() throws IOException {
+            try {
+              super.close();
+            } finally {
+              def.end();
+            }
+          }
+        };
     try {
       stream.write(input);
       stream.close();