You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/11/28 17:23:19 UTC
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19282
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
CatalogD can be OOMKilled due to too mucn natively allocated memory.
The bug is due to a misuse of a Java compression API:
https://bugs.openjdk.org/browse/JDK-8257032
The problem is that we creating our own Deflater object and pass it
to the constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on
the Deflater, and won't invoke its end() method:
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory.
This means that until the GC doesn't destroy the unreachable Deflater
objects they can consume quite much native memory. In some scenarios
it can even result in OOMKills by the kernel.
The fix is to override the DeflaterOutputStream's close() method so
it invokes end() on the Deflater object.
Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
---
M fe/src/main/java/org/apache/impala/util/CompressionUtil.java
1 file changed, 12 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19282/1
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Laszlo Gaal, Gergely Fürnstáhl, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19282
to look at the new patch set (#2).
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
CatalogD can be OOMKilled due to too much natively allocated memory.
The bug is due to a misuse of a Java compression API:
https://bugs.openjdk.org/browse/JDK-8257032
The problem is that we create our own Deflater object and pass it
to the constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on
the Deflater, and won't invoke its end() method:
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory.
This means that until the GC doesn't destroy the unreachable Deflater
objects they can consume quite much native memory. In some scenarios
it can even result in OOMKills by the kernel.
The fix is to override the DeflaterOutputStream's close() method so
it invokes end() on the Deflater object.
Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
---
M fe/src/main/java/org/apache/impala/util/CompressionUtil.java
1 file changed, 12 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19282/2
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
CatalogD can be OOMKilled due to too much natively allocated memory.
The bug is due to a misuse of a Java compression API:
https://bugs.openjdk.org/browse/JDK-8257032
The problem is that we create our own Deflater object and pass it
to the constructor of DeflaterOutputStream:
https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
This means that Java's DeflaterOutputStream won't assume ownership on
the Deflater, and won't invoke its end() method:
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
* https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
The Deflater's methods are implemented in C and allocate native memory.
This means that until the GC doesn't destroy the unreachable Deflater
objects they can consume quite much native memory. In some scenarios
it can even result in OOMKills by the kernel.
The fix is to override the DeflaterOutputStream's close() method so
it invokes end() on the Deflater object.
Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Reviewed-on: http://gerrit.cloudera.org:8080/19282
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/util/CompressionUtil.java
1 file changed, 12 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 1: Code-Review+2
LGTM. Feel free to carry on the +2 after modifying the commit message.
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 10:44:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG@13
PS1, Line 13: we creating
nit: grammar; I'd suggest "we create".
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 10:13:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8840/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 11:06:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11912/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 17:42:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 16:13:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
Nice catch, thanks for the fix!
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG@9
PS1, Line 9: mucn
typo: much
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 09:18:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 2: Code-Review+2
(2 comments)
Thanks Everyone for the comments!
Catching this bug was actually a team work, so I don't take all the credit.
Carry +2
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG@9
PS1, Line 9: much
> typo: much
Done
http://gerrit.cloudera.org:8080/#/c/19282/1//COMMIT_MSG@13
PS1, Line 13: we create o
> nit: grammar; I'd suggest "we create".
Done
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 11:05:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19282 )
Change subject: IMPALA-11753: CatalogD OOMkilled due to natively allocated memory
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I663a21f60871e32d2d0100ea03d92fd8ab460691
Gerrit-Change-Number: 19282
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Nov 2022 11:05:59 +0000
Gerrit-HasComments: No