You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/09/15 07:23:08 UTC
[impala] branch master updated (f8eac781c -> cfd79b40b)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from f8eac781c IMPALA-11557: Fix memory leak in BlockingRowBatchQueue
new c5b490fa4 IMPALA-11422: Bump Apache Hive to 3.1.3
new cfd79b40b IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/impala-config.sh | 2 +-
fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java | 5 +++++
fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java | 8 ++++++--
3 files changed, 12 insertions(+), 3 deletions(-)
[impala] 02/02: IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit cfd79b40beab86f08ad72e0bea41eabf736d0a99
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Tue Sep 13 16:23:40 2022 +0800
IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
In the legacy catalog mode, catalogd propagates incremental metadata
updates at the partition level. While applying the updates, impalad
reuses the existing partition objects and moves them to a new HdfsTable
object. However, the partition objects are immutable, which means their
reference to the old table object remains unchanged. JVM GC cannot
collect the stale table objects since they still have active reference
from the partitions, which results in memory leak.
This patch fixes the issue by recreating a new partition object based on
the existing partition object with the new table field.
Tests:
- Verified locally that after applying the patch, I don’t see the
number of live HdfsTable objects keeps bumping.
Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a
Reviewed-on: http://gerrit.cloudera.org:8080/18978
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
---
fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java | 5 +++++
fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java | 8 ++++++--
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java b/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
index 6acd9b9d8..88b7eec26 100644
--- a/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
+++ b/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
@@ -1278,6 +1278,10 @@ public class HdfsPartition extends CatalogObjectImpl
this(partition.table_);
oldInstance_ = partition;
prevId_ = oldInstance_.id_;
+ copyFromPartition(partition);
+ }
+
+ public Builder copyFromPartition(HdfsPartition partition) {
partitionKeyValues_ = partition.partitionKeyValues_;
fileFormatDescriptor_ = partition.fileFormatDescriptor_;
setFileDescriptors(partition);
@@ -1296,6 +1300,7 @@ public class HdfsPartition extends CatalogObjectImpl
// Take over the in-flight events
inFlightEvents_ = partition.inFlightEvents_;
lastCompactionId_ = partition.lastCompactionId_;
+ return this;
}
public HdfsPartition build() {
diff --git a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
index a03393475..bed23effa 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
@@ -494,8 +494,12 @@ public class ImpaladCatalog extends Catalog implements FeCatalog {
for (PrunablePartition part : ((HdfsTable) existingTable).getPartitions()) {
numExistingParts++;
if (tHdfsTable.partitions.containsKey(part.getId())) {
- Preconditions.checkState(
- newHdfsTable.addPartitionNoThrow((HdfsPartition) part));
+ // Create a new partition instance under the new table object and copy all
+ // the fields of the existing partition.
+ HdfsPartition newPart = new HdfsPartition.Builder(newHdfsTable, part.getId())
+ .copyFromPartition((HdfsPartition) part)
+ .build();
+ Preconditions.checkState(newHdfsTable.addPartitionNoThrow(newPart));
} else {
numDeletedParts++;
}
[impala] 01/02: IMPALA-11422: Bump Apache Hive to 3.1.3
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit c5b490fa40ba6e1fd89416b697ef96b77adec13a
Author: Fucun Chu <ch...@hotmail.com>
AuthorDate: Tue Jul 12 00:33:36 2022 +0800
IMPALA-11422: Bump Apache Hive to 3.1.3
This patch bump up version of Apache Hive to 3.1.3.
Change-Id: I395a99cccf0a8902b3fd47235c32b69c5494f291
Reviewed-on: http://gerrit.cloudera.org:8080/18716
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/impala-config.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 4aec2d408..1cbb46c9f 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -215,7 +215,7 @@ export CDP_TEZ_VERSION=0.9.1.7.2.16.0-164
# Ref: https://infra.apache.org/release-download-pages.html#closer
: ${APACHE_MIRROR:="https://www.apache.org/dyn/closer.cgi"}
export APACHE_MIRROR
-export APACHE_HIVE_VERSION=3.1.2
+export APACHE_HIVE_VERSION=3.1.3
export APACHE_HIVE_STORAGE_API_VERSION=2.7.0
export APACHE_OZONE_VERSION=1.2.1