You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2025/02/25 01:45:23 UTC

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22538


Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................

IMPALA-13783: Fix huge temp list created in catalog-update thread

In catalogd, the catalog-update thread collects new/deleted catalog
objects since the last update. When collecting new partitions of a
HdfsTable, a list of TCatalogObject is created. Each TCatalogObject
has a THdfsPartition corresponding to a new partition. See
HdfsTable#getNewPartitionsSinceLastUpdate().

This temp list could be huge and occupying a comparable memory footprint
to the table metadata. E.g. for a HdfsTable that has 6M partitions and
6M files (one file per partition), the HdfsTable object itself takes
7.3GB in memory. This temp list takes 8.9GB. See more details in the
JIRA description.

This patch fixes the issue by iterating the partition map directly to
avoid creating this temp list. For each partition, we collect its thrift
representation and add it into the topic update directly.

In local-catalog mode, the thrift representation of the partition can be
simplified to just contain the names of db, table, partition and the
partition ids. This is also optimized to reduce the topic update size in
local-catalog mode.

Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
---
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
4 files changed, 28 insertions(+), 28 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/22538/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 1: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Feb 2025 21:30:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11507/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Feb 2025 22:33:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 1: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Feb 2025 22:32:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Feb 2025 22:33:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................

IMPALA-13783: Fix huge temp list created in catalog-update thread

In catalogd, the catalog-update thread collects new/deleted catalog
objects since the last update. When collecting new partitions of a
HdfsTable, a list of TCatalogObject is created. Each TCatalogObject
has a THdfsPartition corresponding to a new partition. See
HdfsTable#getNewPartitionsSinceLastUpdate().

This temp list could be huge and occupying a comparable memory footprint
to the table metadata. E.g. for a HdfsTable that has 6M partitions and
6M files (one file per partition), the HdfsTable object itself takes
7.3GB in memory. This temp list takes 8.9GB. See more details in the
JIRA description.

This patch fixes the issue by iterating the partition map directly to
avoid creating this temp list. For each partition, we collect its thrift
representation and add it into the topic update directly.

In local-catalog mode, the thrift representation of the partition can be
simplified to just contain the names of db, table, partition and the
partition ids. This is also optimized to reduce the topic update size in
local-catalog mode.

Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Reviewed-on: http://gerrit.cloudera.org:8080/22538
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
4 files changed, 28 insertions(+), 28 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 04:00:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13783: Fix huge temp list created in catalog-update thread

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22538 )

Change subject: IMPALA-13783: Fix huge temp list created in catalog-update thread
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/18356/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41dfd48278cb87bdc6f251a13919466d15f8d52d
Gerrit-Change-Number: 22538
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Feb 2025 02:12:41 +0000
Gerrit-HasComments: No