You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "zhangqianqiong (Code Review)" <ge...@cloudera.org> on 2024/10/11 10:07:59 UTC
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
zhangqianqiong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21922
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
After applying the merge request 'IMPALA-10502: Handle CREATE/DROP events correctly',
the `alterTableRecoverPartitions` method changed from batching the `add_partitions` calls
to invoking `addHmsPartitions` all at once. However, for tables with a huge number of
partitions, this can result in the creation of a huge temporary object, List<Partitions>,
leading to OutOfMemory.
In my test environment, where the catalogd JVM Xmx was set to 2GB, running the end-to-end
test custom_cluster/test_wide_table_operations.py on a table with 2000 columns and 50,000
partitions during the recover partitions operation caused catalogd to run into a 'Java heap
space' OutOfMemoryError.
An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object
contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions),
which overwhelmed memory resources.
To resolve this issue, we propose batching the addHmsPartitions calls, ensuring that
temporary objects are released after each batch operation. This solution was tested and
verified to resolve the OutOfMemoryError, ensuring system stability when handling a
large number of partitions.
Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
1 file changed, 16 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/21922/1
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 1
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "zhangqianqiong (Code Review)" <ge...@cloudera.org>.
zhangqianqiong has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG@20
PS2, Line 20: An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object
: contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions),
: which overwhelmed memory resources.
> This seems to be resolved by IMPALA-11812. Do you have that patch in your d
I have reproduced the issue on the latest master branch. Although IMPALA-11812 provides a partial fix, it doesn't fully resolve the problem. The root cause lies in the alterTableRecoverPartitions method, specifically in the line List<Partition> addedPartitions = addHmsPartitions(msClient, tbl, hmsPartitions, partitionToEventId, true, catalogTimeline);. The size of the returned result object is too large, leading to OutOfMemory.
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 2
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Mon, 14 Oct 2024 02:55:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 1:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/21922/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/21922/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6329
PS1, Line 6329: for (List<Partition> hmsSublist : Lists.partition(hmsPartitions, MAX_PARTITION_ADDS_PER_BATCH)) {
line too long (103 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 1
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Oct 2024 10:09:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG@20
PS2, Line 20: An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object
: contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions),
: which overwhelmed memory resources.
> I have reproduced the issue on the latest master branch. Although IMPALA-11
I see. Checked the codes, when event-processor is disabled, the Partition objects come from MetaStoreUtil.addPartitions():
https://github.com/apache/impala/blob/2535e79491078a0353dbeed1a094e91366906149/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5319-L5321
In that method, the FieldSchema list of partitions will be replaced with the reference of the table's. So we shouldn't see lots of FieldSchema objects.
https://github.com/apache/impala/blob/2535e79491078a0353dbeed1a094e91366906149/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java#L248
However, when event-processor is enabled, the Partition objects are extracted from HMS events:
https://github.com/apache/impala/blob/2535e79491078a0353dbeed1a094e91366906149/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5343
The above partition list is not used as the result. I think we should modify getPartitionsFromEvent() to use the table level list of FieldSchema:
https://github.com/apache/impala/blob/2535e79491078a0353dbeed1a094e91366906149/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L2479
http://gerrit.cloudera.org:8080/#/c/21922/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/21922/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2483
PS2, Line 2483: partitionToEventId.put(part, eventId);
As commented above, I think we should trim the Partitions here using MetaStoreUtil.replaceSchemaFromTable() to get rid of lots of FieldSchema objects (they will be GCed).
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 2
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Mon, 14 Oct 2024 06:57:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21922/2//COMMIT_MSG@20
PS2, Line 20: An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object
: contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions),
: which overwhelmed memory resources.
This seems to be resolved by IMPALA-11812. Do you have that patch in your downstream branch?
https://gerrit.cloudera.org/c/19391/
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 2
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Sat, 12 Oct 2024 21:49:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17155/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 2
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Fri, 11 Oct 2024 10:41:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17154/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 1
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Fri, 11 Oct 2024 10:32:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "zhangqianqiong (Code Review)" <ge...@cloudera.org>.
zhangqianqiong has posted comments on this change. ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21922/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/21922/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6329
PS1, Line 6329: for (List<Partition> hmsSublist : Lists.partition(hmsPartitions, MAX_PARTITION_ADDS_PER_BATCH)) {
> line too long (103 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 1
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>
Gerrit-Comment-Date: Fri, 11 Oct 2024 10:16:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
Posted by "zhangqianqiong (Code Review)" <ge...@cloudera.org>.
zhangqianqiong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21922 )
Change subject: IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
......................................................................
IMPALA-13438 Batch the `addHmsPartitions` operations in `alterTableRecoverPartitions`.
After applying the merge request 'IMPALA-10502: Handle CREATE/DROP events correctly',
the `alterTableRecoverPartitions` method changed from batching the `add_partitions` calls
to invoking `addHmsPartitions` all at once. However, for tables with a huge number of
partitions, this can result in the creation of a huge temporary object, List<Partitions>,
leading to OutOfMemory.
In my test environment, where the catalogd JVM Xmx was set to 2GB, running the end-to-end
test custom_cluster/test_wide_table_operations.py on a table with 2000 columns and 50,000
partitions during the recover partitions operation caused catalogd to run into a 'Java
heap space' OutOfMemoryError.
An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object
contained a massive number of FieldSchema objects (2000 columns * 50,000 partitions),
which overwhelmed memory resources.
To resolve this issue, we propose batching the addHmsPartitions calls, ensuring that
temporary objects are released after each batch operation. This solution was tested and
verified to resolve the OutOfMemoryError, ensuring system stability when handling a
large number of partitions.
Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
1 file changed, 17 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/21922/2
--
To view, visit http://gerrit.cloudera.org:8080/21922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I13aaad8a915f75fbe808bf96b1cf891312b1a592
Gerrit-Change-Number: 21922
Gerrit-PatchSet: 2
Gerrit-Owner: zhangqianqiong <qq...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: zhangqianqiong <qq...@gmail.com>