[jira] [Created] (IMPALA-13438) In alterTableRecoverPartitions, we should batch the addHmsPartitions operations.

["zhangqianqiong (Jira)" <ji...@apache.org>]

zhangqianqiong created IMPALA-13438:
---------------------------------------

             Summary: In alterTableRecoverPartitions, we should batch the addHmsPartitions operations.
                 Key: IMPALA-13438
                 URL: https://issues.apache.org/jira/browse/IMPALA-13438
             Project: IMPALA
          Issue Type: Improvement
          Components: Catalog
    Affects Versions: Impala 4.4.1, Impala 4.4.0, Impala 4.3.0, Impala 4.1.2, Impala 4.1.1, Impala 4.2.0, Impala 4.1.0
            Reporter: zhangqianqiong
            Assignee: zhangqianqiong


After applying the merge request 'IMPALA-10502: Handle CREATE/DROP events correctly', the {{alterTableRecoverPartitions}} method changed from batching the {{add_partitions}} calls to invoking {{addHmsPartitions}} all at once. However, for tables with a huge number of partitions, this can result in the creation of a huge temporary object, {{{}List<Partitions>{}}}, leading to OutOfMemory.

In my test environment, where the catalogd JVM {{Xmx}} was set to 2GB, running the end-to-end test {{custom_cluster/test_wide_table_operations.py}} on a table with 2000 columns and 50,000 partitions during the {{recover partitions}} operation caused catalogd to run into a Java heap space {{{}OutOfMemoryError{}}}.

An analysis of the memory dump using the MemoryAnalyzer revealed that the temporary object contained a massive number of {{FieldSchema}} objects (2000 columns * 50,000 partitions), which overwhelmed memory resources.

To resolve this issue, we propose batching the {{addHmsPartitions}} calls, ensuring that temporary objects are released after each batch operation. This solution was tested and verified to resolve the {{{}OutOfMemoryError{}}}, ensuring system stability when handling a large number of partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org