You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2025/02/24 13:50:00 UTC

[jira] [Created] (IMPALA-13789) Avoid holding lots of org.apache.hadoop.fs.Path objects in memory

Quanlong Huang created IMPALA-13789:
---------------------------------------

             Summary: Avoid holding lots of org.apache.hadoop.fs.Path objects in memory
                 Key: IMPALA-13789
                 URL: https://issues.apache.org/jira/browse/IMPALA-13789
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Quanlong Huang
            Assignee: Quanlong Huang
         Attachments: histogram_path_objects.png

When loading file metadata of a table, we create several Java Maps that using org.apache.hadoop.fs.Path as the key type, e.g. in ParallelFileMetadataLoader:
{code:java}
  private final Map<Path, FileMetadataLoader> loaders_;
  private final Map<Path, List<HdfsPartition.Builder>> partsByPath_;{code}
Keeping these Path objects in memory is expensive as there are as many of them as the number of partitions.

The following histogram shows that 4.3M of such Path objects takes 3GB in memory:
!histogram_path_objects.png!

We can use the String of the partition location as the key type and only create Path objects when loading that partition.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org