You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues-all@impala.apache.org by "Mihaly Szjatinya (Jira)" <ji...@apache.org> on 2026/03/23 13:38:00 UTC

[jira] [Resolved] (IMPALA-14583) Limit the number of file descriptors per RPC to avoid JVM OOM in Catalog

     [ https://issues.apache.org/jira/browse/IMPALA-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mihaly Szjatinya resolved IMPALA-14583.
---------------------------------------
    Resolution: Fixed

> Limit the number of file descriptors per RPC to avoid JVM OOM in Catalog
> ------------------------------------------------------------------------
>
>                 Key: IMPALA-14583
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14583
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog, Frontend
>            Reporter: Noémi Pap-Takács
>            Assignee: Mihaly Szjatinya
>            Priority: Critical
>              Labels: Catalog, OOM, impala-iceberg
>
> We often get OOM error when Impala tries to load a very large Iceberg table. This happens because the Catalog loads all the file descriptors and sends them to the Coordinator in one RPC, serializing all the file descriptors into one big byte array. However, JVM has a limit on the array length, so trying to send the entire table in one call can exceed this limit if there are too many files in the table.
> We could limit the number of files per RPC, so that the 2GB JVM array limit is not exceeded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org