You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Noemi Pap-Takacs (Jira)" <ji...@apache.org> on 2025/01/13 11:45:00 UTC

[jira] [Work started] (IMPALA-13656) MERGE redundantly accumulates memory in HDFS WRITER

     [ https://issues.apache.org/jira/browse/IMPALA-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on IMPALA-13656 started by Noemi Pap-Takacs.
-------------------------------------------------
> MERGE redundantly accumulates memory in HDFS WRITER
> ---------------------------------------------------
>
>                 Key: IMPALA-13656
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13656
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>            Reporter: Noemi Pap-Takacs
>            Assignee: Noemi Pap-Takacs
>            Priority: Major
>             Fix For: Impala 4.5.0
>
>
> When we want to merge Iceberg tables that have lots of partitions, the execution will use much more memory than needed, possibly resulting in a Memory Limit Exceeded error.
> It happens because when IcebergMergeImpl creates the insert table sink it doesn't set 'inputIsClustered' to true. Therefore HdfsTableSink expects random input and keeps the output writers open for every partition, which results in high memory consumption.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org