You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2018/05/03 16:50:00 UTC

[jira] [Resolved] (IMPALA-1803) Avoid hitting OOM in HdfsTableSink when inserting to Parquet

     [ https://issues.apache.org/jira/browse/IMPALA-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-1803.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.0

IMPALA-4899 and IMPALA-5293 should address this.

> Avoid hitting OOM in HdfsTableSink when inserting to Parquet
> ------------------------------------------------------------
>
>                 Key: IMPALA-1803
>                 URL: https://issues.apache.org/jira/browse/IMPALA-1803
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.0
>            Reporter: Ippokratis Pandis
>            Priority: Major
>              Labels: resource-management, usability
>             Fix For: Impala 3.0
>
>         Attachments: hdfstablesink-oom.txt
>
>
> Impala's memory consumption is very high when it writes to Parquet and there is a large number of partitions, primarily because we try to buffer data per partition. That however can lead to OOM, see attached profile. Instead we can either spill the buffered data to disk or write to Parquet files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)