You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Abhilash L L (JIRA)" <ji...@apache.org> on 2015/09/29 07:34:04 UTC

[jira] [Commented] (PARQUET-108) Parquet Memory Management in Java

    [ https://issues.apache.org/jira/browse/PARQUET-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934654#comment-14934654 ] 

Abhilash L L commented on PARQUET-108:
--------------------------------------

Hello,

    We are creating parquet files from within our MR code as one of the MultipleOutputs. We observer that it starts giving OOM when the number of partitions exceeded. We are using http://mvnrepository.com/artifact/com.twitter/parquet-avro/1.6.0

    Will this work in our flow and not necessarily being written via hive ?


PS: Sorry for adding a comment on a resolved ticket 

> Parquet Memory Management in Java
> ---------------------------------
>
>                 Key: PARQUET-108
>                 URL: https://issues.apache.org/jira/browse/PARQUET-108
>             Project: Parquet
>          Issue Type: Improvement
>            Reporter: Brock Noland
>            Assignee: Dong Chen
>             Fix For: 1.6.0
>
>
> As discussed in HIVE-7685, Hive + Parquet often runs out memory when writing to many Hive partitions. This is quite problematic for our users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)