You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "anishek (JIRA)" <ji...@apache.org> on 2017/06/15 05:38:00 UTC

[jira] [Assigned] (HIVE-16904) during repl load for large number of partitions the metadata file can be huge and can lead to out of memory

     [ https://issues.apache.org/jira/browse/HIVE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

anishek reassigned HIVE-16904:
------------------------------


> during repl load for large number of partitions the metadata file can be huge and can lead to out of memory 
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16904
>                 URL: https://issues.apache.org/jira/browse/HIVE-16904
>             Project: Hive
>          Issue Type: Sub-task
>    Affects Versions: 3.0.0
>            Reporter: anishek
>            Assignee: anishek
>             Fix For: 3.0.0
>
>
> the metadata pertaining to a table + its partitions is stored in a single file, During repl load all the data is loaded in memory in one shot and then individual partitions processed. This can lead to huge memory overhead as the entire file is read in memory. try to deserialize the partition objects with some sort of streaming json deserializer. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)