You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2013/01/12 22:26:12 UTC

[jira] [Commented] (HIVE-2988) Use of XMLEncoder to serialize MapredWork causes OOM in hive cli

    [ https://issues.apache.org/jira/browse/HIVE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552068#comment-13552068 ] 

Ashutosh Chauhan commented on HIVE-2988:
----------------------------------------

I agree with [~rohini] increasing heap size is not the ideal solution. We should switch to better implementation than XMLEncoder. I don't know much about XStream. Its license is BSD, so its compatible. Does it has better memory footprint or are there other libs which we should look at too?
                
> Use of XMLEncoder to serialize MapredWork causes OOM in hive cli
> ----------------------------------------------------------------
>
>                 Key: HIVE-2988
>                 URL: https://issues.apache.org/jira/browse/HIVE-2988
>             Project: Hive
>          Issue Type: Improvement
>          Components: CLI
>            Reporter: Rohini Palaniswamy
>              Labels: Performance
>
> When running queries on tables with 6000 partitions, hive cli if configured with 128M runs into OOM. Heapdump showed 37MB occupied by one XMLEncoder object while the MapredWork was 500K which is highly inefficient. We should switch to using something more efficient like XStream. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira