You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Prasanth Jayachandran (JIRA)" <ji...@apache.org> on 2018/09/28 04:03:00 UTC
[jira] [Comment Edited] (HIVE-20649) LLAP aware memory manager for
Orc writers
[ https://issues.apache.org/jira/browse/HIVE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631320#comment-16631320 ]
Prasanth Jayachandran edited comment on HIVE-20649 at 9/28/18 4:02 AM:
-----------------------------------------------------------------------
[~sershe] can you please review? small patch. This patch needs ORC-409 (also needs review :)) for it to work correctly (scaling the stripe size).
was (Author: prasanth_j):
[~sershe] can you please review? small patch. This patch needs ORC-409 for it to work correctly (scaling the stripe size).
> LLAP aware memory manager for Orc writers
> -----------------------------------------
>
> Key: HIVE-20649
> URL: https://issues.apache.org/jira/browse/HIVE-20649
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.0.0, 3.2.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
> Attachments: HIVE-20649.1.patch
>
>
> ORC writer has its own memory manager that assumes memory usage or memory available based on JVM heap (MemoryMX bean). This works on tez container mode execution model but not in LLAP where container sizes (and Xmx) are typically high and there are multiple executors per LLAP daemon. This custom memory manager should be aware of memory bounds per executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)