[jira] [Commented] (HIVE-24316) Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1

["Sungwoo (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-24316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403893#comment-17403893 ] 

Sungwoo commented on HIVE-24316:
--------------------------------

Hello,

It seems that with ORC-361, the use of MemoryManagerImpl in LlapAwareMemoryManager is inconsistent. 

Before merging ORC-361, LlapAwareMemoryManager sets its own totalMemoryPool and MemoryManagerImpl accesses totalMemoryPool via getTotalMemoryPool(), so everything is fine.

With ORC-361 merged, we have the following:

1. LlapAwareMemoryManager sets its own totalMemoryPool as a private field.
 2. MemoryManagerImpl sets its own totalMemoryPool as a private field.
 3. LlapAwareMemoryManager overrides getTotalMemoryPool() using its own totalMemoryPool.

Now it is unclear whether or not getTotalMemoryPool() should be overridden.

Here are my thoughts on ORC-361:

1. Is MemoryManagerImpl intended to coordinate all threads writing to ORC files inside a process (like LLAP Daemon)? Then is it necessary to create LlapAwareMemoryManager as a ThreadLocal object? Why not just call OrcFile.getStaticMemoryManager() to obtain the shared MemoryManagerImpl?

3. LlapAwareMemoryManager sets its own totalMemoryPool:
{code:java}
      long memPerExecutor = LlapDaemonInfo.INSTANCE.getMemoryPerExecutor();
      totalMemoryPool = (long) (memPerExecutor * maxLoad);
{code}
From my understanding, this has no effect because MemoryManagerImpl sets its own totalMemoryPool.

Any comment would be appreciated.

> Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
> ---------------------------------------------
>
>                 Key: HIVE-24316
>                 URL: https://issues.apache.org/jira/browse/HIVE-24316
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 3.1.3
>            Reporter: Dongjoon Hyun
>            Assignee: Dongjoon Hyun
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.1.3
>
>          Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> This will bring eleven bug fixes.
>  * ORC 1.5.7: [https://issues.apache.org/jira/projects/ORC/versions/12345702]
>  * ORC 1.5.8: [https://issues.apache.org/jira/projects/ORC/versions/12346462]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)