[jira] [Created] (ORC-846) Refactoring Memory-Manager for better extensibility

["Lei Sun (Jira)" <ji...@apache.org>]

Lei Sun created ORC-846:
---------------------------

             Summary: Refactoring Memory-Manager for better extensibility 
                 Key: ORC-846
                 URL: https://issues.apache.org/jira/browse/ORC-846
             Project: ORC
          Issue Type: Improvement
            Reporter: Lei Sun
            Assignee: Lei Sun


Hi ORC Community, 

In our use case, dynamic partitioning is quite common and our engine ([Gobblin|[https://github.com/apache/gobblin])] doesn't have a shuffle stage so that having single thread dealing with multiple writers (dealing with different partitions) is a usual pattern. When the number of partitions reach a large number, there will be challenges in avoiding OOM given the current memory-manager implementation: 
 * `rows.between.memory.check` is on writer level (the condition gating the expensive check between estimated memory of treeWriter versus memoryLimit of each writer), and memory manager is not aware of it.  memory manager only control the "scale" of allocation which hints each writer to reduce the threshold of memory limit (initial to StripSize) 

 

What is proposed here is to: 
 * Having centralized awareness of how many rows have been bufferred among all writers, in the memoryManager. The data structure being used will be thread-safe so that the issue fixed in https://issues.apache.org/jira/browse/ORC-361# won't re-surface. There's no additional synchronization introduced beyond the intrinsic control from the concurrent data structure managing how much rows buffered in each writer. The existing memory-manager will be interfacing with each writer in a backward-compatible way. 

 

With this, the existing memory manager can be extended in terms of controlling flush in each -writer's granularity and treat different writers with priority.  e.g. some of the writers dealing with less favored partition could be flushed more often so that the overall pressure on memory could be reduced, all these "prioritization" should be localized to the engines that uses orc-writer. 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)