[carbondata] branch master updated: [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when get data from row page

[qi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

qiangcai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
     new c5c1f59  [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when get data from row page
c5c1f59 is described below

commit c5c1f597737f3bd341eb4088ba530b5a81ece89c
Author: Manhua <ma...@apache.org>
AuthorDate: Wed Jul 15 17:08:11 2020 +0800

    [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when get data from row page
    
    Why is this PR needed?
    PR #3804 free the row page after running in-memory intermeidate merge, but the data will still be used in final sort because that merge only acquire the memory address and row page index instead of moving the rows from here to there.
    
    What changes were proposed in this PR?
    Remove code for freeing the row page in case of in-memory intermeidate merge.
    As a special case, add comment for explanation.
    
    Does this PR introduce any user interface change?
    No
    
    Is any new testcase added?
    No
    
    This closes #3845
---
 .../sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java     | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java b/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java
index 558a40a..f562b90 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java
@@ -147,7 +147,10 @@ public class UnsafeInMemoryIntermediateDataMerger implements Callable<Void> {
 
     // check if there no entry present
     if (!poll.hasNext()) {
-      poll.close();
+      // do not close the poll to free the memory of rowPage here
+      // because data in memory will be used in final merge sort
+      // and close in `UnsafeFinalMergePageHolder` in case of `spillDisk`
+      // is false. Please refer method `writeDataToMemory` in this class
       this.recordHolderHeap.poll();
       // change the file counter
       --this.holderCounter;