carbondata git commit: [CARBONDATA-3122]CarbonReader memory leak

[ra...@apache.org]

Repository: carbondata
Updated Branches:
  refs/heads/master b716865bf -> 877efe1a4


[CARBONDATA-3122]CarbonReader memory leak

Issue Detail
CarbonReader has List of initialized RecordReader for each Split and each split holds page data till the reference of RecordReader is present in the List . Same is applicable for GC once user comes out from his/her calling method ( not cleaned even in close() ) but till then from each split , last page will be in memory which is not correct. For ex. if 1K carbon files then last page ( ~32K * 100 ,size if 100 String columns in memory ) of each file will be in memory till last split so total ~3GB memory will be occupied ( 1K * 32K * 100 .
Check heap dump of 3 split after reader.close() is called ,It is be seen that currentreader+all list reader are still holding memory

Solution

Once reader is finished assign currentReader to null in RecordReader List.
OR
Assign future object as null in org.apache.carbondata.core.scan.processor.DataBlockIterator#close()
Solution 2 is adopted so that it will give benefit to other than CarbonReader Flow.

This closes #2944


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/877efe1a
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/877efe1a
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/877efe1a

Branch: refs/heads/master
Commit: 877efe1a44607aa12edb66f58445db08fd3ca63d
Parents: b716865
Author: BJangir <ba...@gmail.com>
Authored: Thu Nov 22 22:34:32 2018 +0530
Committer: ravipesala <ra...@gmail.com>
Committed: Tue Nov 27 12:14:12 2018 +0530

----------------------------------------------------------------------
 .../apache/carbondata/core/scan/processor/DataBlockIterator.java    | 1 +
 .../main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java  | 1 +
 2 files changed, 2 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/877efe1a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java b/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java
index 1eeb579..4f635c6 100644
--- a/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java
+++ b/core/src/main/java/org/apache/carbondata/core/scan/processor/DataBlockIterator.java
@@ -262,6 +262,7 @@ public class DataBlockIterator extends CarbonIterator<List<Object[]>> {
         if (blockletScannedResult != null) {
           blockletScannedResult.freeMemory();
         }
+        future = null;
       } catch (InterruptedException | ExecutionException e) {
         throw new RuntimeException(e);
       }

http://git-wip-us.apache.org/repos/asf/carbondata/blob/877efe1a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java
----------------------------------------------------------------------
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java
index eca3396..1dfead3 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java
@@ -166,6 +166,7 @@ public class CarbonRecordReader<T> extends AbstractRecordReader<T> {
     }
     // close read support
     readSupport.close();
+    carbonIterator.close();
     try {
       queryExecutor.finish();
     } catch (QueryExecutionException e) {