You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ja...@apache.org on 2018/10/18 12:49:57 UTC
carbondata git commit: [CARBONDATA-3026] clear expired property that
may cause GC problem
Repository: carbondata
Updated Branches:
refs/heads/master 0e8588744 -> 1be990f66
[CARBONDATA-3026] clear expired property that may cause GC problem
During data loading, we will write some temp files (sort temp
files and temp fact data files) in some locations. In currently
implementation, we will add the locations to the CarbonProperties and
associated it with a special key that refers to the data loading.
After data loading, the temp locations are cleared, but the added
property is still remain in the CarbonProperties and never to be cleared.
This will cause the CarbonProperties object growing bigger and bigger
and lead to OOM problems if the thrift-server is a long time running
service. A local test shows that after adding different properties for
11 Billion times, the OOM happens.
In this commit, I clear the property for the locations when we clear the
locations.
This closes #2833
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/1be990f6
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/1be990f6
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/1be990f6
Branch: refs/heads/master
Commit: 1be990f66296174a34044d67c79cffae37f06e86
Parents: 0e85887
Author: xuchuanyin <xu...@hust.edu.cn>
Authored: Thu Oct 18 14:12:30 2018 +0800
Committer: Jacky Li <ja...@qq.com>
Committed: Thu Oct 18 20:49:44 2018 +0800
----------------------------------------------------------------------
.../org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java | 3 ---
.../carbondata/processing/loading/TableProcessingOperations.java | 1 +
2 files changed, 1 insertion(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/carbondata/blob/1be990f6/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
----------------------------------------------------------------------
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
index f0ad94d..f0f2858 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
@@ -267,9 +267,6 @@ public class CarbonTableOutputFormat extends FileOutputFormat<NullWritable, Obje
iterator.closeWriter(true);
}
dataLoadExecutor.close();
- // clean up the folders and files created locally for data load operation
- TableProcessingOperations.deleteLocalDataLoadFolderLocation(loadModel, false, false);
-
throw new RuntimeException(e);
} finally {
ThreadLocalSessionInfo.unsetAll();
http://git-wip-us.apache.org/repos/asf/carbondata/blob/1be990f6/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java
----------------------------------------------------------------------
diff --git a/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java b/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java
index 89da224..f08de59 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java
@@ -146,6 +146,7 @@ public class TableProcessingOperations {
}
});
} finally {
+ CarbonProperties.getInstance().removeProperty(tempLocationKey);
if (null != localFolderDeletionService) {
localFolderDeletionService.shutdown();
}