You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by sh...@apache.org on 2019/02/22 14:55:23 UTC
[kylin] branch 2.6.x updated: KYLIN-3824
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch 2.6.x
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/2.6.x by this push:
new feb499a KYLIN-3824
feb499a is described below
commit feb499a4528cc52202a20306dce900500ac76c4a
Author: alexandr.sidorchuk <al...@apm-consult.com>
AuthorDate: Fri Feb 22 17:05:08 2019 +0300
KYLIN-3824
For environment with low RAM, and huge Cube, remove List to avoid java.lang.OutOfMemory and iterate by Iterator.
---
.../main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
index 92101b3..d7df9da 100644
--- a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
+++ b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
@@ -310,12 +310,12 @@ public class SparkFactDistinct extends AbstractApplication implements Serializab
}
}
- List<String[]> rows = Lists.newArrayList(rowIterator);
List<Tuple2<SelfDefineSortableKey, Text>> result = Lists.newArrayList();
int rowCount = 0;
- for (String[] row : rows) {
+ while (rowIterator.hasNext()) {
+ String[] row = rowIterator.next();
bytesWritten.add(countSizeInBytes(row));
for (int i = 0; i < allCols.size(); i++) {