You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by sh...@apache.org on 2019/02/22 14:55:23 UTC

[kylin] branch 2.6.x updated: KYLIN-3824

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.6.x
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/2.6.x by this push:
     new feb499a  KYLIN-3824
feb499a is described below

commit feb499a4528cc52202a20306dce900500ac76c4a
Author: alexandr.sidorchuk <al...@apm-consult.com>
AuthorDate: Fri Feb 22 17:05:08 2019 +0300

    KYLIN-3824
    
    For environment with low RAM, and huge Cube, remove List to avoid java.lang.OutOfMemory and iterate by Iterator.
---
 .../main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java    | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
index 92101b3..d7df9da 100644
--- a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
+++ b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
@@ -310,12 +310,12 @@ public class SparkFactDistinct extends AbstractApplication implements Serializab
                 }
             }
 
-            List<String[]> rows = Lists.newArrayList(rowIterator);
             List<Tuple2<SelfDefineSortableKey, Text>> result = Lists.newArrayList();
 
             int rowCount = 0;
 
-            for (String[] row : rows) {
+            while (rowIterator.hasNext()) {
+                String[] row = rowIterator.next();
                 bytesWritten.add(countSizeInBytes(row));
 
                 for (int i = 0; i < allCols.size(); i++) {