[carbondata] branch master updated: [HOTFIX][DataLoad]fix task assignment issue using NODE_MIN_SIZE_FIRST block assignment strategy

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

manishgupta88 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
     new 39cd2f4  [HOTFIX][DataLoad]fix task assignment issue using NODE_MIN_SIZE_FIRST block assignment strategy
39cd2f4 is described below

commit 39cd2f44cf17d3817d27a11a8ca871fac2500794
Author: ndwangsen <lu...@huawei.com>
AuthorDate: Wed Jan 9 18:44:08 2019 +0800

    [HOTFIX][DataLoad]fix task assignment issue using NODE_MIN_SIZE_FIRST block assignment strategy
    
    This PR sloves the problem of incorrect assignment of tasks if specified minimum data size to load is less than the average size for each node.
    
    This closes #3059
---
 .../org/apache/carbondata/processing/util/CarbonLoaderUtil.java   | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java b/processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java
index 23e4a8f..0ff3eb6 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java
@@ -609,6 +609,14 @@ public final class CarbonLoaderUtil {
           blockAssignmentStrategy = BlockAssignmentStrategy.BLOCK_SIZE_FIRST;
         } else {
           blockAssignmentStrategy = BlockAssignmentStrategy.BLOCK_NUM_FIRST;
+          // fall back to BLOCK_NUM_FIRST strategy need to reset
+          // the average expected size for each node
+          if (numOfNodes == 0) {
+            sizePerNode = 1;
+          } else {
+            sizePerNode = blockInfos.size() / numOfNodes;
+            sizePerNode = sizePerNode <= 0 ? 1 : sizePerNode;
+          }
         }
         LOGGER.info("Specified minimum data size to load is less than the average size "
             + "for each node, fallback to default strategy" + blockAssignmentStrategy);