[jira] [Created] (KYLIN-2136) Enhance cubing algorithm selection

["liyang (JIRA)" <ji...@apache.org>]

liyang created KYLIN-2136:
-----------------------------

             Summary: Enhance cubing algorithm selection
                 Key: KYLIN-2136
                 URL: https://issues.apache.org/jira/browse/KYLIN-2136
             Project: Kylin
          Issue Type: Improvement
            Reporter: liyang
            Assignee: liyang


To be more protective to select in-mem cubing.

There were reports that in-mem cubing was selected but run slowly (even timeout) in a few small PoC clusters. Apart from the limited resource in PoC, another important cause is the algorithm selection didn't concern the scale of the job. When mappers are many but mapper slots are few, it will take many rounds to finish all the mappers, and the total time of in-mem cubing becomes slow.

The scale of the job should be taken into consideration when auto selecting cubing algorithm. Namely a mapper limit configuration is introduced, to stop in-mem cubing when the number of mappers goes beyond the threshold.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)