[jira] [Commented] (KYLIN-2736) Use multiple threads to calculate HyperLogLogPlusCounter in FactDistinctColumnsMapper

["Zhong Yanghong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285535#comment-16285535 ] 

Zhong Yanghong commented on KYLIN-2736:
---------------------------------------

Yes, this will cause CPU over-subscription. However, I think there should be a way to enable this feature to speed up the process.

The thread number mainly depends on the how many cuboids are there whose statistics needs to be calculated. The detailed code is as follows:
{code}
{
        int unitNum = cubeDesc.getConfig().getCuboidNumberPerStatisticsCalculator();
        if (unitNum <= 0) {
            logger.warn("config from getCuboidNumberPerStatisticsCalculator() " + unitNum + " is should larger than 0");
            logger.info("Will use single thread for cuboid statistics calculation");
            return 1;
        }

        int calculatorNum = (cuboidNum - 1) / unitNum + 1;
        if (cuboidNum > cubeDesc.getConfig().getCuboidStatisticsCalculatorMaxNumber()) {
            calculatorNum = cubeDesc.getConfig().getCuboidStatisticsCalculatorMaxNumber();
        }
        return calculatorNum;
}
{code}

> Use multiple threads to calculate HyperLogLogPlusCounter in FactDistinctColumnsMapper
> -------------------------------------------------------------------------------------
>
>                 Key: KYLIN-2736
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2736
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Zhong Yanghong
>            Assignee: Zhong Yanghong
>             Fix For: v2.3.0
>
>         Attachments: APACHE-KYLIN-2736.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)