You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "Zhong Yanghong (JIRA)" <ji...@apache.org> on 2018/03/26 06:22:00 UTC

[jira] [Assigned] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob

     [ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhong Yanghong reassigned KYLIN-2866:
-------------------------------------

    Assignee: Wang Ken  (was: Zhong Yanghong)

> Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
> ------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-2866
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2866
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>            Reporter: Zhong Yanghong
>            Assignee: Wang Ken
>            Priority: Major
>             Fix For: v2.3.0
>
>         Attachments: APACHE-KYLIN-2866-refined.patch, APACHE-KYLIN-2866.patch
>
>
> Currently only one reducer is assigned for hll stats calculation, which may become the bottleneck for slow down this step. Since the stats for different cuboids will not influence each other, it's better to divide the cuboid set into several and assign a reduce for each subset.
> The strategy of this patch is to assign 100 cuboids into a subset. And there's a upper limit of reducers for hll stats calculation. Currently it's 50.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)