You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "kangkaisen (JIRA)" <ji...@apache.org> on 2016/10/28 04:08:58 UTC

[jira] [Updated] (KYLIN-2135) Enlarge FactDistinctColumns reducer number

     [ https://issues.apache.org/jira/browse/KYLIN-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

kangkaisen updated KYLIN-2135:
------------------------------
    Attachment: old.png

This is the {{Extract Fact Table Distinct Columns}} reducers elapsed time in 1.5.4.1 version.

The  14th reducer elapsed almosts 200 minutes. Because the hive table has 2536614834 rows and the 14th column has 28323064 cardinalities.

> Enlarge FactDistinctColumns reducer number
> ------------------------------------------
>
>                 Key: KYLIN-2135
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2135
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>    Affects Versions: v1.5.4.1
>            Reporter: kangkaisen
>            Assignee: kangkaisen
>         Attachments: old.png
>
>
> When the hive table has billions of rows and use global dictionary for precise count distinct measures, the  {{Extract Fact Table Distinct Columns}} job will run o long time.
> So we could use more reducer to deal with the one column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)