You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "kangkaisen (JIRA)" <ji...@apache.org> on 2016/10/28 04:12:58 UTC
[jira] [Comment Edited] (KYLIN-2135) Enlarge FactDistinctColumns
reducer number
[ https://issues.apache.org/jira/browse/KYLIN-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15614228#comment-15614228 ]
kangkaisen edited comment on KYLIN-2135 at 10/28/16 4:12 AM:
-------------------------------------------------------------
As shown in the old.png, this is the {{Extract Fact Table Distinct Columns}} reducers elapsed time in 1.5.4.1 version.
The 14th reducer elapsed almosts 200 minutes. Because the hive table has 2536614834 rows and the 14th column has 28323064 cardinalities.
was (Author: kangkaisen):
This is the {{Extract Fact Table Distinct Columns}} reducers elapsed time in 1.5.4.1 version.
The 14th reducer elapsed almosts 200 minutes. Because the hive table has 2536614834 rows and the 14th column has 28323064 cardinalities.
> Enlarge FactDistinctColumns reducer number
> ------------------------------------------
>
> Key: KYLIN-2135
> URL: https://issues.apache.org/jira/browse/KYLIN-2135
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v1.5.4.1
> Reporter: kangkaisen
> Assignee: kangkaisen
> Attachments: new.png, old.png
>
>
> When the hive table has billions of rows and use global dictionary for precise count distinct measures, the {{Extract Fact Table Distinct Columns}} job will run o long time.
> So we could use more reducer to deal with the one column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)