[jira] [Commented] (KYLIN-2135) Enlarge FactDistinctColumns reducer number

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624018#comment-15624018 ] 

Shaofeng SHI commented on KYLIN-2135:
-------------------------------------

which is the base version that making the patch? I tried to apply it on master and "kylin-1.5.4.1', all got failed:

{code}
bogon:kylin-15 shishaofeng$ git am -s -3 ~/Documents/KYLIN-2135.patch --ignore-whitespace
Applying: KYLIN-2135 Enlarge FactDistinctColumns reducer number
Using index info to reconstruct a base tree...
M	core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
M	core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
M	engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnPartitioner.java
M	engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsJob.java
M	engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsReducer.java
M	engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctHiveColumnsMapper.java
Falling back to patching base and 3-way merge...
Auto-merging engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctHiveColumnsMapper.java
CONFLICT (content): Merge conflict in engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctHiveColumnsMapper.java
Auto-merging engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsReducer.java
CONFLICT (content): Merge conflict in engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsReducer.java
Auto-merging engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsJob.java
CONFLICT (content): Merge conflict in engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsJob.java
Auto-merging engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnPartitioner.java
CONFLICT (content): Merge conflict in engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnPartitioner.java
Auto-merging core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
CONFLICT (content): Merge conflict in core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
Auto-merging core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
error: Failed to merge in the changes.
Patch failed at 0001 KYLIN-2135 Enlarge FactDistinctColumns reducer number
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort"
{code}

> Enlarge FactDistinctColumns reducer number
> ------------------------------------------
>
>                 Key: KYLIN-2135
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2135
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>    Affects Versions: v1.5.4.1
>            Reporter: kangkaisen
>            Assignee: kangkaisen
>         Attachments: KYLIN-2135.patch, new.png, old.png
>
>
> When the hive table has billions of rows and use global dictionary for precise count distinct measures, the  {{Extract Fact Table Distinct Columns}} job will run o long time.
> So we could use more reducer to deal with the one column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)