[jira] [Commented] (KYLIN-4035) Calculate column cardinality by using spark engine

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859102#comment-16859102 ] 

ASF GitHub Bot commented on KYLIN-4035:
---------------------------------------

majic31 commented on pull request #678: KYLIN-4035 Calculate column cardinality by using spark engine
URL: https://github.com/apache/kylin/pull/678
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Calculate column cardinality by using spark engine
> --------------------------------------------------
>
>                 Key: KYLIN-4035
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4035
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Spark Engine
>    Affects Versions: Future
>         Environment: kylin: master/3.0.0-alpha
> spark: 2.4.3
> hadoop: 2.6.5
>            Reporter: Jack
>            Priority: Minor
>             Fix For: Future
>
>
> Kylin will calculate column cardinality when loading hive table. This stage is only supported by MR engine without spark. I think spark engine should be used in this stage because of the following:
> 1) Kylin users can choose which engine they apply when calculating column cardinality;
> 2) Some good spark features(e.g. dynamic resource allocation) can be used; 
> 3) The code written in spark is simple.
> I finish this work and test ok. But "kylin.engin.spark-cardinality=true" should be added in kylin.properties(default is false). Look forwards to suggestions.
> Best regards. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)