You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "Zhichao Zhang (Jira)" <ji...@apache.org> on 2021/04/12 01:35:00 UTC

[jira] [Updated] (KYLIN-4967) Forbid to set 'spark.sql.adaptive.enabled' to true when building cube with Spark 2.X

     [ https://issues.apache.org/jira/browse/KYLIN-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhichao  Zhang updated KYLIN-4967:
----------------------------------
    Fix Version/s: v4.0.0-GA

> Forbid to set 'spark.sql.adaptive.enabled' to true when building cube with Spark 2.X
> ------------------------------------------------------------------------------------
>
>                 Key: KYLIN-4967
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4967
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: Zhichao  Zhang
>            Assignee: Zhichao  Zhang
>            Priority: Minor
>             Fix For: v4.0.0-GA
>
>
> With spark 2.X, when set 'spark.sql.adaptive.enabled' to true, it will impact the actually partition count when doing repartition with spark, which will lead to the wrong results for global dict and repartition by shardby column.
> For example, after writing a cuboid data, kylin will repartition the cuboid data with 3 partition if need, but if 'spark.sql.adaptive.enabled' is true, spark will optimize the partition num to 1, which leads to wrong.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)