[jira] [Resolved] (KYLIN-5646) The build job reports an error at the step of detecting time partition columns in the Yarn Cluster mode

["Xiaoxiang Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaoxiang Yu resolved KYLIN-5646.
---------------------------------
    Resolution: Fixed

> The build job reports an error at the step of detecting time partition columns in the Yarn Cluster mode
> -------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-5646
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5646
>             Project: Kylin
>          Issue Type: Bug
>          Components: Tools, Build and Test
>    Affects Versions: 5.0-alpha
>            Reporter: Zhiting Guo
>            Assignee: Zhiting Guo
>            Priority: Major
>             Fix For: 5.0-beta
>
>
> When building Spark YARN-Cluster mode, when detecting incremental time partition columns, initializing KylinConfig reports an error Didn't find KYLIN_HOME or KYLIN_HOME
> *Reproduce method*
> Build the partition table model incrementally using Spark YARN_Cluster mode, and set kylin.engine.check-partition-col-enabled=true (the default value is true)
> *Root Cause*
> Modified the autoSetShufflePartitions of the pushdown query in [KYLIN-5571], no need to execute when the pre-modification build task detects the delta time column format (only the pushdown query is executed)
> After modification, autoSetShufflePartitions is executed asynchronously, the following two methods will get KylinConfig through KylinConfig.getInstanceFromEnv,
> At this time, the asynchronous execution of the new thread cannot use the built KylinConfig, so the KylinConfig will be initialized,
> However, the build task jvm and the KE main process are not the same machine, and KYLIN_CONF and KYLIN_HOME cannot be obtained, so the build task fails to run
>  * ResourceDetectUtils.getResourceSizeWithTimeoutByConcurrency
>  * ResourceDetectUtils.getResourceSizBySerial
> *fix design*
> In all the logic of newly opened threads, if KylinConfig is used, this method KylinConfig.getInstanceFromEnv() is not used. Unified is obtained by an external thread and passed to the place where it needs to be used



--
This message was sent by Atlassian Jira
(v8.20.10#820010)