You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "wangrupeng (Jira)" <ji...@apache.org> on 2019/10/15 05:39:00 UTC

[jira] [Updated] (KYLIN-4194) Throw KylinConfigCannotInitException at STEP "Extract Fact Table Distinct Columns" with spark

     [ https://issues.apache.org/jira/browse/KYLIN-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

wangrupeng updated KYLIN-4194:
------------------------------
    Description: 
When I use spark to extract fact table distinct columns values, kylin throw an exception,

org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,

see more details in [^kylin_spark.log] ^^ . I tried cdh5.7 and hdp2.4, both platforms got this problem.

It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.

As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build  Dictionary<String> objects,  but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method. 

My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE 

from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.

  was:
When I use spark to extract fact table distinct columns values, kylin throw an exception,

org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,

see more details in [^kylin_spark.log] ^^ . I tried cdh5.7 and hdp2.4, both platforms go this problem.

It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.

As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build  Dictionary<String> objects,  but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method. 

My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE 

from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.


> Throw KylinConfigCannotInitException at STEP "Extract Fact Table Distinct Columns" with spark
> ---------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-4194
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4194
>             Project: Kylin
>          Issue Type: Bug
>    Affects Versions: v3.0.0-beta, v2.6.4
>            Reporter: wangrupeng
>            Assignee: wangrupeng
>            Priority: Major
>         Attachments: kylin_spark.log
>
>
> When I use spark to extract fact table distinct columns values, kylin throw an exception,
> org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,
> see more details in [^kylin_spark.log] ^^ . I tried cdh5.7 and hdp2.4, both platforms got this problem.
> It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.
> As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build  Dictionary<String> objects,  but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method. 
> My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE 
> from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)