[jira] [Created] (KYLIN-5044) Kylin 3.1.2 - Cube processing fails on Step 4 when hive client is switched to beeline.

["Piotr Naszarkowski (Jira)" <ji...@apache.org>]

Piotr Naszarkowski created KYLIN-5044:
-----------------------------------------

             Summary: Kylin 3.1.2 - Cube processing fails on Step 4 when hive client is switched to beeline.
                 Key: KYLIN-5044
                 URL: https://issues.apache.org/jira/browse/KYLIN-5044
             Project: Kylin
          Issue Type: Bug
          Components: Job Engine
    Affects Versions: v3.1.2
         Environment: HDP-2.6.5.0
(2.6.5.0-292)
Kylin 3.1.2 

Centos 7
            Reporter: Piotr Naszarkowski


I switched hive client to beeline and also enabled sparksql for hive source using settings like below:

kylin.source.hive.client=beeline
 kylin.source.hive.beeline-shell=beeline
 kylin.source.hive.beeline-params=-n hive -u jdbc:hive2://hdp...:10016
 kylin.source.hive.enable-sparksql-for-table-ops=true
 kylin.source.hive.sparksql-beeline-shell=beeline

This caused Stage 4 (#4 Step Name: Build Dimension Dictionary) of cube processing to fail, as well as lookup refresh. The error:
{code:java}
org.apache.kylin.engine.mr.exception.HadoopShellException: java.io.IOException: java.lang.NullPointerExceptionorg.apache.kylin.engine.mr.exception.HadoopShellException: java.io.IOException: java.lang.NullPointerException at org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:78) at org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:73) at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:140) at org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1260) at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:1164) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:123) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.NullPointerException at org.apache.kylin.common.util.HadoopUtil.fixWindowsPath(HadoopUtil.java:122) at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:114) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:92) at org.apache.kylin.engine.mr.DFSFileTable.getSizeAndLastModified(DFSFileTable.java:90) at org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:63) ... 16 more
result code:2 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:74) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code}
Investigation showed that my beeline returns metadata in different format than the one used in code in BeelineHiveClient.java, basically sdLocation was not retrieved leading to NPE. This happens on LEARN_KYLIN sample project. I wonder if that something to do with my environment or it's simply a bug (seems severe).

Proposed code changes: [https://github.com/apache/kylin/pull/1698/files] (needs to be rebased with proper KYLIN jira id in commit) that resolved the issue for me.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)