You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ZhouKang (Jira)" <ji...@apache.org> on 2019/11/26 03:18:00 UTC
[jira] [Updated] (KYLIN-4268) build job failed caused by java lib
update
[ https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZhouKang updated KYLIN-4268:
----------------------------
Description:
When we update hadoop package, such as hbase and hive. The job will fill cause by:
{code:java}
// code placeholder
Exception in thread "main" java.io.FileNotFoundException: File file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
at org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
{code}
The root cause is kylin will generate the full command as soon as the job summited. And the parameter '--jars' use file's absolute path.
there will be 2 ways to fix this problem.
# keep the old package in the same path
# change the generation of '--jars' to the stage of job execution.
which is the better choose?
was:
When we update hadoop package, such as hbase and hive. The job will fill cause by:
{code:java}
// code placeholder
Exception in thread "main" java.io.FileNotFoundException: File file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
at org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
{code}
The root cause is kylin will generate the full command as soon as the job summited. And the parameter '--jars' use file's absolute path.
there will be 2 ways to fix this problem.
# keep the old package in the same path
# change the generation of '--jars' to the stage of job execution.
which is the better choose?
> build job failed caused by java lib update
> ------------------------------------------
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
> Reporter: ZhouKang
> Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)