You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by qiumingming <cs...@gmail.com> on 2017/06/02 08:47:05 UTC
File does not exist: hdfs:/kylin/spark-history
Hi All, I got a error at Build Cube with Spark step, below is my error info.
I can not search anything which match "hdfs:/kylin/spark-history" in the
project source code:
ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File does not exist:
hdfs:/kylin/spark-history
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1132)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:100)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:154)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/File-does-not-exist-hdfs-kylin-spark-history-tp8154.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: File does not exist: hdfs:/kylin/spark-history
Posted by qiumingming <cs...@gmail.com>.
I steel have error when cubing with spark. I am working on it, hope it can be
solved today.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Re-File-does-not-exist-hdfs-kylin-spark-history-tp8159p8173.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: File does not exist: hdfs:/kylin/spark-history
Posted by ShaoFeng Shi <sh...@apache.org>.
Cool, how about the Spark engine's performance in your scenario?
2017-06-03 17:20 GMT+08:00 qiumingming <cs...@gmail.com>:
> I have solved this problem yesterday. Thank you all the same.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Re-File-does-not-exist-hdfs-kylin-spark-
> history-tp8159p8164.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: File does not exist: hdfs:/kylin/spark-history
Posted by qiumingming <cs...@gmail.com>.
I have solved this problem yesterday. Thank you all the same.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Re-File-does-not-exist-hdfs-kylin-spark-history-tp8159p8164.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: File does not exist: hdfs:/kylin/spark-history
Posted by ShaoFeng Shi <sh...@apache.org>.
It was a configuration in kylin.properties:
kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history
2017-06-02 16:47 GMT+08:00 qiumingming <cs...@gmail.com>:
> Hi All, I got a error at Build Cube with Spark step, below is my error
> info.
> I can not search anything which match "hdfs:/kylin/spark-history" in the
> project source code:
> ERROR spark.SparkContext: Error initializing SparkContext.
> java.io.FileNotFoundException: File does not exist:
> hdfs:/kylin/spark-history
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.
> doCall(DistributedFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.
> doCall(DistributedFileSystem.java:1124)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:1124)
> at
> org.apache.spark.scheduler.EventLoggingListener.start(
> EventLoggingListener.scala:100)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(
> JavaSparkContext.scala:59)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:154)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/File-does-not-exist-hdfs-kylin-spark-history-tp8154.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋