You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 邵志鹏 <bo...@163.com> on 2019/08/14 06:08:03 UTC
Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/api/java/function/Function
Hi kylin team:
kylin-2.6.3
使用MapReduce已经成功构建smaple cube。
然后clone了一份kylin_sales_cube修改为Spark，build 报错。
Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
$KYLIN_HOME/bin/download-spark.sh
jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
hadoop fs -mkdir -p /kylin/spark/
hadoop fs -put spark-libs.jar /kylin/spark/
...
报错日志：
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91]
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91]
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
报错之后，再去提交build，又报新的错：
Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of them is READY. It's not allowed for parallel building
at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Monitor里面是没有新的构建任务的。
Thanks.
By 邵志鹏
Re:Re: Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/api/java/function/Function
Posted by 邵志鹏 <bo...@163.com>.
设置了kylin.engine.spark-conf.spark.yarn.archive，我也很奇怪，看上去就是没有加载到spark-libs.jar。
## manually upload spark-assembly jar to HDFS and then set this property will avoid repeatedly uploading jar at runtime
kylin.engine.spark-conf.spark.yarn.archive=hdfs://master:9000/kylin/spark/spark-libs.jar
$ hadoop fs -ls hdfs://master:9000/kylin/spark/
Found 1 items
-rw-r--r-- 3 admin supergroup 200878688 2019-08-14 17:16 hdfs://master:9000/kylin/spark/spark-libs.jar
在 2019-08-14 14:46:37，"Chao Long" <ch...@gmail.com> 写道：
这个参数配置了么，kylin.engine.spark-conf.spark.yarn.archive，是否配置为自己集群所在的hdfs目录，而不是文档上的hdfs://sandbox.hortonworks.com:8020
On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 <bo...@163.com> wrote:
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Updating cube instance 'kylin_sales_cube_clone_spark'
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Remove segment kylin_sales_cube_clone_spark[20120101000000_20140201235500]
Action - Delete Segement can Remove segment , and then rebuild it.
Main problem is
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function。
不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。
spark版本：2.3.2
spark-core_2.11-2.3.2
在 2019-08-14 14:08:03，"邵志鹏" <bo...@163.com> 写道：
Hi kylin team:
kylin-2.6.3
使用MapReduce已经成功构建smaple cube。
然后clone了一份kylin_sales_cube修改为Spark，build 报错。
Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
$KYLIN_HOME/bin/download-spark.sh
jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
hadoop fs -mkdir -p /kylin/spark/
hadoop fs -put spark-libs.jar /kylin/spark/
...
报错日志：
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91]
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91]
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
报错之后，再去提交build，又报新的错：
Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of them is READY. It's not allowed for parallel building
at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Monitor里面是没有新的构建任务的。
Thanks.
By 邵志鹏
Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
Posted by Chao Long <ch...@gmail.com>.
这个参数配置了么，*kylin.engine.spark-conf.spark.yarn.archive，*
是否配置为自己集群所在的hdfs目录，而不是文档上的hdfs://sandbox.hortonworks.com:8020
On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 <bo...@163.com> wrote:
> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Updating cube instance
> 'kylin_sales_cube_clone_spark'
> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Remove segment
> kylin_sales_cube_clone_spark[20120101000000_20140201235500]
>
> Action - Delete Segement can Remove segment , and then rebuild it.
>
> Main problem is
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function。
>
> 不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。
> spark版本：2.3.2
> spark-core_2.11-2.3.2
>
>
> 在 2019-08-14 14:08:03，"邵志鹏" <bo...@163.com> 写道：
>
> Hi kylin team:
>
> kylin-2.6.3
> 使用MapReduce已经成功构建smaple cube。
> 然后clone了一份kylin_sales_cube修改为Spark，build 报错。
>
> Spark是使用脚本下载的，配置参考了
> http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
>
>
> $KYLIN_HOME/bin/download-spark.sh
> jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
> hadoop fs -mkdir -p /kylin/spark/
> hadoop fs -put spark-libs.jar /kylin/spark/
> ...
>
> 报错日志：
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/function/Function
> at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> ... 77 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
> ~[catalina.jar:7.0.91]
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
> ~[catalina.jar:7.0.91]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>
>
> 报错之后，再去提交build，又报新的错：
> Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube
> kylin_sales_cube_clone_spark has segments
> [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of
> them is READY. It's not allowed for parallel building
> at
> org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> ... 77 more
>
>
> Monitor里面是没有新的构建任务的。
>
> Thanks.
>
> By 邵志鹏
>
>
>
>
>
>
>
>
>
Re:Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/api/java/function/Function
Posted by 邵志鹏 <bo...@163.com>.
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Updating cube instance 'kylin_sales_cube_clone_spark'
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Remove segment kylin_sales_cube_clone_spark[20120101000000_20140201235500]
Action - Delete Segement can Remove segment , and then rebuild it.
Main problem is
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function。
不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。
spark版本：2.3.2
spark-core_2.11-2.3.2
在 2019-08-14 14:08:03，"邵志鹏" <bo...@163.com> 写道：
Hi kylin team:
kylin-2.6.3
使用MapReduce已经成功构建smaple cube。
然后clone了一份kylin_sales_cube修改为Spark，build 报错。
Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
$KYLIN_HOME/bin/download-spark.sh
jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
hadoop fs -mkdir -p /kylin/spark/
hadoop fs -put spark-libs.jar /kylin/spark/
...
报错日志：
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91]
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91]
at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3]
at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
报错之后，再去提交build，又报新的错：
Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of them is READY. It's not allowed for parallel building
at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3]
... 77 more
Monitor里面是没有新的构建任务的。
Thanks.
By 邵志鹏