You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by hubert stefani <hu...@yahoo.fr> on 2019/01/28 14:24:58 UTC
Error when activating spark interactive on v2.6
Hi, It seems that when we activate the property hereunder , the spark build job fails. Have you noticed such behaviour ? Should I open an issue on Jira ?
Context :
Kylin : v2.6
AWS : EMR 5.18
Property :
kylin.engine.spark-conf.spark.hadoop.yarn.timeline-service.enabled
Error Log :
2019-01-28 10:50:36,384 INFO [pool-13-thread-1] spark.SparkExecutable:38 : 2019-01-28 10:50:36 WARN SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
2019-01-28 10:50:37,732 INFO [pool-13-thread-1] spark.SparkExecutable:38 : Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : ... 12 more
2019-01-28 10:50:37,737 INFO [pool-13-thread-1] spark.SparkExecutable:38 : 2019-01-28 10:50:37 INFO ShutdownHookManager:54 - Shutdown hook called
Hubert
答复: Error when activating spark interactive on v2.6
Posted by Na Zhai <na...@kyligence.io>.
Hi, hubert stefani.
It seems the conflict of the jar, maybe you can try to put the jersey-client jar in the yarn service into the jars directory of the Spark.
发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用
________________________________
发件人: hubert stefani <hu...@yahoo.fr>
发送时间: Monday, January 28, 2019 10:24:58 PM
收件人: user@kylin.apache.org
主题: Error when activating spark interactive on v2.6
Hi,
It seems that when we activate the property hereunder , the spark build job fails. Have you noticed such behaviour ? Should I open an issue on Jira ?
Context :
Kylin : v2.6
AWS : EMR 5.18
Property :
kylin.engine.spark-conf.spark.hadoop.yarn.timeline-service.enabled
Error Log :
2019-01-28 10:50:36,384 INFO [pool-13-thread-1] spark.SparkExecutable:38 : 2019-01-28 10:50:36 WARN SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
2019-01-28 10:50:37,732 INFO [pool-13-thread-1] spark.SparkExecutable:38 : Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38 : ... 12 more
2019-01-28 10:50:37,737 INFO [pool-13-thread-1] spark.SparkExecutable:38 : 2019-01-28 10:50:37 INFO ShutdownHookManager:54 - Shutdown hook called
Hubert
Re: Error when activating spark interactive on v2.6
Posted by Liukaige <et...@gmail.com>.
It's a known issue in EMR. Not only EMR, you will meet this error in HDP as
well. To enable yarn timeline service, you need to resolve these package
dependency conflict.
hubert stefani <hu...@yahoo.fr> 于2019年1月28日周一 下午10:34写道：
>
> Hi,
> It seems that when we activate the property hereunder , the spark
> build job fails. Have you noticed such behaviour ? Should I open an issue
> on Jira ?
>
> *Context : *
> Kylin : v2.6
> AWS : EMR 5.18
>
> *Property : *
> kylin.engine.spark-conf.spark.hadoop.yarn.timeline-service.enabled
>
> *Error Log : *
> 2019-01-28 10:50:36,384 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : 2019-01-28 10:50:36 WARN SparkConf:66 - The configuration key
> 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3
> and may be removed in the future. Please use the new key
> 'spark.executor.memoryOverhead' instead.
> 2019-01-28 10:50:37,732 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jersey/api/client/config/ClientConfig
> 2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
> 2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
> 2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
> 2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> 2019-01-28 10:50:37,733 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
> 2019-01-28 10:50:37,734 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : Caused by: java.lang.ClassNotFoundException:
> com.sun.jersey.api.client.config.ClientConfig
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 2019-01-28 10:50:37,735 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : ... 12 more
> 2019-01-28 10:50:37,737 INFO [pool-13-thread-1] spark.SparkExecutable:38
> : 2019-01-28 10:50:37 INFO ShutdownHookManager:54 - Shutdown hook called
>
> Hubert
>
--
Best regards,
Kaige Liu(刘凯歌)
*"Do small things with great love." *