You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by "Miller, Clifford" <cl...@phoenix-opsgroup.com> on 2018/05/24 23:16:58 UTC
Spark2 with YARN
I've setup a cluster using Hortonworks HDP with Ambari all running in AWS.
I then created a separate EC2 instance and installed PIO 0.12.1, hadoop,
elasticsearch, hbase, and spark2. I copied the configurations from the HDP
cluster and then pio-start-all. The pio-start-all completes successfully
and running "pio status" also shows success. I'm following the "Text
Classification Engine Tutorial". I've imported the data. I'm using the
following command to train: "pio train -- --master yarn". After running
the command I get the following exception. Does anyone have any ideas of
what I may have missed during my setup?
Thanks in advance.
#################
Exception follows:
Exception in thread "main" java.lang.NoClassDefFoundError:
com/sun/jersey/api/client/config/ClientConfig
at
org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:45)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at
org.apache.predictionio.workflow.WorkflowContext$.apply(WorkflowContext.scala:45)
at
org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:59)
at
org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
at
org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.sun.jersey.api.client.config.ClientConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
##############
Re: Spark2 with YARN
Posted by "Miller, Clifford" <cl...@phoenix-opsgroup.com>.
That stopped the one error from occurring.
Thanks
On Fri, May 25, 2018 at 12:20 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Just disable the timeline service of yarn: set yarn.timeline-service.enabled
> to false in yarn-site.xml and restart hadoop
>
>
> Pat Ferrel <pa...@occamsmachete.com>于2018年5月25日周五 上午7:56写道：
>
>> I’m having a java.lang.NoClassDefFoundError in a different context and
>> different class. Have you tried this without Yarn? Sorry I can’t find the
>> rest of this thread.
>>
>>
>> From: Miller, Clifford <cl...@phoenix-opsgroup.com>
>> <cl...@phoenix-opsgroup.com>
>> Reply: user@predictionio.apache.org <us...@predictionio.apache.org>
>> <us...@predictionio.apache.org>
>> Date: May 24, 2018 at 4:16:58 PM
>> To: user@predictionio.apache.org <us...@predictionio.apache.org>
>> <us...@predictionio.apache.org>
>> Subject: Spark2 with YARN
>>
>> I've setup a cluster using Hortonworks HDP with Ambari all running in
>> AWS. I then created a separate EC2 instance and installed PIO 0.12.1,
>> hadoop, elasticsearch, hbase, and spark2. I copied the configurations from
>> the HDP cluster and then pio-start-all. The pio-start-all completes
>> successfully and running "pio status" also shows success. I'm following
>> the "Text Classification Engine Tutorial". I've imported the data. I'm
>> using the following command to train: "pio train -- --master yarn". After
>> running the command I get the following exception. Does anyone have any
>> ideas of what I may have missed during my setup?
>>
>> Thanks in advance.
>>
>> #################
>> Exception follows:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> com/sun/jersey/api/client/config/ClientConfig
>> at org.apache.hadoop.yarn.client.api.TimelineClient.
>> createTimelineClient(TimelineClient.java:45)
>> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.
>> serviceInit(YarnClientImpl.java:163)
>> at org.apache.hadoop.service.AbstractService.init(
>> AbstractService.java:163)
>> at org.apache.spark.deploy.yarn.Client.submitApplication(
>> Client.scala:152)
>> at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.
>> start(YarnClientSchedulerBackend.scala:56)
>> at org.apache.spark.scheduler.TaskSchedulerImpl.start(
>> TaskSchedulerImpl.scala:156)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
>> at org.apache.predictionio.workflow.WorkflowContext$.
>> apply(WorkflowContext.scala:45)
>> at org.apache.predictionio.workflow.CoreWorkflow$.
>> runTrain(CoreWorkflow.scala:59)
>> at org.apache.predictionio.workflow.CreateWorkflow$.main(
>> CreateWorkflow.scala:251)
>> at org.apache.predictionio.workflow.CreateWorkflow.main(
>> CreateWorkflow.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
>> deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
>> SparkSubmit.scala:187)
>> at org.apache.spark.deploy.SparkSubmit$.submit(
>> SparkSubmit.scala:212)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
>> scala:126)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.
>> config.ClientConfig
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 20 more
>>
>> ##############
>>
>>
--
Clifford Miller
Mobile | 321.431.9089
Re: Spark2 with YARN
Posted by Jeff Zhang <zj...@gmail.com>.
Just disable the timeline service of yarn:
set yarn.timeline-service.enabled to false in yarn-site.xml and restart
hadoop
Pat Ferrel <pa...@occamsmachete.com>于2018年5月25日周五 上午7:56写道：
> I’m having a java.lang.NoClassDefFoundError in a different context and
> different class. Have you tried this without Yarn? Sorry I can’t find the
> rest of this thread.
>
>
> From: Miller, Clifford <cl...@phoenix-opsgroup.com>
> <cl...@phoenix-opsgroup.com>
> Reply: user@predictionio.apache.org <us...@predictionio.apache.org>
> <us...@predictionio.apache.org>
> Date: May 24, 2018 at 4:16:58 PM
> To: user@predictionio.apache.org <us...@predictionio.apache.org>
> <us...@predictionio.apache.org>
> Subject: Spark2 with YARN
>
> I've setup a cluster using Hortonworks HDP with Ambari all running in
> AWS. I then created a separate EC2 instance and installed PIO 0.12.1,
> hadoop, elasticsearch, hbase, and spark2. I copied the configurations from
> the HDP cluster and then pio-start-all. The pio-start-all completes
> successfully and running "pio status" also shows success. I'm following
> the "Text Classification Engine Tutorial". I've imported the data. I'm
> using the following command to train: "pio train -- --master yarn". After
> running the command I get the following exception. Does anyone have any
> ideas of what I may have missed during my setup?
>
> Thanks in advance.
>
> #################
> Exception follows:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jersey/api/client/config/ClientConfig
> at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:45)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
> at
> org.apache.predictionio.workflow.WorkflowContext$.apply(WorkflowContext.scala:45)
> at
> org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:59)
> at
> org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
> at
> org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> com.sun.jersey.api.client.config.ClientConfig
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 20 more
>
> ##############
>
>
Re: Spark2 with YARN
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I’m having a java.lang.NoClassDefFoundError in a different context and
different class. Have you tried this without Yarn? Sorry I can’t find the
rest of this thread.
From: Miller, Clifford <cl...@phoenix-opsgroup.com>
<cl...@phoenix-opsgroup.com>
Reply: user@predictionio.apache.org <us...@predictionio.apache.org>
<us...@predictionio.apache.org>
Date: May 24, 2018 at 4:16:58 PM
To: user@predictionio.apache.org <us...@predictionio.apache.org>
<us...@predictionio.apache.org>
Subject: Spark2 with YARN
I've setup a cluster using Hortonworks HDP with Ambari all running in AWS.
I then created a separate EC2 instance and installed PIO 0.12.1, hadoop,
elasticsearch, hbase, and spark2. I copied the configurations from the HDP
cluster and then pio-start-all. The pio-start-all completes successfully
and running "pio status" also shows success. I'm following the "Text
Classification Engine Tutorial". I've imported the data. I'm using the
following command to train: "pio train -- --master yarn". After running
the command I get the following exception. Does anyone have any ideas of
what I may have missed during my setup?
Thanks in advance.
#################
Exception follows:
Exception in thread "main" java.lang.NoClassDefFoundError:
com/sun/jersey/api/client/config/ClientConfig
at
org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:45)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at
org.apache.predictionio.workflow.WorkflowContext$.apply(WorkflowContext.scala:45)
at
org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:59)
at
org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
at
org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.sun.jersey.api.client.config.ClientConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
##############