You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by patcharee <Pa...@uni.no> on 2015/03/18 11:30:56 UTC
Spark Job History Server
Hi,
I am using spark 1.3. I would like to use Spark Job History Server. I
added the following line into conf/spark-defaults.conf
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.history.provider
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.yarn.historyServer.address sandbox.hortonworks.com:19888
But got Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
What class is really needed? How to fix it?
Br,
Patcharee
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Job History Server
Posted by Zhan Zhang <zz...@hortonworks.com>.
Hi Patcharee,
It is an alpha feature in HDP distribution, integrating ATS with Spark history server. If you are using upstream, you can configure spark as regular without these configuration. But other related configuration are still mandatory, such as hdp.version related.
Thanks.
Zhan Zhang
On Mar 18, 2015, at 3:30 AM, patcharee <Pa...@uni.no> wrote:
> Hi,
>
> I am using spark 1.3. I would like to use Spark Job History Server. I added the following line into conf/spark-defaults.conf
>
> spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
> spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
>
> But got Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>
> What class is really needed? How to fix it?
>
> Br,
> Patcharee
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Job History Server
Posted by patcharee <Pa...@uni.no>.
Hi,
My spark was compiled with yarn profile, I can run spark on yarn without
problem.
For the spark job history server problem, I checked
spark-assembly-1.3.0-hadoop2.4.0.jar and found that the package
org.apache.spark.deploy.yarn.history is missing. I don't know why....
BR,
Patcharee
On 18. mars 2015 11:43, Akhil Das wrote:
> You are not having yarn package in the classpath. You need to build
> your spark it with yarn. You can read these docs.
> <http://spark.apache.org/docs/1.3.0/running-on-yarn.html>
>
> Thanks
> Best Regards
>
> On Wed, Mar 18, 2015 at 4:07 PM, patcharee <Patcharee.Thongtra@uni.no
> <ma...@uni.no>> wrote:
>
> I turned it on. But it failed to start. In the log,
>
> Spark assembly has been built with Hive, including Datanucleus
> jars on classpath
> Spark Command: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp
> :/root/spark-1.3.0-bin-hadoop2.4/sbin/../conf:/root/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/etc/hadoop/conf
> -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m
> -Xmx512m org.apache.spark.deploy.history.HistoryServer
> ========================================
>
> 15/03/18 10:23:46 WARN NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:183)
> at
> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
>
> Patcharee
>
>
> On 18. mars 2015 11:35, Akhil Das wrote:
>> You can simply turn it on using:
>> |./sbin/start-history-server.sh|
>>
>> Read more here
>> <http://spark.apache.org/docs/1.3.0/monitoring.html>.
>>
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Mar 18, 2015 at 4:00 PM, patcharee
>> <Patcharee.Thongtra@uni.no <ma...@uni.no>> wrote:
>>
>> Hi,
>>
>> I am using spark 1.3. I would like to use Spark Job History
>> Server. I added the following line into conf/spark-defaults.conf
>>
>> spark.yarn.services
>> org.apache.spark.deploy.yarn.history.YarnHistoryService
>> spark.history.provider
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>> spark.yarn.historyServer.address
>> sandbox.hortonworks.com:19888
>> <http://sandbox.hortonworks.com:19888>
>>
>> But got Exception in thread "main"
>> java.lang.ClassNotFoundException:
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>>
>> What class is really needed? How to fix it?
>>
>> Br,
>> Patcharee
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> <ma...@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org
>> <ma...@spark.apache.org>
>>
>>
>
>
Re: Spark Job History Server
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You are not having yarn package in the classpath. You need to build your
spark it with yarn. You can read these docs.
<http://spark.apache.org/docs/1.3.0/running-on-yarn.html>
Thanks
Best Regards
On Wed, Mar 18, 2015 at 4:07 PM, patcharee <Pa...@uni.no>
wrote:
> I turned it on. But it failed to start. In the log,
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> Spark Command: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp
> :/root/spark-1.3.0-bin-hadoop2.4/sbin/../conf:/root/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/etc/hadoop/conf
> -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
> org.apache.spark.deploy.history.HistoryServer
> ========================================
>
> 15/03/18 10:23:46 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:183)
> at
> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
>
> Patcharee
>
>
> On 18. mars 2015 11:35, Akhil Das wrote:
>
> You can simply turn it on using:
>
> ./sbin/start-history-server.sh
>
>
> Read more here <http://spark.apache.org/docs/1.3.0/monitoring.html>.
>
>
> Thanks
> Best Regards
>
> On Wed, Mar 18, 2015 at 4:00 PM, patcharee <Pa...@uni.no>
> wrote:
>
>> Hi,
>>
>> I am using spark 1.3. I would like to use Spark Job History Server. I
>> added the following line into conf/spark-defaults.conf
>>
>> spark.yarn.services
>> org.apache.spark.deploy.yarn.history.YarnHistoryService
>> spark.history.provider
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
>>
>> But got Exception in thread "main" java.lang.ClassNotFoundException:
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>>
>> What class is really needed? How to fix it?
>>
>> Br,
>> Patcharee
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
Re: Spark Job History Server
Posted by patcharee <Pa...@uni.no>.
I turned it on. But it failed to start. In the log,
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Spark Command: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp
:/root/spark-1.3.0-bin-hadoop2.4/sbin/../conf:/root/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/etc/hadoop/conf
-XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m
-Xmx512m org.apache.spark.deploy.history.HistoryServer
========================================
15/03/18 10:23:46 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at
org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:183)
at
org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Patcharee
On 18. mars 2015 11:35, Akhil Das wrote:
> You can simply turn it on using:
> |./sbin/start-history-server.sh|
>
> Read more here <http://spark.apache.org/docs/1.3.0/monitoring.html>.
>
>
> Thanks
> Best Regards
>
> On Wed, Mar 18, 2015 at 4:00 PM, patcharee <Patcharee.Thongtra@uni.no
> <ma...@uni.no>> wrote:
>
> Hi,
>
> I am using spark 1.3. I would like to use Spark Job History
> Server. I added the following line into conf/spark-defaults.conf
>
> spark.yarn.services
> org.apache.spark.deploy.yarn.history.YarnHistoryService
> spark.history.provider
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
> <http://sandbox.hortonworks.com:19888>
>
> But got Exception in thread "main"
> java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>
> What class is really needed? How to fix it?
>
> Br,
> Patcharee
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
Re: Spark Job History Server
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can simply turn it on using:
./sbin/start-history-server.sh
Read more here <http://spark.apache.org/docs/1.3.0/monitoring.html>.
Thanks
Best Regards
On Wed, Mar 18, 2015 at 4:00 PM, patcharee <Pa...@uni.no>
wrote:
> Hi,
>
> I am using spark 1.3. I would like to use Spark Job History Server. I
> added the following line into conf/spark-defaults.conf
>
> spark.yarn.services org.apache.spark.deploy.yarn.
> history.YarnHistoryService
> spark.history.provider org.apache.spark.deploy.yarn.
> history.YarnHistoryProvider
> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
>
> But got Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>
> What class is really needed? How to fix it?
>
> Br,
> Patcharee
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark Job History Server
Posted by Sean Owen <so...@cloudera.com>.
Uh, does that mean HDP shipped Marcelo's uncommitted patch from
SPARK-1537 anyway? Given the discussion there, that seems kinda
aggressive.
On Wed, Mar 18, 2015 at 8:49 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Those classes are not part of standard Spark. You may want to contact
> Hortonworks directly if they're suggesting you use those.
>
> On Wed, Mar 18, 2015 at 3:30 AM, patcharee <Pa...@uni.no> wrote:
>> Hi,
>>
>> I am using spark 1.3. I would like to use Spark Job History Server. I added
>> the following line into conf/spark-defaults.conf
>>
>> spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
>> spark.history.provider
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
>>
>> But got Exception in thread "main" java.lang.ClassNotFoundException:
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>>
>> What class is really needed? How to fix it?
>>
>> Br,
>> Patcharee
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Job History Server
Posted by Marcelo Vanzin <va...@cloudera.com>.
Those classes are not part of standard Spark. You may want to contact
Hortonworks directly if they're suggesting you use those.
On Wed, Mar 18, 2015 at 3:30 AM, patcharee <Pa...@uni.no> wrote:
> Hi,
>
> I am using spark 1.3. I would like to use Spark Job History Server. I added
> the following line into conf/spark-defaults.conf
>
> spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
> spark.history.provider
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> spark.yarn.historyServer.address sandbox.hortonworks.com:19888
>
> But got Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>
> What class is really needed? How to fix it?
>
> Br,
> Patcharee
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org