You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nelson <ne...@ysance.com> on 2014/05/26 11:48:10 UTC
maprfs and spark libraries
Hi all,
I am trying to run spark over a MapR cluster. I successfully ran several
custom applications on a previous non-mapr hadoop cluster but i can't get
them working on the mapr one. To be more specific, i am not able to read or
write on mfs without running into a serialization error from Java. Note that
everything works fine when i am running the app in local mode which make me
think of a dependancy error.
The test application is built using sbt with the following dependency:
- org.apache.spark spark-core 0.9.1
In my test_app/lib directory i have:
- hadoop-core-1.0.3-mapr-3.0.2.jar
- json-20140107.jar
- maprfs-1.0.3-mapr-3.0.2.jar
Finally, i add those jars with conf.setJars so that they are distributed on
the cluster.
Am I compiling with the wrong dependencies? Should i get a "mapr version" of
spark-core?
Regards, Nelson
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: maprfs and spark libraries
Posted by Surendranauth Hiraman <su...@velos.io>.
When I have stack traces, I usually see the MapR versions of the various
hadoop classes, though maybe that's at a deeper level of the stack trace.
If my memory is right though, this may point to the classpath having
regular hadoop jars before the standard hadoop jars. My guess is that this
is on the driver side, if so.
I had to make sure to put the MapR jars as the very first entries on my
driver classpath.
-Suren
On Mon, May 26, 2014 at 12:11 PM, nelson <ne...@ysance.com> wrote:
> I am using standard readers and writers i believe.
> When i locally run the app, spark is able to write on hdfs. Then i assume
> accessing and reading mfs is doable.
>
> Here is the piece of code i use for testing:
> /val list = List ("dad", "mum", "brother" , "sister")
> val mlist = sc.parallelize(list)
> mlist.saveAsTextFile("maprfs:///user/nelson/test")/
>
> and the stack trace:
> /14/05/26 16:02:54 WARN scheduler.TaskSetManager: Loss was due to
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.hadoop.fs.FileSystem.fixName(FileSystem.java:187)
> at
> org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
> at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:617)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:439)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:412)
> at
> org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
> at
> org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
> at
>
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
> at
>
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
> at scala.Option.map(Option.scala:145)
> at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:111)
> at
> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:154)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at
> org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
> at
>
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)/
>
> The uri name seems to be the issue now as i happen to get rid of the
> serialization issue.
>
> Regards,
> Nelson
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6402.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: maprfs and spark libraries
Posted by nelson <ne...@ysance.com>.
I am using standard readers and writers i believe.
When i locally run the app, spark is able to write on hdfs. Then i assume
accessing and reading mfs is doable.
Here is the piece of code i use for testing:
/val list = List ("dad", "mum", "brother" , "sister")
val mlist = sc.parallelize(list)
mlist.saveAsTextFile("maprfs:///user/nelson/test")/
and the stack trace:
/14/05/26 16:02:54 WARN scheduler.TaskSetManager: Loss was due to
java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.fixName(FileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:617)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:439)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:412)
at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:111)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:154)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)/
The uri name seems to be the issue now as i happen to get rid of the
serialization issue.
Regards,
Nelson
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6402.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: maprfs and spark libraries
Posted by Surendranauth Hiraman <su...@velos.io>.
We use the mapr rpm and have successfully read and written hdfs data.
Are you using custom readers/writers? Maybe the relevant stacktrace might
help.
Maybe also try a standard text reader and writer to see if there is a basic
issue with accessing mfs?
-Suren
On Mon, May 26, 2014 at 11:31 AM, nelson <ne...@ysance.com> wrote:
> thanks for replying guys.
>
> Mayur:
> Indeed i tried the local mode (sparkmaster: local[5]) before and the
> application runs well, no serialization problem. The problem arises as soon
> as i try to run the app over the cluster.
>
> Surendranauth:
> I just double checked my spark_classpath from spark_env.sh and we have a
> similar one.
> Is your team running spark with the mapr rpm [1]? Or did you compile it
> yourself?
>
> [1]: http://doc.mapr.com/display/MapR/Installing+Spark+and+Shark
> <http://doc.mapr.com/display/MapR/Installing+Spark+and+Shark>
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6400.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: maprfs and spark libraries
Posted by nelson <ne...@ysance.com>.
thanks for replying guys.
Mayur:
Indeed i tried the local mode (sparkmaster: local[5]) before and the
application runs well, no serialization problem. The problem arises as soon
as i try to run the app over the cluster.
Surendranauth:
I just double checked my spark_classpath from spark_env.sh and we have a
similar one.
Is your team running spark with the mapr rpm [1]? Or did you compile it
yourself?
[1]: http://doc.mapr.com/display/MapR/Installing+Spark+and+Shark
<http://doc.mapr.com/display/MapR/Installing+Spark+and+Shark>
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6400.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: maprfs and spark libraries
Posted by Surendranauth Hiraman <su...@velos.io>.
My team is successfully running on Spark on MapR.
However, we add the mapr jars to the SPARK_CLASSAPTH on the workers, as
well as making sure they are on the classpath of the driver.
I'm not sure if we need every jar that we currently add but below is what
we currently use. The important file in the conf directory
is mapr-clusters.conf.
I do not think we add the jars to the classpath via addJars and prefer to
reference them from the mapr installation. This is partly because we
colocate our workers on our mapr worker nodes.
export
SPARK_CLASSPATH=/opt/mapr/hadoop/hadoop-0.20.2/conf:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/lib/maprfs-1.0.3-mapr-3.0.2.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-1.0.3-mapr-3.0.2.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.3.6.jar
On Mon, May 26, 2014 at 10:50 AM, Mayur Rustagi <ma...@gmail.com>wrote:
> Did you try in standalone mode. You may not see serialization issues in
> local threaded mode.
> Serialization errors are unlikely to be cause of Mapr hadoop version.
> Regards
> Mayur
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Mon, May 26, 2014 at 3:18 PM, nelson <ne...@ysance.com> wrote:
>
>> Hi all,
>>
>> I am trying to run spark over a MapR cluster. I successfully ran several
>> custom applications on a previous non-mapr hadoop cluster but i can't get
>> them working on the mapr one. To be more specific, i am not able to read
>> or
>> write on mfs without running into a serialization error from Java. Note
>> that
>> everything works fine when i am running the app in local mode which make
>> me
>> think of a dependancy error.
>>
>> The test application is built using sbt with the following dependency:
>> - org.apache.spark spark-core 0.9.1
>>
>> In my test_app/lib directory i have:
>> - hadoop-core-1.0.3-mapr-3.0.2.jar
>> - json-20140107.jar
>> - maprfs-1.0.3-mapr-3.0.2.jar
>>
>> Finally, i add those jars with conf.setJars so that they are distributed
>> on
>> the cluster.
>>
>> Am I compiling with the wrong dependencies? Should i get a "mapr version"
>> of
>> spark-core?
>>
>> Regards, Nelson
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: maprfs and spark libraries
Posted by Mayur Rustagi <ma...@gmail.com>.
Did you try in standalone mode. You may not see serialization issues in
local threaded mode.
Serialization errors are unlikely to be cause of Mapr hadoop version.
Regards
Mayur
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Mon, May 26, 2014 at 3:18 PM, nelson <ne...@ysance.com> wrote:
> Hi all,
>
> I am trying to run spark over a MapR cluster. I successfully ran several
> custom applications on a previous non-mapr hadoop cluster but i can't get
> them working on the mapr one. To be more specific, i am not able to read or
> write on mfs without running into a serialization error from Java. Note
> that
> everything works fine when i am running the app in local mode which make me
> think of a dependancy error.
>
> The test application is built using sbt with the following dependency:
> - org.apache.spark spark-core 0.9.1
>
> In my test_app/lib directory i have:
> - hadoop-core-1.0.3-mapr-3.0.2.jar
> - json-20140107.jar
> - maprfs-1.0.3-mapr-3.0.2.jar
>
> Finally, i add those jars with conf.setJars so that they are distributed on
> the cluster.
>
> Am I compiling with the wrong dependencies? Should i get a "mapr version"
> of
> spark-core?
>
> Regards, Nelson
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: maprfs and spark libraries
Posted by nelson <ne...@ysance.com>.
As simple as that. Indeed, the spark jar i was linking to wasn't the mapr
version. I just added spark-assembly-0.9.1-hadoop1.0.3-mapr-3.0.3.jar to the
lib directory of my project as a unmanaged dependency for sbt.
Thank you Cafe au Lait and to all of you guys.
Regards,
Nelson.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6414.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: maprfs and spark libraries
Posted by "Cafe au Lait (icloud)" <ca...@me.com>.
hi
On May 26, 2014, at 06:48 PM, nelson <ne...@ysance.com> wrote:
The test application is built using sbt with the following dependency:
- org.apache.spark spark-core 0.9.1
you need to remove this dependency. sbt will pack non-mapr version
of hadoop classes.