You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Marek Wiewiorka <ma...@gmail.com> on 2014/06/03 18:17:05 UTC
Spark 1.0.0 fails if mesos.coarse set to true
Hi All,
I'm trying to run my code that used to work with mesos-0.14 and spark-0.9.0
with mesos-0.18.2 and spark-1.0.0. and I'm getting a weird error when I use
coarse mode (see below).
If I use the fine-grained mode everything is ok.
Has anybody of you experienced a similar error?
more stderr
-------------------
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0603 16:07:53.721132 61192 exec.cpp:131] Version: 0.18.2
I0603 16:07:53.725230 61200 exec.cpp:205] Executor registered on slave
201405220917-134217738-5050-27119-0
Exception in thread "main" java.lang.NumberFormatException: For input
string: "sparkseq003.cloudapp.net"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.parseInt(Integer.java:527)
at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:135)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
more stdout
-----------------------
Registered executor on sparkseq003.cloudapp.net
Starting task 5
Forked command at 61202
sh -c '"/home/mesos/spark-1.0.0/bin/spark-class"
org.apache.spark.executor.CoarseGrainedExecutorBackend
-Dspark.mesos.coarse=true akka.tcp://
spark@sparkseq001.cloudapp.net:40312/user/CoarseG
rainedScheduler 201405220917-134217738-5050-27119-0 sparkseq003.cloudapp.net
4'
Command exited with status 1 (pid: 61202)
Many thanks,
Marek
Re: Spark 1.0.0 fails if mesos.coarse set to true
Posted by Patrick Wendell <pw...@gmail.com>.
Hey, thanks a lot for reporting this. Do you mind making a JIRA with
the details so we can track it?
- Patrick
On Wed, Jun 4, 2014 at 9:24 AM, Marek Wiewiorka
<ma...@gmail.com> wrote:
> Exactly the same story - it used to work with 0.9.1 and does not work
> anymore with 1.0.0.
> I ran tests using spark-shell as well as my application(so tested turning on
> coarse mode via env variable and SparkContext properties explicitly)
>
> M.
>
>
> 2014-06-04 18:12 GMT+02:00 ajatix <aj...@sigmoidanalytics.com>:
>
>> I'm running a manually built cluster on EC2. I have mesos (0.18.2) and
>> hdfs
>> (2.0.0-cdh4.5.0) installed on all slaves (3) and masters (3). I have
>> spark-1.0.0 on one master and the executor file is on hdfs for the slaves.
>> Whenever I try to launch a spark application on the cluster, it starts a
>> task on each slave (i'm using default configs) and they start FAILING with
>> the error msg - 'Is spark installed on it?'
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-fails-if-mesos-coarse-set-to-true-tp6817p6945.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>
Re: Spark 1.0.0 fails if mesos.coarse set to true
Posted by Marek Wiewiorka <ma...@gmail.com>.
Exactly the same story - it used to work with 0.9.1 and does not work
anymore with 1.0.0.
I ran tests using spark-shell as well as my application(so tested turning
on coarse mode via env variable and SparkContext properties explicitly)
M.
2014-06-04 18:12 GMT+02:00 ajatix <aj...@sigmoidanalytics.com>:
> I'm running a manually built cluster on EC2. I have mesos (0.18.2) and hdfs
> (2.0.0-cdh4.5.0) installed on all slaves (3) and masters (3). I have
> spark-1.0.0 on one master and the executor file is on hdfs for the slaves.
> Whenever I try to launch a spark application on the cluster, it starts a
> task on each slave (i'm using default configs) and they start FAILING with
> the error msg - 'Is spark installed on it?'
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-fails-if-mesos-coarse-set-to-true-tp6817p6945.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Spark 1.0.0 fails if mesos.coarse set to true
Posted by ajatix <aj...@sigmoidanalytics.com>.
I'm running a manually built cluster on EC2. I have mesos (0.18.2) and hdfs
(2.0.0-cdh4.5.0) installed on all slaves (3) and masters (3). I have
spark-1.0.0 on one master and the executor file is on hdfs for the slaves.
Whenever I try to launch a spark application on the cluster, it starts a
task on each slave (i'm using default configs) and they start FAILING with
the error msg - 'Is spark installed on it?'
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-fails-if-mesos-coarse-set-to-true-tp6817p6945.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark 1.0.0 fails if mesos.coarse set to true
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Are you using spark-submit to run your application?
On Wed, Jun 4, 2014 at 8:49 AM, ajatix <aj...@sigmoidanalytics.com> wrote:
> I am also getting the exact error, with the exact logs when I run Spark
> 1.0.0
> in coarse-grained mode.
> Coarse grained mode works perfectly with earlier versions that I tested -
> 0.9.1 and 0.9.0, but seems to have undergone some modification in spark
> 1.0.0
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-fails-if-mesos-coarse-set-to-true-tp6817p6942.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Spark 1.0.0 fails if mesos.coarse set to true
Posted by ajatix <aj...@sigmoidanalytics.com>.
I am also getting the exact error, with the exact logs when I run Spark 1.0.0
in coarse-grained mode.
Coarse grained mode works perfectly with earlier versions that I tested -
0.9.1 and 0.9.0, but seems to have undergone some modification in spark
1.0.0
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-fails-if-mesos-coarse-set-to-true-tp6817p6942.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.