You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by deric <ba...@gmail.com> on 2014/01/13 18:19:24 UTC
Re: Running Spark on Mesos
I've updated to the newest trunk version, and still all tasks are getting
lost:
java.lang.NullPointerException
at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
at
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Running Spark on Mesos
Posted by deric <ba...@gmail.com>.
I've deleted whole /tmp/mesos on each slave, but it didn't help (this one
was running on mesos 0.15.0). I've tried different mesos versions (0.14,
0.15, 0.16-rc1, 0.16-rc2). Now spark is compiled with mesos-0.15.0.jar, but
it doesn't seem to have any impact on this.
java.lang.NullPointerException
at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"
stdout:
Running spark-executor with framework dir = /usr/share/spark
14/01/14 09:54:46 ERROR MesosExecutorBackend: Received launchTask but
executor was null
On 14 January 2014 03:23, Benjamin Hindman [via Apache Spark User List] <
ml-node+s1001560n528h16@n3.nabble.com> wrote:
> You should be able to use 0.16.0-rc2, but I recommend using 0.16.0-rc3
> since it fixes a bug with the webui (but not related to this).
>
> Did you try restarting your slaves after deleting the meta-directory? Kill
> the slave, delete the meta-directory (by default at /tmp/mesos/meta unless
> you passed --work_dir to the slave) and then restart the slave. If things
> don't work after that please let us know!
>
> Ben.
>
>
> On Mon, Jan 13, 2014 at 4:06 PM, deric <[hidden email]<http://user/SendEmail.jtp?type=node&node=528&i=0>
> > wrote:
>
>> Hi Ben,
>>
>> is it possible that I've checked out that buggy version from 0.16.0-rc2
>> branch? Before that I was running on 0.16.0~c0a3fcf (some version from
>> November). Which Mesos version would you recommend for running Spark?
>> Spark's pom.xml says 0.13.0, which is quite old.
>>
>> Thanks,
>> Tomas
>>
>>
>> On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <[hidden
>> email] <http://user/SendEmail.jtp?type=node&node=520&i=0>> wrote:
>>
>>> What version of Mesos are you using?
>>>
>>> We tagged a release-candidate of Mesos that had a bug when used with
>>> frameworks that were using older JARs (like Spark). The manifestation of
>>> the bug was some protocol buffers didn't parse, such as SlaveInfo,
>>> resulting in a NullPointerException.
>>>
>>> Until Spark gets a new JAR (and assuming you're not using the buggy
>>> release candidate of Mesos) then you can fix this problem by restarting
>>> your slave after removing it's meta-directory.
>>>
>>> Please share the version (in fact, versions of Mesos that you've
>>> upgraded over time would be great). And let us know how it goes!
>>>
>>> Ben.
>>>
>>>
>>>
>>>
>>> On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]<http://user/SendEmail.jtp?type=node&node=516&i=0>
>>> > wrote:
>>>
>>>> I've updated to the newest trunk version, and still all tasks are
>>>> getting
>>>> lost:
>>>>
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
>>>> at
>>>>
>>>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
>>>> at
>>>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
>>>> at
>>>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
>>>> at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
>>>> at
>>>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
>>>> at
>>>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
>>>> at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
>>>> at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
>>>> at
>>>>
>>>> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
>>>> Exception in thread "Thread-0"
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
>>> To unsubscribe from Running Spark on Mesos, click here.
>>> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Running Spark on Mesos<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p520.html>
>>
>> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p528.html
> To unsubscribe from Running Spark on Mesos, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=503&code=YmFydG9uLnRvbWFzQGdtYWlsLmNvbXw1MDN8NDI0MTAyNDUz>
> .
> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p534.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Running Spark on Mesos
Posted by Benjamin Hindman <be...@gmail.com>.
You should be able to use 0.16.0-rc2, but I recommend using 0.16.0-rc3
since it fixes a bug with the webui (but not related to this).
Did you try restarting your slaves after deleting the meta-directory? Kill
the slave, delete the meta-directory (by default at /tmp/mesos/meta unless
you passed --work_dir to the slave) and then restart the slave. If things
don't work after that please let us know!
Ben.
On Mon, Jan 13, 2014 at 4:06 PM, deric <ba...@gmail.com> wrote:
> Hi Ben,
>
> is it possible that I've checked out that buggy version from 0.16.0-rc2
> branch? Before that I was running on 0.16.0~c0a3fcf (some version from
> November). Which Mesos version would you recommend for running Spark?
> Spark's pom.xml says 0.13.0, which is quite old.
>
> Thanks,
> Tomas
>
>
> On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=520&i=0>> wrote:
>
>> What version of Mesos are you using?
>>
>> We tagged a release-candidate of Mesos that had a bug when used with
>> frameworks that were using older JARs (like Spark). The manifestation of
>> the bug was some protocol buffers didn't parse, such as SlaveInfo,
>> resulting in a NullPointerException.
>>
>> Until Spark gets a new JAR (and assuming you're not using the buggy
>> release candidate of Mesos) then you can fix this problem by restarting
>> your slave after removing it's meta-directory.
>>
>> Please share the version (in fact, versions of Mesos that you've upgraded
>> over time would be great). And let us know how it goes!
>>
>> Ben.
>>
>>
>>
>>
>> On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]<http://user/SendEmail.jtp?type=node&node=516&i=0>
>> > wrote:
>>
>>> I've updated to the newest trunk version, and still all tasks are getting
>>> lost:
>>>
>>> java.lang.NullPointerException
>>> at
>>>
>>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
>>> at
>>>
>>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
>>> at
>>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
>>> at
>>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
>>> at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
>>> at
>>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
>>> at
>>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
>>> at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
>>> at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
>>> at
>>>
>>> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
>>> Exception in thread "Thread-0"
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
>> To unsubscribe from Running Spark on Mesos, click here.
>> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Running Spark on Mesos<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p520.html>
>
> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>
Re: Running Spark on Mesos
Posted by deric <ba...@gmail.com>.
Hi Ben,
is it possible that I've checked out that buggy version from 0.16.0-rc2
branch? Before that I was running on 0.16.0~c0a3fcf (some version from
November). Which Mesos version would you recommend for running Spark?
Spark's pom.xml says 0.13.0, which is quite old.
Thanks,
Tomas
On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <
ml-node+s1001560n516h44@n3.nabble.com> wrote:
> What version of Mesos are you using?
>
> We tagged a release-candidate of Mesos that had a bug when used with
> frameworks that were using older JARs (like Spark). The manifestation of
> the bug was some protocol buffers didn't parse, such as SlaveInfo,
> resulting in a NullPointerException.
>
> Until Spark gets a new JAR (and assuming you're not using the buggy
> release candidate of Mesos) then you can fix this problem by restarting
> your slave after removing it's meta-directory.
>
> Please share the version (in fact, versions of Mesos that you've upgraded
> over time would be great). And let us know how it goes!
>
> Ben.
>
>
>
>
> On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]<http://user/SendEmail.jtp?type=node&node=516&i=0>
> > wrote:
>
>> I've updated to the newest trunk version, and still all tasks are getting
>> lost:
>>
>> java.lang.NullPointerException
>> at
>>
>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
>> at
>>
>> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
>> at
>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
>> at
>> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
>> at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
>> at
>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
>> at
>> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
>> at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
>> at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
>> at
>>
>> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
>> Exception in thread "Thread-0"
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
> To unsubscribe from Running Spark on Mesos, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=503&code=YmFydG9uLnRvbWFzQGdtYWlsLmNvbXw1MDN8NDI0MTAyNDUz>
> .
> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p520.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Running Spark on Mesos
Posted by Benjamin Hindman <be...@gmail.com>.
What version of Mesos are you using?
We tagged a release-candidate of Mesos that had a bug when used with
frameworks that were using older JARs (like Spark). The manifestation of
the bug was some protocol buffers didn't parse, such as SlaveInfo,
resulting in a NullPointerException.
Until Spark gets a new JAR (and assuming you're not using the buggy release
candidate of Mesos) then you can fix this problem by restarting your slave
after removing it's meta-directory.
Please share the version (in fact, versions of Mesos that you've upgraded
over time would be great). And let us know how it goes!
Ben.
On Mon, Jan 13, 2014 at 10:19 AM, deric <ba...@gmail.com> wrote:
> I've updated to the newest trunk version, and still all tasks are getting
> lost:
>
> java.lang.NullPointerException
> at
>
> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
> at
>
> com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
> at
> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
> at
> com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
> at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
> at
> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
> at
> com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
> at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
> at
>
> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
> Exception in thread "Thread-0"
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>