You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Steven Cox <sc...@renci.org> on 2014/04/23 02:05:53 UTC
Spark 0.9.1 core dump on Mesos
So I tried a fix found on the list... changing the mesos version from 0.13.0 to 0.18.0. This apparently worked for someone else on the list in their configuration.
In mine, it still core dumps like this:
https://gist.github.com/stevencox/11002498
In this environment:
Ubuntu 13.10
Mesos 0.18.0
Spark 0.9.1
JDK 1.7.0_45
Scala 2.10.1
I've tried multiple JDK implementations/versions. All core dump.
I've switched back to Mesos 0.13.0 per the Spark on mesos docs. Still core dumps.
What am I missing?
RE: Spark 0.9.1 core dump on Mesos
Posted by Steven Cox <sc...@renci.org>.
For posterity, the recommendations below work. After modding mesos and protobuf verisions in the pom, my build:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
export SCALA_HOME=/opt/scala-2.10.1/
mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package
works. Thanks folks.
Steve
________________________________
From: Nathan Howell [nhowell@godaddy.com]
Sent: Wednesday, April 23, 2014 12:43 PM
To: Steven Cox
Cc: user@mesos.apache.org
Subject: Re: Spark 0.9.1 core dump on Mesos
Also see: https://issues.apache.org/jira/browse/MESOS-1056
-n
From: Tim St Clair <ts...@redhat.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Tuesday, April 22, 2014 at 7:43 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Spark 0.9.1 core dump on Mesos
I've run spark 0.9.1 on 0.18.0.
I did need to patch Spark in order for it to work.
diff --git a/pom.xml b/pom.xml
index e82d9e8..b1bc684 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,13 +109,13 @@
<scala.version>2.10.3</scala.version>
<scala.binary.version>2.10</scala.binary.version>
- <mesos.version>0.13.0</mesos.version>
+ <mesos.version>0.18.0-rc6</mesos.version>
<akka.group>org.spark-project.akka</akka.group>
<akka.version>2.2.3-shaded-protobuf</akka.version>
<slf4j.version>1.7.2</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version>
- <protobuf.version>2.4.1</protobuf.version>
+ <protobuf.version>2.5.0</protobuf.version>
<yarn.version>0.23.7</yarn.version>
<hbase.version>0.94.6</hbase.version>
This has been my standard pattern, but I also found wonkiness in their scheduler that I patched.
Cheers,
Tim
________________________________
From: "Steven Cox" <sc...@renci.org>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Tuesday, April 22, 2014 7:05:53 PM
Subject: Spark 0.9.1 core dump on Mesos
So I tried a fix found on the list... changing the mesos version from 0.13.0 to 0.18.0. This apparently worked for someone else on the list in their configuration.
In mine, it still core dumps like this:
https://gist.github.com/stevencox/11002498
In this environment:
Ubuntu 13.10
Mesos 0.18.0
Spark 0.9.1
JDK 1.7.0_45
Scala 2.10.1
I've tried multiple JDK implementations/versions. All core dump.
I've switched back to Mesos 0.13.0 per the Spark on mesos docs. Still core dumps.
What am I missing?
--
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata
Re: Spark 0.9.1 core dump on Mesos
Posted by Nathan Howell <nh...@godaddy.com>.
Also see: https://issues.apache.org/jira/browse/MESOS-1056
-n
From: Tim St Clair <ts...@redhat.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Tuesday, April 22, 2014 at 7:43 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Spark 0.9.1 core dump on Mesos
I've run spark 0.9.1 on 0.18.0.
I did need to patch Spark in order for it to work.
diff --git a/pom.xml b/pom.xml
index e82d9e8..b1bc684 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,13 +109,13 @@
<scala.version>2.10.3</scala.version>
<scala.binary.version>2.10</scala.binary.version>
- <mesos.version>0.13.0</mesos.version>
+ <mesos.version>0.18.0-rc6</mesos.version>
<akka.group>org.spark-project.akka</akka.group>
<akka.version>2.2.3-shaded-protobuf</akka.version>
<slf4j.version>1.7.2</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version>
- <protobuf.version>2.4.1</protobuf.version>
+ <protobuf.version>2.5.0</protobuf.version>
<yarn.version>0.23.7</yarn.version>
<hbase.version>0.94.6</hbase.version>
This has been my standard pattern, but I also found wonkiness in their scheduler that I patched.
Cheers,
Tim
________________________________
From: "Steven Cox" <sc...@renci.org>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Tuesday, April 22, 2014 7:05:53 PM
Subject: Spark 0.9.1 core dump on Mesos
So I tried a fix found on the list... changing the mesos version from 0.13.0 to 0.18.0. This apparently worked for someone else on the list in their configuration.
In mine, it still core dumps like this:
https://gist.github.com/stevencox/11002498
In this environment:
Ubuntu 13.10
Mesos 0.18.0
Spark 0.9.1
JDK 1.7.0_45
Scala 2.10.1
I've tried multiple JDK implementations/versions. All core dump.
I've switched back to Mesos 0.13.0 per the Spark on mesos docs. Still core dumps.
What am I missing?
--
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata
Re: Spark 0.9.1 core dump on Mesos
Posted by Tim St Clair <ts...@redhat.com>.
I've run spark 0.9.1 on 0.18.0.
I did need to patch Spark in order for it to work.
diff --git a/pom.xml b/pom.xml
index e82d9e8..b1bc684 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,13 +109,13 @@
<scala.version>2.10.3</scala.version>
<scala.binary.version>2.10</scala.binary.version>
- <mesos.version>0.13.0</mesos.version>
+ <mesos.version>0.18.0-rc6</mesos.version>
<akka.group>org.spark-project.akka</akka.group>
<akka.version>2.2.3-shaded-protobuf</akka.version>
<slf4j.version>1.7.2</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version>
- <protobuf.version>2.4.1</protobuf.version>
+ <protobuf.version>2.5.0</protobuf.version>
<yarn.version>0.23.7</yarn.version>
<hbase.version>0.94.6</hbase.version>
This has been my standard pattern, but I also found wonkiness in their scheduler that I patched.
Cheers,
Tim
----- Original Message -----
> From: "Steven Cox" <sc...@renci.org>
> To: user@mesos.apache.org
> Sent: Tuesday, April 22, 2014 7:05:53 PM
> Subject: Spark 0.9.1 core dump on Mesos
> So I tried a fix found on the list... changing the mesos version from 0.13.0
> to 0.18.0. This apparently worked for someone else on the list in their
> configuration.
> In mine, it still core dumps like this:
> https://gist.github.com/stevencox/11002498
> In this environment:
> Ubuntu 13.10
> Mesos 0.18.0
> Spark 0.9.1
> JDK 1.7.0_45
> Scala 2.10.1
> I've tried multiple JDK implementations/versions. All core dump.
> I've switched back to Mesos 0.13.0 per the Spark on mesos docs. Still core
> dumps.
> What am I missing?
--
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata