You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Manoj Samel <ma...@gmail.com> on 2014/01/23 20:46:21 UTC
Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC
overhead limit exceeded
I have a standalone master cluster with 512Mb for master and 12 g for each
worker. I start spark-shell with 10g and spark.task.maxFailures=99999
I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK,
then the tasks start failing and after 36K failed tasks, I get following.
Is the master running out of memory ?
Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead
limit exceeded
Jan 23, 2014 6:33:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught error from thread [spark-11] shutting down JVM since
'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jan 23, 2014 6:36:43 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jan 23, 2014 6:36:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Re: Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead limit exceeded
Posted by Kal El <pi...@yahoo.com>.
That will not work, I am currently in the same dilemma. You need to pass some parameters to the JVMs running on the slaves like Xms and Xmx. I managed to do that while running the workload locally but I could not do this on the cluster.
Changing SPARK_DAEMON_MEMORY will have no effect, the memory used will be 512 M
On Thursday, January 23, 2014 11:23 PM, Manoj Samel <ma...@gmail.com> wrote:
I could increase the memory for master.
However, my understanding of the Master is that it just does the DAG scheduling for workers and does not do any RDD processing itself. If this is true and since only one application was running; does the master needs > 512 Mb just to execute the DAG for 3105 tasks ?
On Thu, Jan 23, 2014 at 11:52 AM, Guillaume Pitel <gu...@exensa.com> wrote:
Why not trying to increase the master memory ?
>
>export SPARK_DAEMON_MEMORY=1g in master spark-env.sh
>
>Guillaume
>
>I have a standalone master cluster with 512Mb for master and 12 g for each worker. I start spark-shell with 10g and spark.task.maxFailures=99999
>>
>>I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK, then the tasks start failing and after 36K failed tasks, I get following. Is the master running out of memory ?
>>
>>
>>Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Jan 23, 2014 6:33:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Uncaught error from thread [spark-11] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Jan 23, 2014 6:36:43 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Jan 23, 2014 6:36:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>--
>
> Guillaume PITEL, Président
>+33(0)6 25 48 86 80 / +33(0)9 70 44 67 53
>
>eXenSa S.A.S.
>41, rue Périer - 92120 Montrouge - FRANCE
>Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
Re: Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Posted by Manoj Samel <ma...@gmail.com>.
I could increase the memory for master.
However, my understanding of the Master is that it just does the DAG
scheduling for workers and does not do any RDD processing itself. If this
is true and since only one application was running; does the master needs >
512 Mb just to execute the DAG for 3105 tasks ?
On Thu, Jan 23, 2014 at 11:52 AM, Guillaume Pitel <
guillaume.pitel@exensa.com> wrote:
> Why not trying to increase the master memory ?
>
> export SPARK_DAEMON_MEMORY=1g in master spark-env.sh
>
> Guillaume
>
> I have a standalone master cluster with 512Mb for master and 12 g for
> each worker. I start spark-shell with 10g and spark.task.maxFailures=99999
>
> I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK,
> then the tasks start failing and after 36K failed tasks, I get following.
> Is the master running out of memory ?
>
> Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC
> overhead limit exceeded
> Jan 23, 2014 6:33:36 PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker
> WARNING: Unexpected exception in the selector loop.
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Uncaught error from thread [spark-11] shutting down JVM since
> 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Jan 23, 2014 6:36:43 PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker
> WARNING: Unexpected exception in the selector loop.
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> Jan 23, 2014 6:36:36 PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker
> WARNING: Unexpected exception in the selector loop.
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
>
>
>
>
> --
> [image: eXenSa]
> *Guillaume PITEL, Président*
> +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53
>
> eXenSa S.A.S. <http://www.exensa.com/>
> 41, rue Périer - 92120 Montrouge - FRANCE
> Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
>
Re: Exception in thread "DAGScheduler" java.lang.OutOfMemoryError:
GC overhead limit exceeded
Posted by Guillaume Pitel <gu...@exensa.com>.
Why not trying to increase the master memory ?
export SPARK_DAEMON_MEMORY=1g in master spark-env.sh
Guillaume
> I have a standalone master cluster with 512Mb for master and 12 g for each
> worker. I start spark-shell with 10g and spark.task.maxFailures=99999
>
> I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK, then
> the tasks start failing and after 36K failed tasks, I get following. Is the
> master running out of memory ?
>
> |Exception in thread ||"DAGScheduler"| |java.lang.OutOfMemoryError: GC
> overhead limit exceeded|
> |Jan ||23||, ||2014| |6||:||33||:||36| |PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker|
> |WARNING: Unexpected exception in the selector loop.|
> |java.lang.OutOfMemoryError: GC overhead limit exceeded|
> |Uncaught error from thread [spark-||11||] shutting down JVM
> since ||'akka.jvm-exit-on-fatal-error'| |is enabled ||for| |ActorSystem[spark]|
> |java.lang.OutOfMemoryError: GC overhead limit exceeded|
> |Jan ||23||, ||2014| |6||:||36||:||43| |PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker|
> |WARNING: Unexpected exception in the selector loop.|
> |java.lang.OutOfMemoryError: GC overhead limit exceeded|
> |Jan ||23||, ||2014| |6||:||36||:||36| |PM
> org.jboss.netty.channel.socket.nio.AbstractNioWorker|
> |WARNING: Unexpected exception in the selector loop.|
> |java.lang.OutOfMemoryError: GC overhead limit exceeded|
>
> |
> |
> |
> |
> |
> |
--
eXenSa
*Guillaume PITEL, Président*
+33(0)6 25 48 86 80 / +33(0)9 70 44 67 53
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05