You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by naresh gundla <na...@gmail.com> on 2017/02/15 21:58:00 UTC
Need Help: getting java.lang.OutOfMemory Error : GC overhead limit
exceeded (TransportChannelHandler)
Hi ,
I am running a spark application and getting out of memory errors in yarn
nodemanager logs and container get killed. Please find below for the errors
details.
Has anyone faced with this issue?
*Enabled spark dynamic allocation and yarn shuffle*
2017-02-15 14:50:48,047 WARN io.netty.util.concurrent.DefaultPromise: An
exception was thrown by
org.apache.spark.network.server.TransportRequestHandler$2.operationComplete()
java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-02-15 15:21:09,506 ERROR
org.apache.spark.network.server.TransportRequestHandler: Error opening
block StreamChunkId{streamId=1374579274227, chunkIndex=241} for request
from /10.154.16.83:50042
java.lang.IllegalStateException: Received out-of-order chunk index 241
(expected 114)
at
org.apache.spark.network.server.OneForOneStreamManager.getChunk(OneForOneStreamManager.java:81)
at
org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:121)
at
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:100)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)
2017-02-15 14:50:14,692 WARN
org.apache.spark.network.server.TransportChannelHandler: Exception in
connection from /10.154.16.74:58547
java.lang.OutOfMemoryError: GC overhead limit exceeded
Thanks
Naresh
Re: Need Help: getting java.lang.OutOfMemory Error : GC overhead
limit exceeded (TransportChannelHandler)
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Naresh,
We've configured our Spark JVMs to shut down if there is an
OutOfMemoryError. Otherwise, the error will bring down a random thread an
cause trouble like the IllegalStateException you hit. It is best to let
Spark recover by replacing the executor or failing the job.
rb
On Wed, Feb 15, 2017 at 1:58 PM, naresh gundla <na...@gmail.com>
wrote:
> Hi ,
>
> I am running a spark application and getting out of memory errors in yarn
> nodemanager logs and container get killed. Please find below for the errors
> details.
> Has anyone faced with this issue?
>
> *Enabled spark dynamic allocation and yarn shuffle*
>
> 2017-02-15 14:50:48,047 WARN io.netty.util.concurrent.DefaultPromise: An
> exception was thrown by org.apache.spark.network.server.
> TransportRequestHandler$2.operationComplete()
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2017-02-15 15:21:09,506 ERROR org.apache.spark.network.server.TransportRequestHandler:
> Error opening block StreamChunkId{streamId=1374579274227, chunkIndex=241}
> for request from /10.154.16.83:50042
> java.lang.IllegalStateException: Received out-of-order chunk index 241
> (expected 114)
> at org.apache.spark.network.server.OneForOneStreamManager.
> getChunk(OneForOneStreamManager.java:81)
> at org.apache.spark.network.server.TransportRequestHandler.
> processFetchRequest(TransportRequestHandler.java:121)
> at org.apache.spark.network.server.TransportRequestHandler.handle(
> TransportRequestHandler.java:100)
> at org.apache.spark.network.server.TransportChannelHandler.
> channelRead0(TransportChannelHandler.java:104)
> at org.apache.spark.network.server.TransportChannelHandler.
> channelRead0(TransportChannelHandler.java:51)
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(
> SimpleChannelInboundHandler.java:105)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:319)
> at io.netty.handler.timeout.IdleStateHandler.channelRead(
> IdleStateHandler.java:254)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:319)
> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(
> MessageToMessageDecoder.java:103)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:319)
> at org.apache.spark.network.util.TransportFrameDecoder.
> channelRead(TransportFrameDecoder.java:86)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:319)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(
> DefaultChannelPipeline.java:787)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(
> AbstractNioByteChannel.java:130)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
> NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> 2017-02-15 14:50:14,692 WARN org.apache.spark.network.server.TransportChannelHandler:
> Exception in connection from /10.154.16.74:58547
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Thanks
> Naresh
>
>
--
Ryan Blue
Software Engineer
Netflix