You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by innowireless TaeYun Kim <ta...@innowireless.co.kr> on 2014/07/02 10:58:14 UTC
Help: WARN AbstractNioSelector: Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space
Hi,
When running a Spark job, the following warning message displays and the job
seems no longer progressing.
(Detailed log message are at the bottom of this message.)
---------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
---------------
The specifics of the job is as follows:
- It reads 168016 files on the HDFS, by calling
sc.textFile("hdfs://cluster01/user/data/*/*/*.csv")
- The total size of the files is 164,111,123,686 bytes. (164GB)
- The max size of the files is 8,546,230 bytes. (8.5MB)
- The min size of the files is 3,920 bytes. (3KB)
- The command line options are: --master yarn-client --executor-memory 14G
--executor-cores 7 --num-executors 3
The WARN occurred when processing reduceByKey().
On Spark Stage web page, the number of tasks for reduceByKey() was 168016.
(same as the number of the files)
The WARN occurred when reduceByKey() was progressing about 10%. (not exact)
Actual method call sequence is:
sc.textFile(...).filter().mapToPair().reduceByKey().map().saveAsTextFile().
How can I fix this?
Detailed log messages are as follows: (IP and hostname was replaced.)
----------------------------------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:16 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(<driver>, DriverHostName, 63548, 0) with no recent heart
beats: 63641ms exceeds 45000ms
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:14 ERROR Utils: Uncaught exception in thread Result resolver
thread-2
14/07/02 17:00:37 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Result resolver thread-2" java.lang.OutOfMemoryError:
Java heap space
14/07/02 17:00:19 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(3, cluster03, 45202, 0) with no recent heart beats: 65296ms
exceeds 45000ms
14/07/02 17:00:41 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:00:43 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(1, cluster04, 55341, 0) with no recent heart beats: 65328ms
exceeds 45000ms
14/07/02 17:00:47 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(2, cluster02, 54328, 0) with no recent heart beats: 69631ms
exceeds 45000ms
Exception in thread "SparkListenerBus" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Connection manager future execution context-1" 14/07/02
17:00:55 WARN AbstractNioSelector: Unexpected exception in the selector
loop.
14/07/02 17:01:00 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:01:48 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:02:28 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:02:29 WARN ServletHandler: Error for /stages/
14/07/02 17:02:28 WARN AbstractNioSelector: Failed to accept a connection.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp1359288682-100" java.lang.OutOfMemoryError: Java
heap space
Exception in thread "qtp1359288682-63" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1359288682-57" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1492586839-40" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1359288682-106" java.lang.OutOfMemoryError: Java
heap space
14/07/02 17:02:55 WARN AbstractHttpConnection: /stages/
14/07/02 17:03:16 WARN nio: handle failed
14/07/02 17:03:33 WARN nio: handle failed
14/07/02 17:07:58 WARN nio:
14/07/02 17:10:43 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:18:34 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:19:32 WARN ServletHandler: Error for /stages/
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:20:05 WARN AbstractHttpConnection: /stages/
java.lang.OutOfMemoryError: Java heap space
Thanks in advance.
RE: Help: WARN AbstractNioSelector: Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space
Posted by innowireless TaeYun Kim <ta...@innowireless.co.kr>.
It seems that the driver program gets out of memory.
In Windows Task Manager, the driver program's memory constantly grows until
around 3,434,796, then java OutOfMemory exception occurs.
(BTW, the driver program runs on Windows 7 64bit machine, and cluster is on
CentOS.)
Why the memory of the driver program constantly grows?
-----Original Message-----
From: innowireless TaeYun Kim [mailto:taeyun.kim@innowireless.co.kr]
Sent: Wednesday, July 02, 2014 6:05 PM
To: user@spark.apache.org
Subject: RE: Help: WARN AbstractNioSelector: Unexpected exception in the
selector loop. java.lang.OutOfMemoryError: Java heap space
Also, the machine on which the driver program runs constantly uses about
7~8% of 100Mbps network connection.
Is the driver program involved in the reduceByKey() somehow?
BTW, currently an accumulator is used, but the network usage does not drop
even when accumulator is removed.
Thanks in advance.
-----Original Message-----
From: innowireless TaeYun Kim [mailto:taeyun.kim@innowireless.co.kr]
Sent: Wednesday, July 02, 2014 5:58 PM
To: user@spark.apache.org
Subject: Help: WARN AbstractNioSelector: Unexpected exception in the
selector loop. java.lang.OutOfMemoryError: Java heap space
Hi,
When running a Spark job, the following warning message displays and the job
seems no longer progressing.
(Detailed log message are at the bottom of this message.)
---------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
---------------
The specifics of the job is as follows:
- It reads 168016 files on the HDFS, by calling
sc.textFile("hdfs://cluster01/user/data/*/*/*.csv")
- The total size of the files is 164,111,123,686 bytes. (164GB)
- The max size of the files is 8,546,230 bytes. (8.5MB)
- The min size of the files is 3,920 bytes. (3KB)
- The command line options are: --master yarn-client --executor-memory 14G
--executor-cores 7 --num-executors 3
The WARN occurred when processing reduceByKey().
On Spark Stage web page, the number of tasks for reduceByKey() was 168016.
(same as the number of the files)
The WARN occurred when reduceByKey() was progressing about 10%. (not exact)
Actual method call sequence is:
sc.textFile(...).filter().mapToPair().reduceByKey().map().saveAsTextFile().
How can I fix this?
Detailed log messages are as follows: (IP and hostname was replaced.)
----------------------------------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:16 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(<driver>, DriverHostName, 63548, 0) with no recent heart
beats: 63641ms exceeds 45000ms
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:14 ERROR Utils: Uncaught exception in thread Result resolver
thread-2
14/07/02 17:00:37 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Result resolver thread-2" java.lang.OutOfMemoryError:
Java heap space
14/07/02 17:00:19 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(3, cluster03, 45202, 0) with no recent heart beats: 65296ms
exceeds 45000ms
14/07/02 17:00:41 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:00:43 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(1, cluster04, 55341, 0) with no recent heart beats: 65328ms
exceeds 45000ms
14/07/02 17:00:47 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(2, cluster02, 54328, 0) with no recent heart beats: 69631ms
exceeds 45000ms Exception in thread "SparkListenerBus"
java.lang.OutOfMemoryError: Java heap space Exception in thread "Connection
manager future execution context-1" 14/07/02
17:00:55 WARN AbstractNioSelector: Unexpected exception in the selector
loop.
14/07/02 17:01:00 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:01:48 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:02:28 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:02:29 WARN ServletHandler: Error for /stages/
14/07/02 17:02:28 WARN AbstractNioSelector: Failed to accept a connection.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space Exception in thread
"qtp1359288682-100" java.lang.OutOfMemoryError: Java heap space Exception in
thread "qtp1359288682-63" java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp1359288682-57" java.lang.OutOfMemoryError: Java heap
space Exception in thread "qtp1492586839-40" java.lang.OutOfMemoryError:
Java heap space Exception in thread "qtp1359288682-106"
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:02:55 WARN AbstractHttpConnection: /stages/
14/07/02 17:03:16 WARN nio: handle failed
14/07/02 17:03:33 WARN nio: handle failed
14/07/02 17:07:58 WARN nio:
14/07/02 17:10:43 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:18:34 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:19:32 WARN ServletHandler: Error for /stages/
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:20:05 WARN AbstractHttpConnection: /stages/
java.lang.OutOfMemoryError: Java heap space
Thanks in advance.
RE: Help: WARN AbstractNioSelector: Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space
Posted by innowireless TaeYun Kim <ta...@innowireless.co.kr>.
Also, the machine on which the driver program runs constantly uses about
7~8% of 100Mbps network connection.
Is the driver program involved in the reduceByKey() somehow?
BTW, currently an accumulator is used, but the network usage does not drop
even when accumulator is removed.
Thanks in advance.
-----Original Message-----
From: innowireless TaeYun Kim [mailto:taeyun.kim@innowireless.co.kr]
Sent: Wednesday, July 02, 2014 5:58 PM
To: user@spark.apache.org
Subject: Help: WARN AbstractNioSelector: Unexpected exception in the
selector loop. java.lang.OutOfMemoryError: Java heap space
Hi,
When running a Spark job, the following warning message displays and the job
seems no longer progressing.
(Detailed log message are at the bottom of this message.)
---------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
---------------
The specifics of the job is as follows:
- It reads 168016 files on the HDFS, by calling
sc.textFile("hdfs://cluster01/user/data/*/*/*.csv")
- The total size of the files is 164,111,123,686 bytes. (164GB)
- The max size of the files is 8,546,230 bytes. (8.5MB)
- The min size of the files is 3,920 bytes. (3KB)
- The command line options are: --master yarn-client --executor-memory 14G
--executor-cores 7 --num-executors 3
The WARN occurred when processing reduceByKey().
On Spark Stage web page, the number of tasks for reduceByKey() was 168016.
(same as the number of the files)
The WARN occurred when reduceByKey() was progressing about 10%. (not exact)
Actual method call sequence is:
sc.textFile(...).filter().mapToPair().reduceByKey().map().saveAsTextFile().
How can I fix this?
Detailed log messages are as follows: (IP and hostname was replaced.)
----------------------------------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:16 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(<driver>, DriverHostName, 63548, 0) with no recent heart
beats: 63641ms exceeds 45000ms
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:14 ERROR Utils: Uncaught exception in thread Result resolver
thread-2
14/07/02 17:00:37 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Result resolver thread-2" java.lang.OutOfMemoryError:
Java heap space
14/07/02 17:00:19 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(3, cluster03, 45202, 0) with no recent heart beats: 65296ms
exceeds 45000ms
14/07/02 17:00:41 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:00:43 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(1, cluster04, 55341, 0) with no recent heart beats: 65328ms
exceeds 45000ms
14/07/02 17:00:47 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(2, cluster02, 54328, 0) with no recent heart beats: 69631ms
exceeds 45000ms Exception in thread "SparkListenerBus"
java.lang.OutOfMemoryError: Java heap space Exception in thread "Connection
manager future execution context-1" 14/07/02
17:00:55 WARN AbstractNioSelector: Unexpected exception in the selector
loop.
14/07/02 17:01:00 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:01:48 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:02:28 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:02:29 WARN ServletHandler: Error for /stages/
14/07/02 17:02:28 WARN AbstractNioSelector: Failed to accept a connection.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space Exception in thread
"qtp1359288682-100" java.lang.OutOfMemoryError: Java heap space Exception in
thread "qtp1359288682-63" java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp1359288682-57" java.lang.OutOfMemoryError: Java heap
space Exception in thread "qtp1492586839-40" java.lang.OutOfMemoryError:
Java heap space Exception in thread "qtp1359288682-106"
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:02:55 WARN AbstractHttpConnection: /stages/
14/07/02 17:03:16 WARN nio: handle failed
14/07/02 17:03:33 WARN nio: handle failed
14/07/02 17:07:58 WARN nio:
14/07/02 17:10:43 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:18:34 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:19:32 WARN ServletHandler: Error for /stages/
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:20:05 WARN AbstractHttpConnection: /stages/
java.lang.OutOfMemoryError: Java heap space
Thanks in advance.