You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Viswanathan J <ja...@gmail.com> on 2013/12/14 18:13:28 UTC
Hadoop Jobtracker job and UI hangs - Deadlock detection
Hi,
JT memory reaches 6.68/8.89 GB and not able to submit the job and UI is not
loading at all. But didn't see any JT OOM exceptions.
Have taken the thread dump of Jobtracker, and the JT thread dump as follows:
Deadlock Detection:
Can't print deadlocks:null
Thread 25817: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may
be imprecise)
- org.apache.hadoop.hdfs.LeaseRenewer.run(int) @bci=274, line=397
(Compiled frame)
- org.apache.hadoop.hdfs.LeaseRenewer.access$600(org.apache.hadoop.hdfs.LeaseRenewer,
int) @bci=2, line=69 (Interpreted frame)
- org.apache.hadoop.hdfs.LeaseRenewer$1.run() @bci=8, line=273
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 25815: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run()
@bci=245, line=3000 (Compiled frame)
Locked ownable synchronizers:
- None
Thread 25813: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
line=747 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
(Compiled frame)
Locked ownable synchronizers:
- None
Thread 25812: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
line=747 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
(Compiled frame)
Locked ownable synchronizers:
- None
Thread 25790: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
Thread 25788: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
line=747 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
(Compiled frame)
Locked ownable synchronizers:
- None
Thread 25786: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
line=747 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
(Compiled frame)
Locked ownable synchronizers:
- None
Thread 25761: (state = BLOCKED)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
(Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Interpreted
frame)
- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel,
int, long) @bci=46, line=332 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer,
int) @bci=80, line=157 (Compiled frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer)
@bci=6, line=155 (Compiled frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7,
line=128 (Compiled frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=116
(Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[],
int, int) @bci=4, line=364 (Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=218 (Compiled frame)
- java.io.BufferedInputStream.read() @bci=12, line=237 (Compiled frame)
- java.io.DataInputStream.readInt() @bci=4, line=370 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19,
line=845 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=790
(Compiled frame)
-------------------------------------------------------------------------------------------------------------------------------
And Jobtracker heap summary as follows:
using thread-local object allocation.
Parallel GC with 10 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 10737418240 (10240.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 6488064 (6.1875MB)
used = 6488064 (6.1875MB)
free = 0 (0.0MB)
100.0% used
>From Space:
capacity = 9764864 (9.3125MB)
used = 0 (0.0MB)
free = 9764864 (9.3125MB)
0.0% used
To Space:
capacity = 9764864 (9.3125MB)
used = 0 (0.0MB)
free = 9764864 (9.3125MB)
0.0% used
PS Old Generation
capacity = 7158300672 (6826.6875MB)
used = 7158240200 (6826.629829406738MB)
free = 60472 (0.05767059326171875MB)
99.99915521849708% used
PS Perm Generation
capacity = 26738688 (25.5MB)
used = 26428648 (25.204322814941406MB)
free = 310040 (0.29567718505859375MB)
98.8404816272212% used
Please help. It affects our production system.
--
Regards,
Viswa.J