You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Wu, Mandy" <ma...@intel.com> on 2012/09/10 10:45:38 UTC
Hadoop long initializing issue
Hi,
We built up a Hadoop cluster as one master node and 4 several slave nodes, using Hadoop 0.20.2 + Cygwin on Windows 7. We submit job with Hadoop streaming. The mapper is a console application written by C++, and no reducer. When streaming job is executed, sometime there is one task will be in "initializing" state for quite long time (even one hour). After we increase slave node to 6, this issue happened more frequently.
We checked jobtracker, tasktracker logs, there is no error/warning recorded.
On taskmgr, we found the java process hang. Below is the dump info. Any suggestion or solution?
Attaching to process ID 12832, please wait...
Client compiler detected.
JVM version is 21.0-b17
Deadlock Detection:
No deadlocks found.
Finding object size using Printezis bits and skipping over...
Thread 14: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 13: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.ping(org.apache.hadoop.mapred.TaskAttemptID) @bci=16 (Interpreted frame)
- org.apache.hadoop.mapred.Task$TaskReporter.run() @bci=168, line=548 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 11: (state = IN_NATIVE)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(long, int, int[], int[], int[], long) @bci=0 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @bci=43, line=295 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(sun.nio.ch.WindowsSelectorImpl$SubSelector) @bci=1, line=277 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci=63, line=158 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel, int, long) @bci=46, line=332 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=80, line=157 (Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=155 (Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=128 (Interpreted frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=276 (Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=235 (Interpreted frame)
- java.io.BufferedInputStream.read() @bci=12, line=254 (Interpreted frame)
- java.io.DataInputStream.readInt() @bci=4, line=387 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19, line=501 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=446 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 10: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 6: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 5: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 4: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 3: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 1: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.commitPending(org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.mapred.TaskStatus) @bci=20 (Interpreted frame)
- org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=80, line=678 (Interpreted frame)
- org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=116, line=309 (Interpreted frame)
- org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=455, line=170 (Interpreted frame)
Locked ownable synchronizers:
- None