You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Wu, Mandy" <ma...@intel.com> on 2012/09/10 10:45:38 UTC

Hadoop long initializing issue

Hi,
We built up a Hadoop cluster as one master node and 4 several slave nodes, using Hadoop 0.20.2 + Cygwin on Windows 7. We submit job with Hadoop streaming. The mapper is a console application written by C++, and no reducer. When streaming job is executed, sometime there is one task will be in "initializing" state for quite long time (even one hour). After we increase slave node to 6, this issue happened more frequently.

We checked jobtracker, tasktracker logs, there is no error/warning recorded.

On taskmgr, we found the java process hang. Below is the dump info. Any suggestion or solution?

Attaching to process ID 12832, please wait...
Client compiler detected.
JVM version is 21.0-b17
Deadlock Detection:

No deadlocks found.

Finding object size using Printezis bits and skipping over...
Thread 14: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 13: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.ping(org.apache.hadoop.mapred.TaskAttemptID) @bci=16 (Interpreted frame)
- org.apache.hadoop.mapred.Task$TaskReporter.run() @bci=168, line=548 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)

Locked ownable synchronizers:
    - None

Thread 11: (state = IN_NATIVE)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(long, int, int[], int[], int[], long) @bci=0 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @bci=43, line=295 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(sun.nio.ch.WindowsSelectorImpl$SubSelector) @bci=1, line=277 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci=63, line=158 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel, int, long) @bci=46, line=332 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=80, line=157 (Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=155 (Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=128 (Interpreted frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=276 (Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=235 (Interpreted frame)
- java.io.BufferedInputStream.read() @bci=12, line=254 (Interpreted frame)
- java.io.DataInputStream.readInt() @bci=4, line=387 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19, line=501 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=446 (Interpreted frame)

Locked ownable synchronizers:
    - None

Thread 10: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 6: (state = BLOCKED)

Locked ownable synchronizers:
    - None

Thread 5: (state = BLOCKED)

Locked ownable synchronizers:
    - None

Thread 4: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 3: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 1: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.commitPending(org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.mapred.TaskStatus) @bci=20 (Interpreted frame)
- org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=80, line=678 (Interpreted frame)
- org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=116, line=309 (Interpreted frame)
- org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=455, line=170 (Interpreted frame)

Locked ownable synchronizers:
    - None