You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Johan Oskarson (JIRA)" <ji...@apache.org> on 2006/11/10 15:32:37 UTC
[jira] Created: (HADOOP-707) Final map task gets stuck
Final map task gets stuck
-------------------------
Key: HADOOP-707
URL: http://issues.apache.org/jira/browse/HADOOP-707
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.8.0
Environment: using latest trunk
Reporter: Johan Oskarson
Priority: Critical
I've seen numerous jobs lately where the final map task gets stuck, never finishing.
The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue and the job can finish.
In the web interface it turns up as:
task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12 10-Nov-2006 12:22:19 (1mins, 6sec)
Task failed to report status for 604 seconds. Killing.
Only exception I find in that tasktracker log is this (a few times):
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-707) Final map task gets stuck
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533654 ]
Devaraj Das commented on HADOOP-707:
------------------------------------
This could be related to HADOOP-2016
> Final map task gets stuck
> -------------------------
>
> Key: HADOOP-707
> URL: https://issues.apache.org/jira/browse/HADOOP-707
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.8.0
> Environment: using latest trunk
> Reporter: Johan Oskarsson
> Assignee: Owen O'Malley
> Priority: Critical
>
> I've seen numerous jobs lately where the final map task gets stuck, never finishing.
> The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue and the job can finish.
> In the web interface it turns up as:
> task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12 10-Nov-2006 12:22:19 (1mins, 6sec)
> Task failed to report status for 604 seconds. Killing.
> Only exception I find in that tasktracker log is this (a few times):
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-707) Final map task gets stuck
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-707?page=comments#action_12450855 ]
Owen O'Malley commented on HADOOP-707:
--------------------------------------
Where is the task being lost? Use the web/ui to see if the JobTracker thinks the map is running somewhere. If so, follow the link to the TaskTracker to see if the assigned TaskTracker thinks it is still running. If so, look to see if the the process is still running.
Otherwise, look at the logs to determine where the task the task ran and look at the logs on the task tracker's node to see what the task tracker thinks happened to it.
> Final map task gets stuck
> -------------------------
>
> Key: HADOOP-707
> URL: http://issues.apache.org/jira/browse/HADOOP-707
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.8.0
> Environment: using latest trunk
> Reporter: Johan Oskarson
> Priority: Critical
>
> I've seen numerous jobs lately where the final map task gets stuck, never finishing.
> The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue and the job can finish.
> In the web interface it turns up as:
> task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12 10-Nov-2006 12:22:19 (1mins, 6sec)
> Task failed to report status for 604 seconds. Killing.
> Only exception I find in that tasktracker log is this (a few times):
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-707) Final map task gets stuck
Posted by "Johan Oskarson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-707?page=comments#action_12451007 ]
Johan Oskarson commented on HADOOP-707:
---------------------------------------
The jobtracker ui shows the task as still running. The tasktracker web ui also shows the task as running. Howerver, on that node there is only a datanode and tasktracker running.
In the tasktracker log on that node:
2006-11-18 01:39:53,722 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 06/11/18 01:39:53 WARN mapred.TaskRunner: java.net.SocketTimeoutException: timed out waiting for rpc response
2006-11-18 01:39:53,722 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.ipc.Client.call(Client.java:460)
2006-11-18 01:39:53,722 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:164)
2006-11-18 01:39:53,722 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.$Proxy0.progress(Unknown Source)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.Task.reportProgress(Task.java:173)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.Task.reportProgress(Task.java:162)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.MapTask$3.next(MapTask.java:200)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:215)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0 at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1247)
2006-11-18 01:39:53,723 INFO org.apache.hadoop.mapred.TaskRunner: task_0197_m_000137_0
2006-11-18 01:39:54,166 INFO org.apache.hadoop.mapred.TaskTracker: task_0197_m_000137_0 0.8654954% /user/hadoop/data/submissions/1160000000/1163000000/1163730000/1163730000:30169435+6033887
2006-11-18 01:39:54,166 INFO org.apache.hadoop.mapred.TaskRunner: task_0194_m_001875_0 done; removing files.
2006-11-18 01:39:54,894 INFO org.apache.hadoop.mapred.TaskTracker: task_0197_m_000137_0 1.0% /user/hadoop/data/submissions/1160000000/1163000000/1163730000/1163730000:30169435+6033887
2006-11-18 01:39:54,894 INFO org.apache.hadoop.mapred.TaskRunner: task_0194_m_001888_0 done; removing files.
2006-11-18 01:39:55,268 INFO org.apache.hadoop.mapred.TaskTracker: Task task_0197_m_000137_0 is done.
And a bit further down:
2006-11-18 01:39:56,257 WARN org.apache.hadoop.ipc.Server: handler output error
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
> Final map task gets stuck
> -------------------------
>
> Key: HADOOP-707
> URL: http://issues.apache.org/jira/browse/HADOOP-707
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.8.0
> Environment: using latest trunk
> Reporter: Johan Oskarson
> Priority: Critical
>
> I've seen numerous jobs lately where the final map task gets stuck, never finishing.
> The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue and the job can finish.
> In the web interface it turns up as:
> task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12 10-Nov-2006 12:22:19 (1mins, 6sec)
> Task failed to report status for 604 seconds. Killing.
> Only exception I find in that tasktracker log is this (a few times):
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-707) Final map task gets stuck
Posted by "Johan Oskarson (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-707?page=comments#action_12450802 ]
Johan Oskarson commented on HADOOP-707:
---------------------------------------
Same problem, different job.
Full thread dump.
2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker: task_0133_m_000948_0: Task failed to report status for 608 seconds. Killing.
2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process Thread Dump: lost task
65 active threads
Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
State: WAITING
Blocked count: 1
Waited count: 2
Waiting on org.apache.hadoop.ipc.Client$Connection@74e65f
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
Thread 46826 (btpool0-893):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46825 (btpool0-892):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46822 (btpool0-891):
State: RUNNABLE
Blocked count: 0
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46811 (btpool0-890):
State: RUNNABLE
Blocked count: 3
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46810 (btpool0-889):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46803 (btpool0-888):
State: RUNNABLE
Blocked count: 0
Waited count: 8
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46408 (btpool0-887):
State: RUNNABLE
Blocked count: 0
Waited count: 41
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46402 (btpool0-886):
State: RUNNABLE
Blocked count: 0
Waited count: 26
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46396 (btpool0-885):
State: RUNNABLE
Blocked count: 4
Waited count: 48
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:687)
Thread 46345 (btpool0-883):
State: TIMED_WAITING
Blocked count: 0
Waited count: 26
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 46333 (btpool0-881):
State: RUNNABLE
Blocked count: 0
Waited count: 42
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46305 (btpool0-880):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46304 (btpool0-879):
State: RUNNABLE
Blocked count: 0
Waited count: 37
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46295 (btpool0-878):
State: RUNNABLE
Blocked count: 1
Waited count: 36
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46253 (btpool0-877):
State: RUNNABLE
Blocked count: 1
Waited count: 48
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46252 (btpool0-876):
State: RUNNABLE
Blocked count: 0
Waited count: 46
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46251 (btpool0-875):
State: RUNNABLE
Blocked count: 0
Waited count: 51
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46242 (btpool0-874):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 45039 (Thread-28307):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45038 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45031 (Thread-28302):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45030 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45026 (Thread-28299):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45025 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44998 (Thread-28283):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44997 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44987 (Thread-28277):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44986 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44905 (btpool0-872):
State: TIMED_WAITING
Blocked count: 2
Waited count: 87
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 44903 (btpool0-870):
State: TIMED_WAITING
Blocked count: 1
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 44891 (btpool0-865):
State: RUNNABLE
Blocked count: 5
Waited count: 55
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44888 (btpool0-863):
State: RUNNABLE
Blocked count: 7
Waited count: 135
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44887 (btpool0-862):
State: RUNNABLE
Blocked count: 4
Waited count: 131
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44882 (btpool0-857):
State: RUNNABLE
Blocked count: 9
Waited count: 138
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44881 (btpool0-856):
State: RUNNABLE
Blocked count: 3
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44841 (btpool0-851):
State: RUNNABLE
Blocked count: 6
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44752 (btpool0-849):
State: RUNNABLE
Blocked count: 7
Waited count: 134
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43631 (Thread-27401):
State: RUNNABLE
Blocked count: 114
Waited count: 1730
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43610 (Thread-27384):
State: RUNNABLE
Blocked count: 116
Waited count: 1687
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43591 (Thread-27369):
State: RUNNABLE
Blocked count: 79
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43583 (Thread-27361):
State: RUNNABLE
Blocked count: 109
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43575 (Thread-27353):
State: RUNNABLE
Blocked count: 128
Waited count: 1726
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43052 (btpool0-844):
State: RUNNABLE
Blocked count: 6
Waited count: 217
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43049 (btpool0-842):
State: RUNNABLE
Blocked count: 4
Waited count: 213
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43042 (btpool0-835):
State: RUNNABLE
Blocked count: 12
Waited count: 229
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.readFully(DataInputStream.java:176)
java.io.DataInputStream.readFully(DataInputStream.java:152)
org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(FSDataInputStream.java:60)
org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java:279)
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1409)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
Thread 40895 (btpool0-803):
State: RUNNABLE
Blocked count: 3
Waited count: 361
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 40380 (btpool0-790):
State: RUNNABLE
Blocked count: 7
Waited count: 391
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:687)
Thread 39121 (btpool0-784):
State: RUNNABLE
Blocked count: 16
Waited count: 439
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 39112 (btpool0-775):
State: RUNNABLE
Blocked count: 13
Waited count: 396
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 28 (org.apache.hadoop.dfs.DFSClient$LeaseChecker@1a7ddcf):
State: TIMED_WAITING
Blocked count: 0
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
java.lang.Thread.run(Thread.java:595)
Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
State: TIMED_WAITING
Blocked count: 154
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 18 (IPC Server handler 4 on 50050):
State: TIMED_WAITING
Blocked count: 1769
Waited count: 51161
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 17 (IPC Server handler 3 on 50050):
State: TIMED_WAITING
Blocked count: 1673
Waited count: 51202
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 16 (IPC Server handler 2 on 50050):
State: TIMED_WAITING
Blocked count: 1685
Waited count: 51188
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 15 (IPC Server handler 1 on 50050):
State: TIMED_WAITING
Blocked count: 1694
Waited count: 51183
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 14 (IPC Server handler 0 on 50050):
State: TIMED_WAITING
Blocked count: 1913
Waited count: 51137
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 13 (IPC Server listener on 50050):
State: RUNNABLE
Blocked count: 634
Waited count: 0
Stack:
sun.nio.ch.PollArrayWrapper.poll0(Native Method)
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
State: RUNNABLE
Blocked count: 85
Waited count: 1517
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:630)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 11 (btpool0-0 - Invalidator - /):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:933)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 9 (taskCleanup):
State: WAITING
Blocked count: 457
Waited count: 118
Waiting on java.util.ArrayList@15192a
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java:783)
org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 99047
Waited count: 39648
Waiting on java.lang.ref.ReferenceQueue$Lock@ac6214
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 102986
Waited count: 3131
Waiting on java.lang.ref.Reference$Lock@15b40a1
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 1538
Waited count: 107465
Stack:
sun.management.ThreadImpl.getThreadInfo0(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:93)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:141)
org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:553)
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)
> Final map task gets stuck
> -------------------------
>
> Key: HADOOP-707
> URL: http://issues.apache.org/jira/browse/HADOOP-707
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.8.0
> Environment: using latest trunk
> Reporter: Johan Oskarson
> Priority: Critical
>
> I've seen numerous jobs lately where the final map task gets stuck, never finishing.
> The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue and the job can finish.
> In the web interface it turns up as:
> task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12 10-Nov-2006 12:22:19 (1mins, 6sec)
> Task failed to report status for 604 seconds. Killing.
> Only exception I find in that tasktracker log is this (a few times):
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
> at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira