You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kevin Chen <be...@hotmail.com> on 2010/08/13 12:16:43 UTC
Hadoop 0.20.2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Hello,
I've confused the problem for a week already, Please sharing if you know what could be causing this, Thinks in advance!
Hadoop version: 0.20.2
Machines:
machine 1 - NameNode,JobTracker,SecondNameNode:
OS: ubuntu
Hostname: master
ulimit -n: 10240
disk space: enough
${mapred.local.dir}'s permissions: 777
machine 2 - DataNode, TaskTracker:
OS: ubuntu
Hostname: slave1
ulimit -n: 10240
disk space: enough
${mapred.local.dir}'s permissions: 777
machine 3 - DataNode, TaskTracker:
OS: ubuntu
Hostname: slave2
ulimit -n: 10240
disk space: enough
${mapred.local.dir}'s permissions: 777
Configurations:
core-site.xml:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
hdfs-site.xml:
<configuration>
</configuration>
mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
</configuration>
masters:
bestgembler@master
slaves:
hadoop@slave1
hadoop@slave2
Parts of error:
Terminal:
10/08/13 17:35:03 INFO mapred.FileInputFormat: Total input paths to process : 1
10/08/13 17:35:03 INFO mapred.JobClient: Running job: job_201008131730_0001
10/08/13 17:35:04 INFO mapred.JobClient: map 0% reduce 0%
10/08/13 17:35:14 INFO mapred.JobClient: map 50% reduce 0%
10/08/13 17:35:17 INFO mapred.JobClient: map 100% reduce 0%
10/08/13 17:35:23 INFO mapred.JobClient: Task Id : attempt_201008131730_0001_m_000001_0, Status : FAILED
Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
hadoop-bestgembler-jobtracker-master.log:
2010-08-13 17:32:54,127 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2010-08-13 17:32:54,128 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
2010-08-13 17:32:54,471 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave1
2010-08-13 17:32:57,603 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave2
2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201008131730_0001
2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201008131730_0001
2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201008131730_0001 = 1366. Number of splits = 2
2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave2
2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave1
2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave2
2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave1
2010-08-13 17:35:06,945 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000003_0' to tip task_201008131730_0001_m_000003, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
2010-08-13 17:35:10,062 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000003_0' has completed task_201008131730_0001_m_000003 successfully.
2010-08-13 17:35:10,068 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_0' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
2010-08-13 17:35:10,071 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
2010-08-13 17:35:12,958 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_0' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
2010-08-13 17:35:12,959 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
2010-08-13 17:35:13,084 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_0' has completed task_201008131730_0001_m_000000 successfully.
2010-08-13 17:35:13,109 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_r_000000_0' to tip task_201008131730_0001_r_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
2010-08-13 17:35:16,016 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_0' has completed task_201008131730_0001_m_000001 successfully.
2010-08-13 17:35:22,027 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000001_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:22,029 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000001_0' from 'tracker_slave1:localhost/127.0.0.1:56514'
2010-08-13 17:35:22,138 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:22,140 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_1' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000000_0' from 'tracker_slave2:localhost/127.0.0.1:42915'
2010-08-13 17:35:25,032 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_1' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
2010-08-13 17:35:25,033 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
2010-08-13 17:35:25,150 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_1' has completed task_201008131730_0001_m_000001 successfully.
2010-08-13 17:35:28,043 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_1' has completed task_201008131730_0001_m_000000 successfully.
2010-08-13 17:35:31,049 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_1: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
hadoop-hadoop-tasktracker-slave1.log:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = slave1/192.168.0.186
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-08-13 17:31:46,410 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2010-08-13 17:31:46,499 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2010-08-13 17:31:46,506 INFO org.mortbay.log: jetty-6.1.14
2010-08-13 17:31:46,714 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
2010-08-13 17:31:46,721 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
2010-08-13 17:31:46,731 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=56514
2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 56514: starting
2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 56514: starting
2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:56514
2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 56514: starting
2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 56514: starting
2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 56514: starting
2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave1:localhost/127.0.0.1:56514
2010-08-13 17:33:54,121 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave1:localhost/127.0.0.1:56514
2010-08-13 17:33:54,124 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@102f729e
2010-08-13 17:33:54,133 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2010-08-13 17:33:54,136 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2010-08-13 17:36:12,961 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_0 task's state:UNASSIGNED
2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_0
2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_0
2010-08-13 17:36:13,481 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-841871629
2010-08-13 17:36:13,482 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-841871629 spawned.
2010-08-13 17:36:13,918 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-841871629 given task: attempt_201008131730_0001_m_000001_0
2010-08-13 17:36:14,455 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_0 is done.
2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_0 was 0
2010-08-13 17:36:14,497 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:36:14,651 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-841871629 exited. Number of tasks it ran: 1
2010-08-13 17:36:15,961 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out in any of the configured local directories
2010-08-13 17:36:19,657 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:36:19,657 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_0
2010-08-13 17:36:19,666 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:45984, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_0
2010-08-13 17:36:19,666 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
2010-08-13 17:36:24,987 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_1 task's state:UNASSIGNED
2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_1
2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_1
2010-08-13 17:36:25,095 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-673950863
2010-08-13 17:36:25,096 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-673950863 spawned.
2010-08-13 17:36:25,475 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-673950863 given task: attempt_201008131730_0001_m_000000_1
2010-08-13 17:36:25,999 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_1 is done.
2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_1 was 0
2010-08-13 17:36:26,019 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:36:26,181 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-673950863 exited. Number of tasks it ran: 1
2010-08-13 17:36:27,988 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out in any of the configured local directories
2010-08-13 17:36:29,740 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:36:29,740 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_1
2010-08-13 17:36:29,746 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34390, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_1
2010-08-13 17:36:29,746 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
2010-08-13 17:36:31,005 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_2 task's state:UNASSIGNED
2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_2
2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_2
2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1281957734
2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1281957734 spawned.
2010-08-13 17:36:31,557 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1281957734 given task: attempt_201008131730_0001_m_000000_2
2010-08-13 17:36:32,079 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_2 is done.
2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_2 was 0
2010-08-13 17:36:32,086 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:36:32,271 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1281957734 exited. Number of tasks it ran: 1
2010-08-13 17:36:34,006 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out in any of the configured local directories
2010-08-13 17:36:39,785 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_2,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:36:39,785 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_2
2010-08-13 17:36:39,790 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34399, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_2
2010-08-13 17:36:39,790 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_3 task's state:UNASSIGNED
2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:40,157 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-485455839
2010-08-13 17:36:40,182 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-485455839 spawned.
2010-08-13 17:36:40,584 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-485455839 given task: attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:41,072 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_3 is done.
2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_3 was 0
2010-08-13 17:36:41,096 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:36:41,300 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-485455839 exited. Number of tasks it ran: 1
2010-08-13 17:36:43,027 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out in any of the configured local directories
2010-08-13 17:36:44,812 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_3,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:36:44,813 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:44,825 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34406, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:44,825 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
2010-08-13 17:36:46,044 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000002_0 task's state:UNASSIGNED
2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000002_0
2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000002_0
2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000000_3
2010-08-13 17:36:46,047 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_3 done; removing files.
2010-08-13 17:36:46,049 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_3 not found in cache
2010-08-13 17:36:46,151 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1831604016
2010-08-13 17:36:46,152 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1831604016 spawned.
2010-08-13 17:36:46,538 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1831604016 given task: attempt_201008131730_0001_m_000002_0
2010-08-13 17:36:46,766 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0%
2010-08-13 17:36:46,769 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0% cleanup
2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000002_0 is done.
2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000002_0 was 0
2010-08-13 17:36:46,790 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:36:46,978 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1831604016 exited. Number of tasks it ran: 1
2010-08-13 17:36:49,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000002_0/output/file.out in any of the configured local directories
2010-08-13 17:36:49,096 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
2010-08-13 17:36:49,097 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_2 done; removing files.
2010-08-13 17:36:49,099 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_2 not found in cache
2010-08-13 17:36:49,100 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_1 done; removing files.
2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_1 not found in cache
2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000002_0 done; removing files.
2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000002_0 not found in cache
2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_0 done; removing files.
2010-08-13 17:36:49,108 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_0 not found in cache
2010-08-13 17:37:01,309 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
2010-08-13 17:37:01,309 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '61
2010-08-13 17:37:01,388 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at slave1/192.168.0.186
************************************************************/
hadoop-hadoop-tasktracker-slave2.log:
2010-08-13 17:30:35,572 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = slave2/192.168.0.187
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-08-13 17:30:35,847 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2010-08-13 17:30:36,025 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2010-08-13 17:30:36,037 INFO org.mortbay.log: jetty-6.1.14
2010-08-13 17:32:45,915 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
2010-08-13 17:32:45,930 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
2010-08-13 17:32:45,952 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=42915
2010-08-13 17:32:46,023 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:42915
2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave2:localhost/127.0.0.1:42915
2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 42915: starting
2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 42915: starting
2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 42915: starting
2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 42915: starting
2010-08-13 17:32:46,034 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 42915: starting
2010-08-13 17:32:46,121 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@13a78071
2010-08-13 17:32:46,126 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2010-08-13 17:32:46,128 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2010-08-13 17:32:46,130 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave2:localhost/127.0.0.1:42915
2010-08-13 17:34:55,817 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000003_0 task's state:UNASSIGNED
2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000003_0
2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000003_0
2010-08-13 17:34:56,531 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1485663808
2010-08-13 17:34:56,532 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1485663808 spawned.
2010-08-13 17:34:57,324 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1485663808 given task: attempt_201008131730_0001_m_000003_0
2010-08-13 17:34:57,740 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000003_0 0.0% setup
2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000003_0 is done.
2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000003_0 was 0
2010-08-13 17:34:57,746 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:34:57,914 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1485663808 exited. Number of tasks it ran: 1
2010-08-13 17:34:58,817 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000003_0/output/file.out in any of the configured local directories
2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_0 task's state:UNASSIGNED
2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_0
2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000003_0
2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_0
2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000003_0
2010-08-13 17:34:58,843 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000003_0 done; removing files.
2010-08-13 17:34:58,844 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000003_0 not found in cache
2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_2093675286
2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_2093675286 spawned.
2010-08-13 17:34:59,647 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_2093675286 given task: attempt_201008131730_0001_m_000000_0
2010-08-13 17:35:00,527 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_0 is done.
2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_0 was 0
2010-08-13 17:35:00,530 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:35:00,635 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_2093675286 exited. Number of tasks it ran: 1
2010-08-13 17:35:01,840 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out in any of the configured local directories
2010-08-13 17:35:01,891 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_r_000000_0 task's state:UNASSIGNED
2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_r_000000_0
2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_r_000000_0
2010-08-13 17:35:02,003 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_r_-1485663808
2010-08-13 17:35:02,004 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_r_-1485663808 spawned.
2010-08-13 17:35:02,866 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_r_-1485663808 given task: attempt_201008131730_0001_r_000000_0
2010-08-13 17:35:08,507 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:08,508 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_0
2010-08-13 17:35:08,533 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:45820, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_0
2010-08-13 17:35:08,534 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
2010-08-13 17:35:09,311 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:10,908 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_1 task's state:UNASSIGNED
2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_1
2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_1
2010-08-13 17:35:11,067 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-83208921
2010-08-13 17:35:11,068 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-83208921 spawned.
2010-08-13 17:35:11,863 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-83208921 given task: attempt_201008131730_0001_m_000001_1
2010-08-13 17:35:12,316 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:12,733 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_1 is done.
2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_1 was 0
2010-08-13 17:35:12,736 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:35:12,876 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-83208921 exited. Number of tasks it ran: 1
2010-08-13 17:35:13,909 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out in any of the configured local directories
2010-08-13 17:35:18,323 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:18,548 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_1,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:18,551 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_1
2010-08-13 17:35:18,573 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34209, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_1
2010-08-13 17:35:18,573 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
2010-08-13 17:35:19,931 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_2 task's state:UNASSIGNED
2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_2
2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_2
2010-08-13 17:35:20,103 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-916850178
2010-08-13 17:35:20,106 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-916850178 spawned.
2010-08-13 17:35:20,947 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-916850178 given task: attempt_201008131730_0001_m_000001_2
2010-08-13 17:35:21,768 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_2 is done.
2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_2 was 0
2010-08-13 17:35:21,771 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:35:22,000 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-916850178 exited. Number of tasks it ran: 1
2010-08-13 17:35:22,931 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out in any of the configured local directories
2010-08-13 17:35:24,329 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:27,333 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:28,597 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_2,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:28,597 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_2
2010-08-13 17:35:28,616 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34216, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_2
2010-08-13 17:35:28,616 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
2010-08-13 17:35:28,952 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_3 task's state:UNASSIGNED
2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:29,108 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_302025266
2010-08-13 17:35:29,109 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_302025266 spawned.
2010-08-13 17:35:29,845 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_302025266 given task: attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:30,701 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_3 is done.
2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_3 was 0
2010-08-13 17:35:30,704 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:35:30,828 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_302025266 exited. Number of tasks it ran: 1
2010-08-13 17:35:31,953 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out in any of the configured local directories
2010-08-13 17:35:33,339 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
2010-08-13 17:35:33,626 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_3,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-08-13 17:35:33,627 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:33,638 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34225, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:33,638 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
2010-08-13 17:35:34,969 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:34,970 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000001_3
2010-08-13 17:35:34,971 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_3 done; removing files.
2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_3 not found in cache
2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_r_000000_0
2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_r_000000_0
2010-08-13 17:35:34,975 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2010-08-13 17:35:34,976 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
2010-08-13 17:35:35,073 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_r_-1485663808 exited. Number of tasks it ran: 0
2010-08-13 17:35:37,975 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_0 done; removing files.
2010-08-13 17:35:37,982 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_0 not found in cache
2010-08-13 17:35:37,983 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_1 done; removing files.
2010-08-13 17:35:37,984 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_1 not found in cache
2010-08-13 17:35:37,985 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_2 done; removing files.
2010-08-13 17:35:37,987 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_2 not found in cache
2010-08-13 17:35:50,122 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
2010-08-13 17:35:50,123 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '60
2010-08-13 17:35:50,513 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at slave2/192.168.0.187
************************************************************/
Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Hemanth Yamijala <yh...@gmail.com>.
Hi,
> Hi, Hemanth. Thinks for your reply!
>
> I tried your recommendation, absolute path, it worked, I was able to run the jobs successfully. Thank you!
> I was wondering why hadoop.tmp.dir ( or mapred.local.dir ? ) with relative path didn't work.
I am not entirely sure, but when the daemon is launched on a slave
node, I don't really know what the current directory is set to. Hence,
it is unpredictable.
>
> Thanks.
>
>> Date: Fri, 13 Aug 2010 16:35:24 +0530
>> Subject: Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
>> From: yhemanth@gmail.com
>> To: common-user@hadoop.apache.org
>>
>> Hi,
>>
>> I am not sure if a relative part works. When I tried with a similar
>> setting, I wasn't able to run jobs, though admittedly the problem was
>> not the same as what you got. Could you try with a full path just in
>> case...
>>
>> Thanks
>> hemanth
>
RE: Hadoop 0.20.2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by "Kevin ." <be...@hotmail.com>.
> From: awittenauer@linkedin.com
> To: common-user@hadoop.apache.org
> Date: Mon, 16 Aug 2010 06:44:37 +0000
>
>
> On Aug 15, 2010, at 8:07 PM, Kevin . wrote:
> > I tried your recommendation, absolute path, it worked, I was able to run the jobs successfully. Thank you!
> > I was wondering why hadoop.tmp.dir ( or mapred.local.dir ? ) with relative path didn't work.
>
>
> Allen's Hadoop Operations Rule #1: Nothing works the way you expect it would.
>
> ;)
You've hit the nail on the head, Allen.
:-)
Re: Hadoop 0.20.2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Aug 15, 2010, at 8:07 PM, Kevin . wrote:
> I tried your recommendation, absolute path, it worked, I was able to run the jobs successfully. Thank you!
> I was wondering why hadoop.tmp.dir ( or mapred.local.dir ? ) with relative path didn't work.
Allen's Hadoop Operations Rule #1: Nothing works the way you expect it would.
;)
Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Patrick Angeles <pa...@cloudera.com>.
I'd also recommend setting mapred.local.dir and dfs.data.dir to something
that is not under /tmp.
Aside from your HDFS data getting wiped, these settings should ideally be
comma separated paths, one for each physical disk in your server so you can
aggregate disk I/O.
2010/8/15 Kevin . <be...@hotmail.com>
>
> Hi, Hemanth. Thinks for your reply!
>
> I tried your recommendation, absolute path, it worked, I was able to run
> the jobs successfully. Thank you!
> I was wondering why hadoop.tmp.dir ( or mapred.local.dir ? ) with relative
> path didn't work.
>
> Thanks.
>
> > Date: Fri, 13 Aug 2010 16:35:24 +0530
> > Subject: Re: Hadoop 0.20.2:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
> in any of the configured local directories
> > From: yhemanth@gmail.com
> > To: common-user@hadoop.apache.org
> >
> > Hi,
> >
> > I am not sure if a relative part works. When I tried with a similar
> > setting, I wasn't able to run jobs, though admittedly the problem was
> > not the same as what you got. Could you try with a full path just in
> > case...
> >
> > Thanks
> > hemanth
>
>
RE: Hadoop 0.20.2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by "Kevin ." <be...@hotmail.com>.
Hi, Hemanth. Thinks for your reply!
I tried your recommendation, absolute path, it worked, I was able to run the jobs successfully. Thank you!
I was wondering why hadoop.tmp.dir ( or mapred.local.dir ? ) with relative path didn't work.
Thanks.
> Date: Fri, 13 Aug 2010 16:35:24 +0530
> Subject: Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
> From: yhemanth@gmail.com
> To: common-user@hadoop.apache.org
>
> Hi,
>
> I am not sure if a relative part works. When I tried with a similar
> setting, I wasn't able to run jobs, though admittedly the problem was
> not the same as what you got. Could you try with a full path just in
> case...
>
> Thanks
> hemanth
Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Hemanth Yamijala <yh...@gmail.com>.
Hi,
> 1. I login through SSH without password from master and slaves, it's all right :-)
>
> 2.
> <property>
> <name>hadoop.tmp.dir</name>
> <value>tmp</value>
> </property>
>
> In fact, 'tmp' is what I want :-)
>
> $HADOOP_HOME
> + tmp
> + dfs
> + mapred
>
I am not sure if a relative part works. When I tried with a similar
setting, I wasn't able to run jobs, though admittedly the problem was
not the same as what you got. Could you try with a full path just in
case...
Thanks
hemanth
>
>> From: yhemanth@gmail.com
>>
>> Aside from usual checks regarding connectivity between master and
>> slaves, one quick observation is that the hadoop.tmp.dir says 'tmp'.
>> Shouldn't this be '/tmp' ?
>>
>> Thanks
>> Hemanth
>
RE: Hadoop 0.20.2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Kevin Chen <be...@hotmail.com>.
Thinks for your reply!
1. I login through SSH without password from master and slaves, it's all right :-)
2.
<property>
<name>hadoop.tmp.dir</name>
<value>tmp</value>
</property>
In fact, 'tmp' is what I want :-)
$HADOOP_HOME
+ tmp
+ dfs
+ mapred
> From: yhemanth@gmail.com
>
> Aside from usual checks regarding connectivity between master and
> slaves, one quick observation is that the hadoop.tmp.dir says 'tmp'.
> Shouldn't this be '/tmp' ?
>
> Thanks
> Hemanth
Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index
in any of the configured local directories
Posted by Hemanth Yamijala <yh...@gmail.com>.
Aside from usual checks regarding connectivity between master and
slaves, one quick observation is that the hadoop.tmp.dir says 'tmp'.
Shouldn't this be '/tmp' ?
Thanks
Hemanth
2010/8/13 Kevin Chen <be...@hotmail.com>:
>
> Hello,
>
> I've confused the problem for a week already, Please sharing if you know what could be causing this, Thinks in advance!
>
> Hadoop version: 0.20.2
>
> Machines:
>
> machine 1 - NameNode,JobTracker,SecondNameNode:
> OS: ubuntu
> Hostname: master
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
> machine 2 - DataNode, TaskTracker:
>
> OS: ubuntu
>
> Hostname: slave1
>
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
> machine 3 - DataNode, TaskTracker:
>
> OS: ubuntu
>
> Hostname: slave2
>
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
>
> Configurations:
>
> core-site.xml:
> <configuration>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>tmp</value>
> </property>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master:9000</value>
> </property>
> </configuration>
>
>
> hdfs-site.xml:
> <configuration>
> </configuration>
>
>
> mapred-site.xml:
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>master:9001</value>
> </property>
> </configuration>
>
>
> masters:
> bestgembler@master
>
>
> slaves:
> hadoop@slave1
> hadoop@slave2
>
> Parts of error:
>
>
> Terminal:
> 10/08/13 17:35:03 INFO mapred.FileInputFormat: Total input paths to process : 1
> 10/08/13 17:35:03 INFO mapred.JobClient: Running job: job_201008131730_0001
> 10/08/13 17:35:04 INFO mapred.JobClient: map 0% reduce 0%
> 10/08/13 17:35:14 INFO mapred.JobClient: map 50% reduce 0%
> 10/08/13 17:35:17 INFO mapred.JobClient: map 100% reduce 0%
> 10/08/13 17:35:23 INFO mapred.JobClient: Task Id : attempt_201008131730_0001_m_000001_0, Status : FAILED
> Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> hadoop-bestgembler-jobtracker-master.log:
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:32:54,128 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
> 2010-08-13 17:32:54,471 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave1
> 2010-08-13 17:32:57,603 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave2
> 2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201008131730_0001
> 2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201008131730_0001
> 2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201008131730_0001 = 1366. Number of splits = 2
> 2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave2
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave1
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave2
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave1
> 2010-08-13 17:35:06,945 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000003_0' to tip task_201008131730_0001_m_000003, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:10,062 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000003_0' has completed task_201008131730_0001_m_000003 successfully.
> 2010-08-13 17:35:10,068 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_0' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:10,071 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
> 2010-08-13 17:35:12,958 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_0' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:12,959 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
> 2010-08-13 17:35:13,084 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_0' has completed task_201008131730_0001_m_000000 successfully.
> 2010-08-13 17:35:13,109 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_r_000000_0' to tip task_201008131730_0001_r_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:16,016 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_0' has completed task_201008131730_0001_m_000001 successfully.
> 2010-08-13 17:35:22,027 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000001_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:22,029 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000001_0' from 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:22,138 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:22,140 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_1' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
> 2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000000_0' from 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:25,032 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_1' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:25,033 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
> 2010-08-13 17:35:25,150 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_1' has completed task_201008131730_0001_m_000001 successfully.
> 2010-08-13 17:35:28,043 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_1' has completed task_201008131730_0001_m_000000 successfully.
> 2010-08-13 17:35:31,049 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_1: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
>
> hadoop-hadoop-tasktracker-slave1.log:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = slave1/192.168.0.186
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-08-13 17:31:46,410 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2010-08-13 17:31:46,499 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2010-08-13 17:31:46,506 INFO org.mortbay.log: jetty-6.1.14
> 2010-08-13 17:31:46,714 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2010-08-13 17:31:46,721 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-08-13 17:31:46,731 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=56514
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 56514: starting
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:56514
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave1:localhost/127.0.0.1:56514
> 2010-08-13 17:33:54,121 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave1:localhost/127.0.0.1:56514
> 2010-08-13 17:33:54,124 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@102f729e
> 2010-08-13 17:33:54,133 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2010-08-13 17:33:54,136 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2010-08-13 17:36:12,961 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_0 task's state:UNASSIGNED
> 2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:13,481 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-841871629
> 2010-08-13 17:36:13,482 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-841871629 spawned.
> 2010-08-13 17:36:13,918 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-841871629 given task: attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:14,455 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_0 is done.
> 2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_0 was 0
> 2010-08-13 17:36:14,497 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:14,651 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-841871629 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:15,961 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out in any of the configured local directories
> 2010-08-13 17:36:19,657 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:19,657 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:19,666 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:45984, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:19,666 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:24,987 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_1 task's state:UNASSIGNED
> 2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:25,095 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-673950863
> 2010-08-13 17:36:25,096 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-673950863 spawned.
> 2010-08-13 17:36:25,475 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-673950863 given task: attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:25,999 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_1 is done.
> 2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_1 was 0
> 2010-08-13 17:36:26,019 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:26,181 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-673950863 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:27,988 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out in any of the configured local directories
> 2010-08-13 17:36:29,740 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:29,740 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:29,746 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34390, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:29,746 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:31,005 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_2 task's state:UNASSIGNED
> 2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1281957734
> 2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1281957734 spawned.
> 2010-08-13 17:36:31,557 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1281957734 given task: attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:32,079 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_2 is done.
> 2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_2 was 0
> 2010-08-13 17:36:32,086 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:32,271 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1281957734 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:34,006 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out in any of the configured local directories
> 2010-08-13 17:36:39,785 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_2,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:39,785 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:39,790 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34399, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:39,790 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_3 task's state:UNASSIGNED
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:40,157 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-485455839
> 2010-08-13 17:36:40,182 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-485455839 spawned.
> 2010-08-13 17:36:40,584 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-485455839 given task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:41,072 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_3 is done.
> 2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_3 was 0
> 2010-08-13 17:36:41,096 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:41,300 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-485455839 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:43,027 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out in any of the configured local directories
> 2010-08-13 17:36:44,812 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_3,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:44,813 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:44,825 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34406, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:44,825 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:46,044 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000002_0 task's state:UNASSIGNED
> 2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:46,047 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_3 done; removing files.
> 2010-08-13 17:36:46,049 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_3 not found in cache
> 2010-08-13 17:36:46,151 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1831604016
> 2010-08-13 17:36:46,152 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1831604016 spawned.
> 2010-08-13 17:36:46,538 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1831604016 given task: attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,766 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0%
> 2010-08-13 17:36:46,769 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0% cleanup
> 2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000002_0 is done.
> 2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000002_0 was 0
> 2010-08-13 17:36:46,790 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:46,978 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1831604016 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:49,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000002_0/output/file.out in any of the configured local directories
> 2010-08-13 17:36:49,096 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
> 2010-08-13 17:36:49,097 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_2 done; removing files.
> 2010-08-13 17:36:49,099 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_2 not found in cache
> 2010-08-13 17:36:49,100 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_1 done; removing files.
> 2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_1 not found in cache
> 2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000002_0 done; removing files.
> 2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000002_0 not found in cache
> 2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_0 done; removing files.
> 2010-08-13 17:36:49,108 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_0 not found in cache
> 2010-08-13 17:37:01,309 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
> at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
> at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> 2010-08-13 17:37:01,309 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '61
> 2010-08-13 17:37:01,388 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at slave1/192.168.0.186
> ************************************************************/
>
>
> hadoop-hadoop-tasktracker-slave2.log:
> 2010-08-13 17:30:35,572 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = slave2/192.168.0.187
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-08-13 17:30:35,847 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2010-08-13 17:30:36,025 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2010-08-13 17:30:36,037 INFO org.mortbay.log: jetty-6.1.14
> 2010-08-13 17:32:45,915 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2010-08-13 17:32:45,930 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-08-13 17:32:45,952 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=42915
> 2010-08-13 17:32:46,023 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:42915
> 2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave2:localhost/127.0.0.1:42915
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 42915: starting
> 2010-08-13 17:32:46,034 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 42915: starting
> 2010-08-13 17:32:46,121 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@13a78071
> 2010-08-13 17:32:46,126 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2010-08-13 17:32:46,128 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2010-08-13 17:32:46,130 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave2:localhost/127.0.0.1:42915
> 2010-08-13 17:34:55,817 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000003_0 task's state:UNASSIGNED
> 2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:56,531 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1485663808
> 2010-08-13 17:34:56,532 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1485663808 spawned.
> 2010-08-13 17:34:57,324 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1485663808 given task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:57,740 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000003_0 0.0% setup
> 2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000003_0 is done.
> 2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000003_0 was 0
> 2010-08-13 17:34:57,746 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:34:57,914 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1485663808 exited. Number of tasks it ran: 1
> 2010-08-13 17:34:58,817 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000003_0/output/file.out in any of the configured local directories
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_0 task's state:UNASSIGNED
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:58,843 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000003_0 done; removing files.
> 2010-08-13 17:34:58,844 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000003_0 not found in cache
> 2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_2093675286
> 2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_2093675286 spawned.
> 2010-08-13 17:34:59,647 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_2093675286 given task: attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:00,527 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_0 is done.
> 2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_0 was 0
> 2010-08-13 17:35:00,530 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:00,635 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_2093675286 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:01,840 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out in any of the configured local directories
> 2010-08-13 17:35:01,891 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_r_000000_0 task's state:UNASSIGNED
> 2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:02,003 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_r_-1485663808
> 2010-08-13 17:35:02,004 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_r_-1485663808 spawned.
> 2010-08-13 17:35:02,866 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_r_-1485663808 given task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:08,507 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:08,508 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:08,533 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:45820, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:08,534 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:09,311 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:10,908 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_1 task's state:UNASSIGNED
> 2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:11,067 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-83208921
> 2010-08-13 17:35:11,068 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-83208921 spawned.
> 2010-08-13 17:35:11,863 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-83208921 given task: attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:12,316 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:12,733 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_1 is done.
> 2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_1 was 0
> 2010-08-13 17:35:12,736 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:12,876 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-83208921 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:13,909 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out in any of the configured local directories
> 2010-08-13 17:35:18,323 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:18,548 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:18,551 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:18,573 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34209, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:18,573 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:19,931 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_2 task's state:UNASSIGNED
> 2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:20,103 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-916850178
> 2010-08-13 17:35:20,106 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-916850178 spawned.
> 2010-08-13 17:35:20,947 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-916850178 given task: attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:21,768 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_2 is done.
> 2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_2 was 0
> 2010-08-13 17:35:21,771 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:22,000 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-916850178 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:22,931 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out in any of the configured local directories
> 2010-08-13 17:35:24,329 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:27,333 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:28,597 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_2,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:28,597 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:28,616 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34216, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:28,616 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:28,952 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_3 task's state:UNASSIGNED
> 2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:29,108 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_302025266
> 2010-08-13 17:35:29,109 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_302025266 spawned.
> 2010-08-13 17:35:29,845 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_302025266 given task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:30,701 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_3 is done.
> 2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_3 was 0
> 2010-08-13 17:35:30,704 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:30,828 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_302025266 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:31,953 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out in any of the configured local directories
> 2010-08-13 17:35:33,339 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:33,626 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_3,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:33,627 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:33,638 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34225, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:33,638 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:34,969 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:34,970 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:34,971 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_3 done; removing files.
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_3 not found in cache
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:34,975 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:34,976 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
> 2010-08-13 17:35:35,073 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_r_-1485663808 exited. Number of tasks it ran: 0
> 2010-08-13 17:35:37,975 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
> 2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
> 2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_0 done; removing files.
> 2010-08-13 17:35:37,982 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_0 not found in cache
> 2010-08-13 17:35:37,983 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_1 done; removing files.
> 2010-08-13 17:35:37,984 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_1 not found in cache
> 2010-08-13 17:35:37,985 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_2 done; removing files.
> 2010-08-13 17:35:37,987 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_2 not found in cache
> 2010-08-13 17:35:50,122 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
> at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
> at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> 2010-08-13 17:35:50,123 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '60
> 2010-08-13 17:35:50,513 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at slave2/192.168.0.187
> ************************************************************/
>