You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Abdelrahman Kamel <ab...@gmail.com> on 2011/09/26 17:16:37 UTC
Too many fetch failures. Help!
Hi,
This is my first post here.
I'm new to Hadoop.
I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
slave and the other is only slave).
When I run a Wordcount example on 5 small txt files, the process never
completes and I get a "Too many fetch failures" error on my terminal.
If you can help me, I cant post my terminal's output and any log files
needed.
Great thanks.
--
Abdelrahman Kamel
Re: Too many fetch failures. Help!
Posted by Abdelrahman Kamel <ab...@gmail.com>.
Thanks again.
I have solved my problem by commenting out "127.0.1.1 hdslave" from
/etc/hosts in all nodes.
On Wed, Sep 28, 2011 at 10:15 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Try commenting out "127.0.0.1 localhost" line from /etc/hosts in all our
> systems.
>
>
> On Tue, Sep 27, 2011 at 11:59 PM, Abdelrahman Kamel <abdoukamel@gmail.com
> >wrote:
>
> > Thanks very much for all your fast replies.
> >
> > *Here is my terminal output:*
> >
> > hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar
> > hadoop-0.20.2-examples.jar
> > wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> > 11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to
> process
> > :
> > 5
> > 11/09/28 07:38:23 INFO mapred.JobClient: Running job:
> job_201109280735_0002
> > 11/09/28 07:38:24 INFO mapred.JobClient: map 0% reduce 0%
> > 11/09/28 07:38:42 INFO mapred.JobClient: map 20% reduce 0%
> > 11/09/28 07:38:44 INFO mapred.JobClient: map 40% reduce 0%
> > 11/09/28 07:38:45 INFO mapred.JobClient: map 60% reduce 0%
> > 11/09/28 07:38:47 INFO mapred.JobClient: map 80% reduce 0%
> > 11/09/28 07:38:51 INFO mapred.JobClient: map 100% reduce 0%
> > 11/09/28 07:38:54 INFO mapred.JobClient: map 100% reduce 13%
> > 11/09/28 07:39:01 INFO mapred.JobClient: map 100% reduce 20%
> >
> > *The terminal is stuck here.*
> >
> > *And here is my JobTracker log:*
> >
> > 2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler
> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=JobTracker, port=54311
> > 2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50030
> > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50030
> > webServer.getConnectors()[0].getLocalPort() returned 50030
> > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50030
> > 2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:53,816 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50030
> > 2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=JobTracker, sessionId=
> > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker
> > up at: 54311
> > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker
> > webserver: 50030
> > 2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> > Safe mode will be turned off automatically.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> > Safe mode will be turned off automatically.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 25 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 15 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 5 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:37:10,107 INFO
> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> > inactive
> > 2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54311: starting
> > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 54311: starting
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker:
> Starting
> > RUNNING
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 54311: starting
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 54311: starting
> > 2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 54311: starting
> > 2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology:
> Adding
> > a
> > new node: /default-rack/hdmaster
> > 2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology:
> Adding
> > a
> > new node: /default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker:
> > Initializing job_201109280735_0002
> > 2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress:
> > Initializing job_201109280735_0002
> > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
> Input
> > size for job job_201109280735_0002 = 3430958. Number of splits = 5
> > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000000 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000000 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000001 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000001 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000002 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000002 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000003 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000003 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000004 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000004 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000006_0' to tip
> > task_201109280735_0002_m_000006, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000006_0' has completed
> > task_201109280735_0002_m_000006 successfully.
> > 2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000000_0' to tip
> > task_201109280735_0002_m_000000, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000000
> > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000001_0' to tip
> > task_201109280735_0002_m_000001, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000001
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000002_0' to tip
> > task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/
> > 127.0.0.1:54873'
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000002
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000003_0' to tip
> > task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/
> > 127.0.0.1:54873'
> > 2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000003
> > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000000_0' has completed
> > task_201109280735_0002_m_000000 successfully.
> > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:1 completedMapsInputSize:1423802
> > completedMapsOutputSize:466814
> > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000004_0' to tip
> > task_201109280735_0002_m_000004, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000004
> > 2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_r_000000_0' to tip
> > task_201109280735_0002_r_000000, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000003_0' has completed
> > task_201109280735_0002_m_000003 successfully.
> > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:2 completedMapsInputSize:1817766
> > completedMapsOutputSize:719434
> > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000001_0' has completed
> > task_201109280735_0002_m_000001 successfully.
> > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:3 completedMapsInputSize:2492333
> > completedMapsOutputSize:986447
> > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000002_0' has completed
> > task_201109280735_0002_m_000002 successfully.
> > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:4 completedMapsInputSize:3087267
> > completedMapsOutputSize:1204101
> > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000004_0' has completed
> > task_201109280735_0002_m_000004 successfully.
> > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:5 completedMapsInputSize:3430963
> > completedMapsOutputSize:1367489
> > 2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #1 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #2 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #3 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too
> > many fetch-failures for output of task:
> > attempt_201109280735_0002_m_000003_0
> > ... killing it
> > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress:
> Error
> > from attempt_201109280735_0002_m_000003_0: Too many fetch-failures
> > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000003_1' to tip
> > task_201109280735_0002_m_000003, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000003
> > 2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed
> > completed task 'attempt_201109280735_0002_m_000003_0' from
> > 'tracker_hdmaster:localhost/127.0.0.1:54873'
> > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000003_1' has completed
> > task_201109280735_0002_m_000003 successfully.
> > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:6 completedMapsInputSize:3824927
> > completedMapsOutputSize:1620109
> >
> > *And here is my TaskTracker #1 log:*
> >
> > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50060
> > webServer.getConnectors()[0].getLocalPort() returned 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50060
> > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50060
> > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=TaskTracker, sessionId=
> > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=TaskTracker, port=54873
> > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker up at: localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > tracker tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54873: starting
> > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54873: starting
> > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_1909679810 spawned.
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-622388657 spawned.
> > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_1909679810 given task:
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-622388657 given task:
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000003_0 1.0%
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000003_0 is done.
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000003_0 was 0
> > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000002_0 1.0%
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000002_0 is done.
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000002_0 was 0
> > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
> >
> >
> > *And here is my TaskTracker #2 log:*
> >
> > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50060
> > webServer.getConnectors()[0].getLocalPort() returned 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50060
> > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50060
> > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=TaskTracker, sessionId=
> > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=TaskTracker, port=54873
> > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker up at: localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > tracker tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54873: starting
> > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54873: starting
> > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_1909679810 spawned.
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-622388657 spawned.
> > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_1909679810 given task:
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-622388657 given task:
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000003_0 1.0%
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000003_0 is done.
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000003_0 was 0
> > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000002_0 1.0%
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000002_0 is done.
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000002_0 was 0
> > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
> > 2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/10.0.0.1:54311. Already tried 0 time(s).
> > 2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/10.0.0.1:54311. Already tried 1 time(s).
> > 2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342
> > 2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad
> > 2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481
> > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_774270481 spawned.
> > 2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_774270481 given task:
> > attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000006_0 0.0% setup
> > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000006_0 is done.
> > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000006_0 was 0
> > 2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker:
> > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> >
> >
> taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out
> > in any of the configured local directories
> > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction
> > (registerTask): attempt_201109280735_0002_m_000000_0 task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker:
> Received
> > KillTaskAction for task: attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About
> to
> > purge task: attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner:
> > attempt_201109280735_0002_m_000006_0 done; removing files.
> > 2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID
> > attempt_201109280735_0002_m_000006_0 not found in cache
> > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571
> > 2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-1878969571 spawned.
> > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930
> > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-364970930 spawned.
> > 2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-364970930 given task:
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-1878969571 given task:
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000000_0 1.0%
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000000_0 is done.
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000000_0 was 0
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236
> > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_531367236 spawned.
> > 2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481
> > 2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_r_774270481 spawned.
> > 2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_r_774270481 given task:
> > attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_531367236 given task:
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000001_0 1.0%
> > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000001_0 is done.
> > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000001_0 was 0
> > 2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000004_0 1.0%
> > 2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO
> > org.apache.hadoop.mapred.TaskTracker: reported output size for
> > attempt_201109280735_0002_m_000004_0 was 0
> > 2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0
> > given 466814/466810
> > 2011-09-28 07:37:47,276 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0
> > given 267013/267009
> > 2011-09-28 07:37:47,287 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> > 0.12 MB/s) >
> > 2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> > 0.12 MB/s) >
> > 2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0
> > given 163388/163384
> > 2011-09-28 07:37:51,176 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >2011-09-28 07:38:42,150 INFO
> > org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0
> > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> > 2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0
> > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> > 2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker:
> >
> >
> > *Great thanks hoping you could help me.*
> >
> >
> > On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k <de...@huawei.com> wrote:
> >
> > > Hi Bharath,
> > >
> > > There are few reasons to cause this problem. I have listed below some
> > > reasons with solutions. This might help you to solve this. If you post
> > the
> > > logs, the problem can be figured out.
> > >
> > >
> > > Reason 1:
> > > It could be that the mapping in the /etc/hosts file is not present.
> > > The DNS server is down as a result of which the hostnames cannot be
> > > resolved.
> > > The DNS server is in-correctly configured.
> > >
> > > Solution: Setting the slave.host.name property can be one solution.
> > > Appropriate changes need to be done based on the problem.
> > >
> > >
> > > Reason 2: If the map outputs are larger, we may get
> > > java.lang.OutOfMemoryError: Java heap space. Because of this there are
> > too
> > > many fetch failures.
> > >
> > > Solution: The error, java.lang.OutOfMemoryError: Java heap space in
> task
> > > tracker logs can be solved by any of the following methods:
> > >
> > > By decreasing the value configured for
> > > mapred.job.shuffle.input.buffer.percent.
> > > By increasing the heap memory of child JVM options for the property
> > > mapred.child.java.opts.
> > >
> > >
> > >
> > >
> > > Thanks
> > > Devaraj
> > >
> > > ________________________________________
> > > From: bharath vissapragada [bharathvissapragada1990@gmail.com]
> > > Sent: Monday, September 26, 2011 8:54 PM
> > > To: common-user@hadoop.apache.org
> > > Subject: Re: Too many fetch failures. Help!
> > >
> > > Hey,
> > >
> > > Try configuring your cluster with hostnames instead of ips and add
> > > those entries to /etc/hosts and sync it across all the nodes in the
> > > cluster. You need to restart the cluster after making these changes.
> > >
> > > Hope this helps,
> > >
> > > On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <
> abdoukamel@gmail.com
> > >
> > > wrote:
> > > > Hi,
> > > > This is my first post here.
> > > > I'm new to Hadoop.
> > > > I've already installed Hadoop on 2 Ubuntu boxes (one is both master
> and
> > > > slave and the other is only slave).
> > > > When I run a Wordcount example on 5 small txt files, the process
> never
> > > > completes and I get a "Too many fetch failures" error on my terminal.
> > > > If you can help me, I cant post my terminal's output and any log
> files
> > > > needed.
> > > > Great thanks.
> > > >
> > > > --
> > > > Abdelrahman Kamel
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Bharath .V
> > > w:http://researchweb.iiit.ac.in/~bharath.v
> > >
> >
> >
> >
> > --
> > Abdelrahman Kamel
> >
>
--
Abdelrahman Kamel
Re: Too many fetch failures. Help!
Posted by praveenesh kumar <pr...@gmail.com>.
Try commenting out "127.0.0.1 localhost" line from /etc/hosts in all our
systems.
On Tue, Sep 27, 2011 at 11:59 PM, Abdelrahman Kamel <ab...@gmail.com>wrote:
> Thanks very much for all your fast replies.
>
> *Here is my terminal output:*
>
> hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar
> hadoop-0.20.2-examples.jar
> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> 11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to process
> :
> 5
> 11/09/28 07:38:23 INFO mapred.JobClient: Running job: job_201109280735_0002
> 11/09/28 07:38:24 INFO mapred.JobClient: map 0% reduce 0%
> 11/09/28 07:38:42 INFO mapred.JobClient: map 20% reduce 0%
> 11/09/28 07:38:44 INFO mapred.JobClient: map 40% reduce 0%
> 11/09/28 07:38:45 INFO mapred.JobClient: map 60% reduce 0%
> 11/09/28 07:38:47 INFO mapred.JobClient: map 80% reduce 0%
> 11/09/28 07:38:51 INFO mapred.JobClient: map 100% reduce 0%
> 11/09/28 07:38:54 INFO mapred.JobClient: map 100% reduce 13%
> 11/09/28 07:39:01 INFO mapred.JobClient: map 100% reduce 20%
>
> *The terminal is stuck here.*
>
> *And here is my JobTracker log:*
>
> 2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = hdmaster/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
> configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1.
> Opening the listener on 50030
> 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 50030
> 2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14
> 2011-09-28 07:35:53,816 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> up at: 54311
> 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> webserver: 50030
> 2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem
> cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> Safe mode will be turned off automatically.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> 2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem
> cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> Safe mode will be turned off automatically.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> 2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem
> cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
> mode will be turned off automatically in 25 seconds.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> 2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem
> cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
> mode will be turned off automatically in 15 seconds.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> 2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem
> cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
> mode will be turned off automatically in 5 seconds.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy4.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> 2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2011-09-28 07:37:10,107 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 54311: starting
> 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54311: starting
> 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 54311: starting
> 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 54311: starting
> 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 54311: starting
> 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 54311: starting
> 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 54311: starting
> 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 54311: starting
> 2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker: Starting
> RUNNING
> 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 54311: starting
> 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 54311: starting
> 2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 54311: starting
> 2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology: Adding
> a
> new node: /default-rack/hdmaster
> 2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology: Adding
> a
> new node: /default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker:
> Initializing job_201109280735_0002
> 2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress:
> Initializing job_201109280735_0002
> 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress: Input
> size for job job_201109280735_0002 = 3430958. Number of splits = 5
> 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000000 has split on
> node:/default-rack/hdmaster
> 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000000 has split on
> node:/default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000001 has split on
> node:/default-rack/hdmaster
> 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000001 has split on
> node:/default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000002 has split on
> node:/default-rack/hdmaster
> 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000002 has split on
> node:/default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000003 has split on
> node:/default-rack/hdmaster
> 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000003 has split on
> node:/default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000004 has split on
> node:/default-rack/hdmaster
> 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201109280735_0002_m_000004 has split on
> node:/default-rack/hdslave.ubuntu-domain
> 2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000006_0' to tip
> task_201109280735_0002_m_000006, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000006_0' has completed
> task_201109280735_0002_m_000006 successfully.
> 2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000000_0' to tip
> task_201109280735_0002_m_000000, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000000
> 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000001_0' to tip
> task_201109280735_0002_m_000001, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000001
> 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000002_0' to tip
> task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/
> 127.0.0.1:54873'
> 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000002
> 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000003_0' to tip
> task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/
> 127.0.0.1:54873'
> 2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000003
> 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000000_0' has completed
> task_201109280735_0002_m_000000 successfully.
> 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:1 completedMapsInputSize:1423802
> completedMapsOutputSize:466814
> 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000004_0' to tip
> task_201109280735_0002_m_000004, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000004
> 2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_r_000000_0' to tip
> task_201109280735_0002_r_000000, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000003_0' has completed
> task_201109280735_0002_m_000003 successfully.
> 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:2 completedMapsInputSize:1817766
> completedMapsOutputSize:719434
> 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000001_0' has completed
> task_201109280735_0002_m_000001 successfully.
> 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:3 completedMapsInputSize:2492333
> completedMapsOutputSize:986447
> 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000002_0' has completed
> task_201109280735_0002_m_000002 successfully.
> 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:4 completedMapsInputSize:3087267
> completedMapsOutputSize:1204101
> 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000004_0' has completed
> task_201109280735_0002_m_000004 successfully.
> 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:5 completedMapsInputSize:3430963
> completedMapsOutputSize:1367489
> 2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress: Failed
> fetch notification #1 for task attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress: Failed
> fetch notification #2 for task attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress: Failed
> fetch notification #3 for task attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too
> many fetch-failures for output of task:
> attempt_201109280735_0002_m_000003_0
> ... killing it
> 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> from attempt_201109280735_0002_m_000003_0: Too many fetch-failures
> 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task 'attempt_201109280735_0002_m_000003_1' to tip
> task_201109280735_0002_m_000003, for tracker
> 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local task task_201109280735_0002_m_000003
> 2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed
> completed task 'attempt_201109280735_0002_m_000003_0' from
> 'tracker_hdmaster:localhost/127.0.0.1:54873'
> 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201109280735_0002_m_000003_1' has completed
> task_201109280735_0002_m_000003 successfully.
> 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:6 completedMapsInputSize:3824927
> completedMapsOutputSize:1620109
>
> *And here is my TaskTracker #1 log:*
>
> 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = hdmaster/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1.
> Opening the listener on 50060
> 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 50060
> 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=54873
> 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost/127.0.0.1:54873
> 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tracker tracker_hdmaster:localhost/127.0.0.1:54873
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 54873: starting
> 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 54873: starting
> 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> thread: Map-events fetcher for all reduce tasks on
> tracker_hdmaster:localhost/127.0.0.1:54873
> 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> MemoryCalculatorPlugin :
> org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache
> created with max memory = 10485760
> 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 1 and trying to launch
> attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_1909679810 spawned.
> 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_-622388657 spawned.
> 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_1909679810 given task:
> attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_-622388657 given task:
> attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000003_0 1.0%
> 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000003_0 is done.
> 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000003_0 was 0
> 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 1
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000002_0 1.0%
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000002_0 is done.
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000002_0 was 0
> 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
>
>
> *And here is my TaskTracker #2 log:*
>
> 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = hdmaster/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1.
> Opening the listener on 50060
> 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 50060
> 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=54873
> 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost/127.0.0.1:54873
> 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tracker tracker_hdmaster:localhost/127.0.0.1:54873
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 54873: starting
> 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 54873: starting
> 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 54873: starting
> 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> thread: Map-events fetcher for all reduce tasks on
> tracker_hdmaster:localhost/127.0.0.1:54873
> 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> MemoryCalculatorPlugin :
> org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache
> created with max memory = 10485760
> 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 1 and trying to launch
> attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_1909679810 spawned.
> 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_-622388657 spawned.
> 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_1909679810 given task:
> attempt_201109280735_0002_m_000002_0
> 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_-622388657 given task:
> attempt_201109280735_0002_m_000003_0
> 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000003_0 1.0%
> 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000003_0 is done.
> 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000003_0 was 0
> 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 1
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000002_0 1.0%
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000002_0 is done.
> 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000002_0 was 0
> 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
> 2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/10.0.0.1:54311. Already tried 0 time(s).
> 2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/10.0.0.1:54311. Already tried 1 time(s).
> 2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> thread: Map-events fetcher for all reduce tasks on
> tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342
> 2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker: Using
> MemoryCalculatorPlugin :
> org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad
> 2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache
> created with max memory = 10485760
> 2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000006_0
> 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201109280735_0002_m_000006_0
> 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481
> 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_774270481 spawned.
> 2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_774270481 given task:
> attempt_201109280735_0002_m_000006_0
> 2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000006_0 0.0% setup
> 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000006_0 is done.
> 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000006_0 was 0
> 2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1
> 2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out
> in any of the configured local directories
> 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction
> (registerTask): attempt_201109280735_0002_m_000000_0 task's
> state:UNASSIGNED
> 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000000_0
> 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201109280735_0002_m_000000_0
> 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Received
> KillTaskAction for task: attempt_201109280735_0002_m_000006_0
> 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About to
> purge task: attempt_201109280735_0002_m_000006_0
> 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_201109280735_0002_m_000006_0 done; removing files.
> 2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID
> attempt_201109280735_0002_m_000006_0 not found in cache
> 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000001_0
> 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 1 and trying to launch
> attempt_201109280735_0002_m_000001_0
> 2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571
> 2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_-1878969571 spawned.
> 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930
> 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_-364970930 spawned.
> 2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_-364970930 given task:
> attempt_201109280735_0002_m_000001_0
> 2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_-1878969571 given task:
> attempt_201109280735_0002_m_000000_0
> 2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000000_0 1.0%
> 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000000_0 is done.
> 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000000_0 was 0
> 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 1
> 2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1
> 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_m_000004_0
> 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 1 and trying to launch
> attempt_201109280735_0002_m_000004_0
> 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0
> task's
> state:UNASSIGNED
> 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_201109280735_0002_r_000000_0
> 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201109280735_0002_r_000000_0
> 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236
> 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_m_531367236 spawned.
> 2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481
> 2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_201109280735_0002_r_774270481 spawned.
> 2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_r_774270481 given task:
> attempt_201109280735_0002_r_000000_0
> 2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201109280735_0002_m_531367236 given task:
> attempt_201109280735_0002_m_000004_0
> 2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000001_0 1.0%
> 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000001_0 is done.
> 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_201109280735_0002_m_000001_0 was 0
> 2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 1
> 2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1
> 2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_m_000004_0 1.0%
> 2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO
> org.apache.hadoop.mapred.TaskTracker: reported output size for
> attempt_201109280735_0002_m_000004_0 was 0
> 2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1
> 2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
> 466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0
> given 466814/466810
> 2011-09-28 07:37:47,276 INFO
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID:
> attempt_201109280735_0002_m_000000_0
> 2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
> 267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0
> given 267013/267009
> 2011-09-28 07:37:47,287 INFO
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID:
> attempt_201109280735_0002_m_000001_0
> 2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> 0.12 MB/s) >
> 2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> 0.12 MB/s) >
> 2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
> 163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0
> given 163388/163384
> 2011-09-28 07:37:51,176 INFO
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID:
> attempt_201109280735_0002_m_000004_0
> 2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >2011-09-28 07:38:42,150 INFO
> org.apache.hadoop.mapred.TaskTracker: attempt_201109280735_0002_r_000000_0
> 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> 2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0
> 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> 2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> 0.09 MB/s) >
> 2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker:
>
>
> *Great thanks hoping you could help me.*
>
>
> On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k <de...@huawei.com> wrote:
>
> > Hi Bharath,
> >
> > There are few reasons to cause this problem. I have listed below some
> > reasons with solutions. This might help you to solve this. If you post
> the
> > logs, the problem can be figured out.
> >
> >
> > Reason 1:
> > It could be that the mapping in the /etc/hosts file is not present.
> > The DNS server is down as a result of which the hostnames cannot be
> > resolved.
> > The DNS server is in-correctly configured.
> >
> > Solution: Setting the slave.host.name property can be one solution.
> > Appropriate changes need to be done based on the problem.
> >
> >
> > Reason 2: If the map outputs are larger, we may get
> > java.lang.OutOfMemoryError: Java heap space. Because of this there are
> too
> > many fetch failures.
> >
> > Solution: The error, java.lang.OutOfMemoryError: Java heap space in task
> > tracker logs can be solved by any of the following methods:
> >
> > By decreasing the value configured for
> > mapred.job.shuffle.input.buffer.percent.
> > By increasing the heap memory of child JVM options for the property
> > mapred.child.java.opts.
> >
> >
> >
> >
> > Thanks
> > Devaraj
> >
> > ________________________________________
> > From: bharath vissapragada [bharathvissapragada1990@gmail.com]
> > Sent: Monday, September 26, 2011 8:54 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: Too many fetch failures. Help!
> >
> > Hey,
> >
> > Try configuring your cluster with hostnames instead of ips and add
> > those entries to /etc/hosts and sync it across all the nodes in the
> > cluster. You need to restart the cluster after making these changes.
> >
> > Hope this helps,
> >
> > On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <abdoukamel@gmail.com
> >
> > wrote:
> > > Hi,
> > > This is my first post here.
> > > I'm new to Hadoop.
> > > I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
> > > slave and the other is only slave).
> > > When I run a Wordcount example on 5 small txt files, the process never
> > > completes and I get a "Too many fetch failures" error on my terminal.
> > > If you can help me, I cant post my terminal's output and any log files
> > > needed.
> > > Great thanks.
> > >
> > > --
> > > Abdelrahman Kamel
> > >
> >
> >
> >
> > --
> > Regards,
> > Bharath .V
> > w:http://researchweb.iiit.ac.in/~bharath.v
> >
>
>
>
> --
> Abdelrahman Kamel
>
Re: Too many fetch failures. Help!
Posted by Abdelrahman Kamel <ab...@gmail.com>.
Thanks very much for all your fast replies.
*Here is my terminal output:*
hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar hadoop-0.20.2-examples.jar
wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to process :
5
11/09/28 07:38:23 INFO mapred.JobClient: Running job: job_201109280735_0002
11/09/28 07:38:24 INFO mapred.JobClient: map 0% reduce 0%
11/09/28 07:38:42 INFO mapred.JobClient: map 20% reduce 0%
11/09/28 07:38:44 INFO mapred.JobClient: map 40% reduce 0%
11/09/28 07:38:45 INFO mapred.JobClient: map 60% reduce 0%
11/09/28 07:38:47 INFO mapred.JobClient: map 80% reduce 0%
11/09/28 07:38:51 INFO mapred.JobClient: map 100% reduce 0%
11/09/28 07:38:54 INFO mapred.JobClient: map 100% reduce 13%
11/09/28 07:39:01 INFO mapred.JobClient: map 100% reduce 20%
*The terminal is stuck here.*
*And here is my JobTracker log:*
2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = hdmaster/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=54311
2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50030
2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:53,816 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 54311
2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
Safe mode will be turned off automatically.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
Safe mode will be turned off automatically.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 25 seconds.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 15 seconds.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 5 seconds.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:37:10,107 INFO
org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
inactive
2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54311: starting
2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 54311: starting
2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker: Starting
RUNNING
2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 54311: starting
2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 54311: starting
2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 54311: starting
2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/hdmaster
2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201109280735_0002
2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201109280735_0002
2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress: Input
size for job job_201109280735_0002 = 3430958. Number of splits = 5
2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000000 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000000 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000001 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000001 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000002 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000002 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000003 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000003 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000004 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000004 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000006_0' to tip
task_201109280735_0002_m_000006, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000006_0' has completed
task_201109280735_0002_m_000006 successfully.
2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000000_0' to tip
task_201109280735_0002_m_000000, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000000
2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000001_0' to tip
task_201109280735_0002_m_000001, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000001
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000002_0' to tip
task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/
127.0.0.1:54873'
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000002
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000003_0' to tip
task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/
127.0.0.1:54873'
2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000003
2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000000_0' has completed
task_201109280735_0002_m_000000 successfully.
2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:1 completedMapsInputSize:1423802
completedMapsOutputSize:466814
2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000004_0' to tip
task_201109280735_0002_m_000004, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000004
2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_r_000000_0' to tip
task_201109280735_0002_r_000000, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000003_0' has completed
task_201109280735_0002_m_000003 successfully.
2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:2 completedMapsInputSize:1817766
completedMapsOutputSize:719434
2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000001_0' has completed
task_201109280735_0002_m_000001 successfully.
2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:3 completedMapsInputSize:2492333
completedMapsOutputSize:986447
2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000002_0' has completed
task_201109280735_0002_m_000002 successfully.
2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:4 completedMapsInputSize:3087267
completedMapsOutputSize:1204101
2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000004_0' has completed
task_201109280735_0002_m_000004 successfully.
2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:5 completedMapsInputSize:3430963
completedMapsOutputSize:1367489
2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #1 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #2 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #3 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too
many fetch-failures for output of task: attempt_201109280735_0002_m_000003_0
... killing it
2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from attempt_201109280735_0002_m_000003_0: Too many fetch-failures
2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000003_1' to tip
task_201109280735_0002_m_000003, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000003
2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed
completed task 'attempt_201109280735_0002_m_000003_0' from
'tracker_hdmaster:localhost/127.0.0.1:54873'
2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000003_1' has completed
task_201109280735_0002_m_000003 successfully.
2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:6 completedMapsInputSize:3824927
completedMapsOutputSize:1620109
*And here is my TaskTracker #1 log:*
2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = hdmaster/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=54873
2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost/127.0.0.1:54873
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54873: starting
2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54873: starting
2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_1909679810 spawned.
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-622388657 spawned.
2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_1909679810 given task:
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-622388657 given task:
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000003_0 1.0%
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000003_0 is done.
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000003_0 was 0
2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000002_0 1.0%
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000002_0 is done.
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000002_0 was 0
2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
*And here is my TaskTracker #2 log:*
2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = hdmaster/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=54873
2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost/127.0.0.1:54873
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54873: starting
2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54873: starting
2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_1909679810 spawned.
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-622388657 spawned.
2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_1909679810 given task:
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-622388657 given task:
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000003_0 1.0%
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000003_0 is done.
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000003_0 was 0
2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000002_0 1.0%
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000002_0 is done.
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000002_0 was 0
2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/10.0.0.1:54311. Already tried 0 time(s).
2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/10.0.0.1:54311. Already tried 1 time(s).
2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342
2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker: Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad
2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0 task's
state:UNASSIGNED
2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481
2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_774270481 spawned.
2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_774270481 given task:
attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000006_0 0.0% setup
2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000006_0 is done.
2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000006_0 was 0
2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1
2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out
in any of the configured local directories
2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction
(registerTask): attempt_201109280735_0002_m_000000_0 task's state:UNASSIGNED
2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0 task's
state:UNASSIGNED
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About to
purge task: attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201109280735_0002_m_000006_0 done; removing files.
2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID
attempt_201109280735_0002_m_000006_0 not found in cache
2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571
2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-1878969571 spawned.
2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930
2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-364970930 spawned.
2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-364970930 given task:
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-1878969571 given task:
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000000_0 1.0%
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000000_0 is done.
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000000_0 was 0
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1
2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0 task's
state:UNASSIGNED
2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0 task's
state:UNASSIGNED
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236
2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_531367236 spawned.
2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481
2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_r_774270481 spawned.
2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_r_774270481 given task:
attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_531367236 given task:
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000001_0 1.0%
2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000001_0 is done.
2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000001_0 was 0
2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1
2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000004_0 1.0%
2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO
org.apache.hadoop.mapred.TaskTracker: reported output size for
attempt_201109280735_0002_m_000004_0 was 0
2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1
2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0
given 466814/466810
2011-09-28 07:37:47,276 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0
given 267013/267009
2011-09-28 07:37:47,287 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
0.12 MB/s) >
2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
0.12 MB/s) >
2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0
given 163388/163384
2011-09-28 07:37:51,176 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >2011-09-28 07:38:42,150 INFO
org.apache.hadoop.mapred.TaskTracker: attempt_201109280735_0002_r_000000_0
0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0
0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker:
*Great thanks hoping you could help me.*
On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k <de...@huawei.com> wrote:
> Hi Bharath,
>
> There are few reasons to cause this problem. I have listed below some
> reasons with solutions. This might help you to solve this. If you post the
> logs, the problem can be figured out.
>
>
> Reason 1:
> It could be that the mapping in the /etc/hosts file is not present.
> The DNS server is down as a result of which the hostnames cannot be
> resolved.
> The DNS server is in-correctly configured.
>
> Solution: Setting the slave.host.name property can be one solution.
> Appropriate changes need to be done based on the problem.
>
>
> Reason 2: If the map outputs are larger, we may get
> java.lang.OutOfMemoryError: Java heap space. Because of this there are too
> many fetch failures.
>
> Solution: The error, java.lang.OutOfMemoryError: Java heap space in task
> tracker logs can be solved by any of the following methods:
>
> By decreasing the value configured for
> mapred.job.shuffle.input.buffer.percent.
> By increasing the heap memory of child JVM options for the property
> mapred.child.java.opts.
>
>
>
>
> Thanks
> Devaraj
>
> ________________________________________
> From: bharath vissapragada [bharathvissapragada1990@gmail.com]
> Sent: Monday, September 26, 2011 8:54 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Too many fetch failures. Help!
>
> Hey,
>
> Try configuring your cluster with hostnames instead of ips and add
> those entries to /etc/hosts and sync it across all the nodes in the
> cluster. You need to restart the cluster after making these changes.
>
> Hope this helps,
>
> On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <ab...@gmail.com>
> wrote:
> > Hi,
> > This is my first post here.
> > I'm new to Hadoop.
> > I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
> > slave and the other is only slave).
> > When I run a Wordcount example on 5 small txt files, the process never
> > completes and I get a "Too many fetch failures" error on my terminal.
> > If you can help me, I cant post my terminal's output and any log files
> > needed.
> > Great thanks.
> >
> > --
> > Abdelrahman Kamel
> >
>
>
>
> --
> Regards,
> Bharath .V
> w:http://researchweb.iiit.ac.in/~bharath.v
>
--
Abdelrahman Kamel
RE: Too many fetch failures. Help!
Posted by Devaraj k <de...@huawei.com>.
Hi Bharath,
There are few reasons to cause this problem. I have listed below some reasons with solutions. This might help you to solve this. If you post the logs, the problem can be figured out.
Reason 1:
It could be that the mapping in the /etc/hosts file is not present.
The DNS server is down as a result of which the hostnames cannot be resolved.
The DNS server is in-correctly configured.
Solution: Setting the slave.host.name property can be one solution. Appropriate changes need to be done based on the problem.
Reason 2: If the map outputs are larger, we may get java.lang.OutOfMemoryError: Java heap space. Because of this there are too many fetch failures.
Solution: The error, java.lang.OutOfMemoryError: Java heap space in task tracker logs can be solved by any of the following methods:
By decreasing the value configured for mapred.job.shuffle.input.buffer.percent.
By increasing the heap memory of child JVM options for the property mapred.child.java.opts.
Thanks
Devaraj
________________________________________
From: bharath vissapragada [bharathvissapragada1990@gmail.com]
Sent: Monday, September 26, 2011 8:54 PM
To: common-user@hadoop.apache.org
Subject: Re: Too many fetch failures. Help!
Hey,
Try configuring your cluster with hostnames instead of ips and add
those entries to /etc/hosts and sync it across all the nodes in the
cluster. You need to restart the cluster after making these changes.
Hope this helps,
On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <ab...@gmail.com> wrote:
> Hi,
> This is my first post here.
> I'm new to Hadoop.
> I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
> slave and the other is only slave).
> When I run a Wordcount example on 5 small txt files, the process never
> completes and I get a "Too many fetch failures" error on my terminal.
> If you can help me, I cant post my terminal's output and any log files
> needed.
> Great thanks.
>
> --
> Abdelrahman Kamel
>
--
Regards,
Bharath .V
w:http://researchweb.iiit.ac.in/~bharath.v
Re: Too many fetch failures. Help!
Posted by bharath vissapragada <bh...@gmail.com>.
Hey,
Try configuring your cluster with hostnames instead of ips and add
those entries to /etc/hosts and sync it across all the nodes in the
cluster. You need to restart the cluster after making these changes.
Hope this helps,
On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <ab...@gmail.com> wrote:
> Hi,
> This is my first post here.
> I'm new to Hadoop.
> I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
> slave and the other is only slave).
> When I run a Wordcount example on 5 small txt files, the process never
> completes and I get a "Too many fetch failures" error on my terminal.
> If you can help me, I cant post my terminal's output and any log files
> needed.
> Great thanks.
>
> --
> Abdelrahman Kamel
>
--
Regards,
Bharath .V
w:http://researchweb.iiit.ac.in/~bharath.v
Re: Too many fetch failures. Help!
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
Hello Abdelrahman,
Are you able to ping from one machine to other with the configured hostname?
configure both the hostnames in /etc/hosts file properly and try.
Regards,
Uma
----- Original Message -----
From: Abdelrahman Kamel <ab...@gmail.com>
Date: Monday, September 26, 2011 8:47 pm
Subject: Too many fetch failures. Help!
To: common-user@hadoop.apache.org
> Hi,
> This is my first post here.
> I'm new to Hadoop.
> I've already installed Hadoop on 2 Ubuntu boxes (one is both master
> andslave and the other is only slave).
> When I run a Wordcount example on 5 small txt files, the process never
> completes and I get a "Too many fetch failures" error on my terminal.
> If you can help me, I cant post my terminal's output and any log files
> needed.
> Great thanks.
>
> --
> Abdelrahman Kamel
>