You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Adarsh Sharma <ad...@orkash.com> on 2011/01/07 13:47:45 UTC
Too-many fetch failure Reduce Error
Dear all,
I am researching about the below error and could not able to find the
reason :
Data Size : 3.4 GB
Hadoop-0.20.0
hadoop@ws32-test-lin:~/project/hadoop-0.20.2$ bin/hadoop jar
hadoop-0.20.2-examples.jar wordcount /user/hadoop/page_content.txt
page_content_output.txt
11/01/07 16:11:14 INFO input.FileInputFormat: Total input paths to
process : 1
11/01/07 16:11:15 INFO mapred.JobClient: Running job: job_201101071129_0001
11/01/07 16:11:16 INFO mapred.JobClient: map 0% reduce 0%
11/01/07 16:11:41 INFO mapred.JobClient: map 1% reduce 0%
11/01/07 16:11:45 INFO mapred.JobClient: map 2% reduce 0%
11/01/07 16:11:48 INFO mapred.JobClient: map 3% reduce 0%
11/01/07 16:11:52 INFO mapred.JobClient: map 4% reduce 0%
11/01/07 16:11:56 INFO mapred.JobClient: map 5% reduce 0%
11/01/07 16:12:00 INFO mapred.JobClient: map 6% reduce 0%
11/01/07 16:12:05 INFO mapred.JobClient: map 7% reduce 0%
11/01/07 16:12:08 INFO mapred.JobClient: map 8% reduce 0%
11/01/07 16:12:11 INFO mapred.JobClient: map 9% reduce 0%
11/01/07 16:12:14 INFO mapred.JobClient: map 10% reduce 0%
11/01/07 16:12:17 INFO mapred.JobClient: map 11% reduce 0%
11/01/07 16:12:21 INFO mapred.JobClient: map 12% reduce 0%
11/01/07 16:12:24 INFO mapred.JobClient: map 13% reduce 0%
11/01/07 16:12:27 INFO mapred.JobClient: map 14% reduce 0%
11/01/07 16:12:30 INFO mapred.JobClient: map 15% reduce 0%
11/01/07 16:12:33 INFO mapred.JobClient: map 16% reduce 0%
11/01/07 16:12:36 INFO mapred.JobClient: map 17% reduce 0%
11/01/07 16:12:40 INFO mapred.JobClient: map 18% reduce 0%
11/01/07 16:12:45 INFO mapred.JobClient: map 19% reduce 0%
11/01/07 16:12:48 INFO mapred.JobClient: map 20% reduce 0%
11/01/07 16:12:54 INFO mapred.JobClient: map 21% reduce 0%
11/01/07 16:13:00 INFO mapred.JobClient: map 22% reduce 0%
11/01/07 16:13:04 INFO mapred.JobClient: map 22% reduce 1%
11/01/07 16:13:13 INFO mapred.JobClient: map 23% reduce 1%
11/01/07 16:13:19 INFO mapred.JobClient: map 24% reduce 1%
11/01/07 16:13:25 INFO mapred.JobClient: map 25% reduce 1%
11/01/07 16:13:30 INFO mapred.JobClient: map 26% reduce 1%
11/01/07 16:13:34 INFO mapred.JobClient: map 26% reduce 3%
11/01/07 16:13:36 INFO mapred.JobClient: map 27% reduce 3%
11/01/07 16:13:37 INFO mapred.JobClient: map 27% reduce 4%
11/01/07 16:13:39 INFO mapred.JobClient: map 28% reduce 4%
11/01/07 16:13:43 INFO mapred.JobClient: map 29% reduce 4%
11/01/07 16:13:46 INFO mapred.JobClient: map 30% reduce 4%
11/01/07 16:13:49 INFO mapred.JobClient: map 31% reduce 4%
11/01/07 16:13:52 INFO mapred.JobClient: map 32% reduce 4%
11/01/07 16:13:55 INFO mapred.JobClient: map 33% reduce 4%
11/01/07 16:13:58 INFO mapred.JobClient: map 34% reduce 4%
11/01/07 16:14:02 INFO mapred.JobClient: map 35% reduce 4%
11/01/07 16:14:05 INFO mapred.JobClient: map 36% reduce 4%
11/01/07 16:14:08 INFO mapred.JobClient: map 37% reduce 4%
11/01/07 16:14:11 INFO mapred.JobClient: map 38% reduce 4%
11/01/07 16:14:15 INFO mapred.JobClient: map 39% reduce 4%
11/01/07 16:14:19 INFO mapred.JobClient: map 40% reduce 4%
11/01/07 16:14:20 INFO mapred.JobClient: map 40% reduce 5%
11/01/07 16:14:25 INFO mapred.JobClient: map 41% reduce 5%
11/01/07 16:14:32 INFO mapred.JobClient: map 42% reduce 5%
11/01/07 16:14:38 INFO mapred.JobClient: map 43% reduce 5%
11/01/07 16:14:41 INFO mapred.JobClient: map 43% reduce 6%
11/01/07 16:14:43 INFO mapred.JobClient: map 44% reduce 6%
11/01/07 16:14:47 INFO mapred.JobClient: map 45% reduce 6%
11/01/07 16:14:50 INFO mapred.JobClient: map 46% reduce 6%
11/01/07 16:14:54 INFO mapred.JobClient: map 47% reduce 7%
11/01/07 16:14:59 INFO mapred.JobClient: map 48% reduce 7%
11/01/07 16:15:02 INFO mapred.JobClient: map 49% reduce 7%
11/01/07 16:15:05 INFO mapred.JobClient: map 50% reduce 7%
11/01/07 16:15:11 INFO mapred.JobClient: map 51% reduce 7%
11/01/07 16:15:14 INFO mapred.JobClient: map 52% reduce 7%
11/01/07 16:15:16 INFO mapred.JobClient: map 52% reduce 8%
11/01/07 16:15:20 INFO mapred.JobClient: map 53% reduce 8%
11/01/07 16:15:25 INFO mapred.JobClient: map 54% reduce 8%
11/01/07 16:15:29 INFO mapred.JobClient: map 55% reduce 8%
11/01/07 16:15:31 INFO mapred.JobClient: map 55% reduce 9%
11/01/07 16:15:33 INFO mapred.JobClient: map 56% reduce 9%
11/01/07 16:15:38 INFO mapred.JobClient: map 57% reduce 9%
11/01/07 16:15:42 INFO mapred.JobClient: map 58% reduce 9%
11/01/07 16:15:43 INFO mapred.JobClient: map 58% reduce 10%
11/01/07 16:15:46 INFO mapred.JobClient: map 59% reduce 10%
11/01/07 16:15:49 INFO mapred.JobClient: map 60% reduce 10%
11/01/07 16:15:53 INFO mapred.JobClient: map 61% reduce 10%
11/01/07 16:15:56 INFO mapred.JobClient: map 62% reduce 10%
11/01/07 16:16:00 INFO mapred.JobClient: map 63% reduce 10%
11/01/07 16:16:06 INFO mapred.JobClient: map 64% reduce 10%
11/01/07 16:16:10 INFO mapred.JobClient: map 65% reduce 10%
11/01/07 16:16:15 INFO mapred.JobClient: map 66% reduce 10%
11/01/07 16:16:18 INFO mapred.JobClient: map 67% reduce 10%
11/01/07 16:16:19 INFO mapred.JobClient: map 67% reduce 12%
11/01/07 16:16:21 INFO mapred.JobClient: map 68% reduce 12%
11/01/07 16:16:25 INFO mapred.JobClient: map 69% reduce 12%
11/01/07 16:16:28 INFO mapred.JobClient: map 70% reduce 12%
11/01/07 16:16:31 INFO mapred.JobClient: map 71% reduce 12%
11/01/07 16:16:35 INFO mapred.JobClient: map 72% reduce 12%
11/01/07 16:16:38 INFO mapred.JobClient: map 73% reduce 12%
11/01/07 16:16:43 INFO mapred.JobClient: map 74% reduce 12%
11/01/07 16:16:47 INFO mapred.JobClient: map 75% reduce 12%
11/01/07 16:16:53 INFO mapred.JobClient: map 75% reduce 13%
11/01/07 16:16:54 INFO mapred.JobClient: map 76% reduce 13%
11/01/07 16:17:01 INFO mapred.JobClient: map 77% reduce 13%
11/01/07 16:17:03 INFO mapred.JobClient: map 77% reduce 14%
11/01/07 16:17:06 INFO mapred.JobClient: map 78% reduce 15%
11/01/07 16:17:11 INFO mapred.JobClient: map 79% reduce 15%
11/01/07 16:17:15 INFO mapred.JobClient: map 80% reduce 15%
11/01/07 16:17:23 INFO mapred.JobClient: map 81% reduce 15%
11/01/07 16:17:27 INFO mapred.JobClient: map 82% reduce 15%
11/01/07 16:17:31 INFO mapred.JobClient: map 83% reduce 15%
11/01/07 16:17:33 INFO mapred.JobClient: map 84% reduce 15%
11/01/07 16:17:38 INFO mapred.JobClient: map 85% reduce 15%
11/01/07 16:17:41 INFO mapred.JobClient: map 86% reduce 15%
11/01/07 16:17:45 INFO mapred.JobClient: map 87% reduce 15%
11/01/07 16:17:50 INFO mapred.JobClient: map 88% reduce 15%
11/01/07 16:17:54 INFO mapred.JobClient: map 89% reduce 15%
11/01/07 16:18:00 INFO mapred.JobClient: map 90% reduce 15%
11/01/07 16:18:02 INFO mapred.JobClient: map 90% reduce 16%
11/01/07 16:18:03 INFO mapred.JobClient: map 91% reduce 16%
11/01/07 16:18:06 INFO mapred.JobClient: map 92% reduce 16%
11/01/07 16:18:09 INFO mapred.JobClient: map 93% reduce 16%
11/01/07 16:18:14 INFO mapred.JobClient: map 94% reduce 16%
11/01/07 16:18:17 INFO mapred.JobClient: map 95% reduce 16%
11/01/07 16:18:21 INFO mapred.JobClient: map 96% reduce 16%
11/01/07 16:18:27 INFO mapred.JobClient: map 97% reduce 16%
11/01/07 16:18:29 INFO mapred.JobClient: map 97% reduce 17%
11/01/07 16:18:35 INFO mapred.JobClient: map 98% reduce 17%
11/01/07 16:18:48 INFO mapred.JobClient: map 99% reduce 18%
11/01/07 16:18:55 INFO mapred.JobClient: map 99% reduce 19%
11/01/07 16:19:12 INFO mapred.JobClient: map 100% reduce 19%
11/01/07 16:19:13 INFO mapred.JobClient: map 100% reduce 20%
11/01/07 16:19:34 INFO mapred.JobClient: map 100% reduce 21%
11/01/07 16:22:54 INFO mapred.JobClient: Task Id :
attempt_201101071129_0001_m_000012_0, Status : FAILED
Too many fetch-failures
11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stdout
11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stderr
11/01/07 16:22:58 INFO mapred.JobClient: map 98% reduce 21%
11/01/07 16:23:16 INFO mapred.JobClient: map 99% reduce 21%
11/01/07 16:23:34 INFO mapred.JobClient: map 100% reduce 21%
11/01/07 16:25:28 INFO mapred.JobClient: map 100% reduce 23%
11/01/07 16:35:17 INFO mapred.JobClient: Task Id :
attempt_201101071129_0001_m_000011_0, Status : FAILED
Too many fetch-failures
11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stdout
11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stderr
11/01/07 16:35:21 INFO mapred.JobClient: map 98% reduce 23%
11/01/07 16:35:39 INFO mapred.JobClient: map 99% reduce 23%
11/01/07 16:35:54 INFO mapred.JobClient: map 100% reduce 23%
11/01/07 16:37:49 INFO mapred.JobClient: map 100% reduce 24%
11/01/07 16:47:32 INFO mapred.JobClient: Task Id :
attempt_201101071129_0001_m_000015_0, Status : FAILED
Too many fetch-failures
11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stdout
11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stderr
11/01/07 16:47:36 INFO mapred.JobClient: map 98% reduce 24%
11/01/07 16:47:54 INFO mapred.JobClient: map 99% reduce 24%
11/01/07 16:48:12 INFO mapred.JobClient: map 100% reduce 24%
11/01/07 16:50:03 INFO mapred.JobClient: map 100% reduce 25%
11/01/07 16:59:48 INFO mapred.JobClient: Task Id :
attempt_201101071129_0001_m_000019_0, Status : FAILED
Too many fetch-failures
11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stdout
11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stderr
11/01/07 16:59:52 INFO mapred.JobClient: map 98% reduce 25%
11/01/07 17:00:10 INFO mapred.JobClient: map 99% reduce 25%
11/01/07 17:00:26 INFO mapred.JobClient: map 100% reduce 25%
11/01/07 17:00:38 INFO mapred.JobClient: map 100% reduce 26%
Would anyone Please send me the reason as I am interested to find the
cause to clear my understanding.
I configured properly Hadoop on 2 clusters. I am sure there is no
configuration problem because in my other cluster it runs fine.
One Cluster is on Standalone Servers ( 4 nodes ). Job is executed
successfully.
Other Cluster is on VM's (Cloud) .
My tasktracker log says this
2011-01-07 16:20:23,234 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
in any of the configured local directories
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2011-01-07 16:20:23,236 WARN org.apache.hadoop.mapred.TaskTracker:
Unknown child with bad map output: attempt_201101071129_0001_m_000012_0.
Ignored.
2011-01-07 16:20:23,239 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 172.16.1.3:50060,
dest: 172.16.1.5:47135, bytes: 0, op: MAPRED_SHUFFLE, cliID:
attempt_201101071129_0001_m_000012_0
2011-01-07 16:20:23,239 WARN org.mortbay.log: /mapOutput:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
in any of the configured local directories
2011-01-07 16:22:53,266 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
in any of the configured local directories
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Let's have some discussion.
Thanks & Regards
Adarsh Sharma
Re: Too-many fetch failure Reduce Error
Posted by Adarsh Sharma <ad...@orkash.com>.
Any update on this error.
Thanks
Adarsh Sharma wrote:
> Esteban Gutierrez Moguel wrote:
>> Adarsh,
>>
>> Dou you have in /etc/hosts the hostnames for masters and slaves?
>>
>
> Yes I know this issue. But did you think the error occurs while
> reading the output of map.
> I want to know the proper reason of below lines :
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>
>
>
>> esteban.
>>
>> On Fri, Jan 7, 2011 at 06:47, Adarsh Sharma
>> <ad...@orkash.com>wrote:
>>
>>
>>> Dear all,
>>>
>>> I am researching about the below error and could not able to find the
>>> reason :
>>>
>>> Data Size : 3.4 GB
>>> Hadoop-0.20.0
>>>
>>> hadoop@ws32-test-lin:~/project/hadoop-0.20.2$ bin/hadoop jar
>>> hadoop-0.20.2-examples.jar wordcount /user/hadoop/page_content.txt
>>> page_content_output.txt
>>> 11/01/07 16:11:14 INFO input.FileInputFormat: Total input paths to
>>> process
>>> : 1
>>> 11/01/07 16:11:15 INFO mapred.JobClient: Running job:
>>> job_201101071129_0001
>>> 11/01/07 16:11:16 INFO mapred.JobClient: map 0% reduce 0%
>>> 11/01/07 16:11:41 INFO mapred.JobClient: map 1% reduce 0%
>>> 11/01/07 16:11:45 INFO mapred.JobClient: map 2% reduce 0%
>>> 11/01/07 16:11:48 INFO mapred.JobClient: map 3% reduce 0%
>>> 11/01/07 16:11:52 INFO mapred.JobClient: map 4% reduce 0%
>>> 11/01/07 16:11:56 INFO mapred.JobClient: map 5% reduce 0%
>>> 11/01/07 16:12:00 INFO mapred.JobClient: map 6% reduce 0%
>>> 11/01/07 16:12:05 INFO mapred.JobClient: map 7% reduce 0%
>>> 11/01/07 16:12:08 INFO mapred.JobClient: map 8% reduce 0%
>>> 11/01/07 16:12:11 INFO mapred.JobClient: map 9% reduce 0%
>>> 11/01/07 16:12:14 INFO mapred.JobClient: map 10% reduce 0%
>>> 11/01/07 16:12:17 INFO mapred.JobClient: map 11% reduce 0%
>>> 11/01/07 16:12:21 INFO mapred.JobClient: map 12% reduce 0%
>>> 11/01/07 16:12:24 INFO mapred.JobClient: map 13% reduce 0%
>>> 11/01/07 16:12:27 INFO mapred.JobClient: map 14% reduce 0%
>>> 11/01/07 16:12:30 INFO mapred.JobClient: map 15% reduce 0%
>>> 11/01/07 16:12:33 INFO mapred.JobClient: map 16% reduce 0%
>>> 11/01/07 16:12:36 INFO mapred.JobClient: map 17% reduce 0%
>>> 11/01/07 16:12:40 INFO mapred.JobClient: map 18% reduce 0%
>>> 11/01/07 16:12:45 INFO mapred.JobClient: map 19% reduce 0%
>>> 11/01/07 16:12:48 INFO mapred.JobClient: map 20% reduce 0%
>>> 11/01/07 16:12:54 INFO mapred.JobClient: map 21% reduce 0%
>>> 11/01/07 16:13:00 INFO mapred.JobClient: map 22% reduce 0%
>>> 11/01/07 16:13:04 INFO mapred.JobClient: map 22% reduce 1%
>>> 11/01/07 16:13:13 INFO mapred.JobClient: map 23% reduce 1%
>>> 11/01/07 16:13:19 INFO mapred.JobClient: map 24% reduce 1%
>>> 11/01/07 16:13:25 INFO mapred.JobClient: map 25% reduce 1%
>>> 11/01/07 16:13:30 INFO mapred.JobClient: map 26% reduce 1%
>>> 11/01/07 16:13:34 INFO mapred.JobClient: map 26% reduce 3%
>>> 11/01/07 16:13:36 INFO mapred.JobClient: map 27% reduce 3%
>>> 11/01/07 16:13:37 INFO mapred.JobClient: map 27% reduce 4%
>>> 11/01/07 16:13:39 INFO mapred.JobClient: map 28% reduce 4%
>>> 11/01/07 16:13:43 INFO mapred.JobClient: map 29% reduce 4%
>>> 11/01/07 16:13:46 INFO mapred.JobClient: map 30% reduce 4%
>>> 11/01/07 16:13:49 INFO mapred.JobClient: map 31% reduce 4%
>>> 11/01/07 16:13:52 INFO mapred.JobClient: map 32% reduce 4%
>>> 11/01/07 16:13:55 INFO mapred.JobClient: map 33% reduce 4%
>>> 11/01/07 16:13:58 INFO mapred.JobClient: map 34% reduce 4%
>>> 11/01/07 16:14:02 INFO mapred.JobClient: map 35% reduce 4%
>>> 11/01/07 16:14:05 INFO mapred.JobClient: map 36% reduce 4%
>>> 11/01/07 16:14:08 INFO mapred.JobClient: map 37% reduce 4%
>>> 11/01/07 16:14:11 INFO mapred.JobClient: map 38% reduce 4%
>>> 11/01/07 16:14:15 INFO mapred.JobClient: map 39% reduce 4%
>>> 11/01/07 16:14:19 INFO mapred.JobClient: map 40% reduce 4%
>>> 11/01/07 16:14:20 INFO mapred.JobClient: map 40% reduce 5%
>>> 11/01/07 16:14:25 INFO mapred.JobClient: map 41% reduce 5%
>>> 11/01/07 16:14:32 INFO mapred.JobClient: map 42% reduce 5%
>>> 11/01/07 16:14:38 INFO mapred.JobClient: map 43% reduce 5%
>>> 11/01/07 16:14:41 INFO mapred.JobClient: map 43% reduce 6%
>>> 11/01/07 16:14:43 INFO mapred.JobClient: map 44% reduce 6%
>>> 11/01/07 16:14:47 INFO mapred.JobClient: map 45% reduce 6%
>>> 11/01/07 16:14:50 INFO mapred.JobClient: map 46% reduce 6%
>>> 11/01/07 16:14:54 INFO mapred.JobClient: map 47% reduce 7%
>>> 11/01/07 16:14:59 INFO mapred.JobClient: map 48% reduce 7%
>>> 11/01/07 16:15:02 INFO mapred.JobClient: map 49% reduce 7%
>>> 11/01/07 16:15:05 INFO mapred.JobClient: map 50% reduce 7%
>>> 11/01/07 16:15:11 INFO mapred.JobClient: map 51% reduce 7%
>>> 11/01/07 16:15:14 INFO mapred.JobClient: map 52% reduce 7%
>>> 11/01/07 16:15:16 INFO mapred.JobClient: map 52% reduce 8%
>>> 11/01/07 16:15:20 INFO mapred.JobClient: map 53% reduce 8%
>>> 11/01/07 16:15:25 INFO mapred.JobClient: map 54% reduce 8%
>>> 11/01/07 16:15:29 INFO mapred.JobClient: map 55% reduce 8%
>>> 11/01/07 16:15:31 INFO mapred.JobClient: map 55% reduce 9%
>>> 11/01/07 16:15:33 INFO mapred.JobClient: map 56% reduce 9%
>>> 11/01/07 16:15:38 INFO mapred.JobClient: map 57% reduce 9%
>>> 11/01/07 16:15:42 INFO mapred.JobClient: map 58% reduce 9%
>>> 11/01/07 16:15:43 INFO mapred.JobClient: map 58% reduce 10%
>>> 11/01/07 16:15:46 INFO mapred.JobClient: map 59% reduce 10%
>>> 11/01/07 16:15:49 INFO mapred.JobClient: map 60% reduce 10%
>>> 11/01/07 16:15:53 INFO mapred.JobClient: map 61% reduce 10%
>>> 11/01/07 16:15:56 INFO mapred.JobClient: map 62% reduce 10%
>>> 11/01/07 16:16:00 INFO mapred.JobClient: map 63% reduce 10%
>>> 11/01/07 16:16:06 INFO mapred.JobClient: map 64% reduce 10%
>>> 11/01/07 16:16:10 INFO mapred.JobClient: map 65% reduce 10%
>>> 11/01/07 16:16:15 INFO mapred.JobClient: map 66% reduce 10%
>>> 11/01/07 16:16:18 INFO mapred.JobClient: map 67% reduce 10%
>>> 11/01/07 16:16:19 INFO mapred.JobClient: map 67% reduce 12%
>>> 11/01/07 16:16:21 INFO mapred.JobClient: map 68% reduce 12%
>>> 11/01/07 16:16:25 INFO mapred.JobClient: map 69% reduce 12%
>>> 11/01/07 16:16:28 INFO mapred.JobClient: map 70% reduce 12%
>>> 11/01/07 16:16:31 INFO mapred.JobClient: map 71% reduce 12%
>>> 11/01/07 16:16:35 INFO mapred.JobClient: map 72% reduce 12%
>>> 11/01/07 16:16:38 INFO mapred.JobClient: map 73% reduce 12%
>>> 11/01/07 16:16:43 INFO mapred.JobClient: map 74% reduce 12%
>>> 11/01/07 16:16:47 INFO mapred.JobClient: map 75% reduce 12%
>>> 11/01/07 16:16:53 INFO mapred.JobClient: map 75% reduce 13%
>>> 11/01/07 16:16:54 INFO mapred.JobClient: map 76% reduce 13%
>>> 11/01/07 16:17:01 INFO mapred.JobClient: map 77% reduce 13%
>>> 11/01/07 16:17:03 INFO mapred.JobClient: map 77% reduce 14%
>>> 11/01/07 16:17:06 INFO mapred.JobClient: map 78% reduce 15%
>>> 11/01/07 16:17:11 INFO mapred.JobClient: map 79% reduce 15%
>>> 11/01/07 16:17:15 INFO mapred.JobClient: map 80% reduce 15%
>>> 11/01/07 16:17:23 INFO mapred.JobClient: map 81% reduce 15%
>>> 11/01/07 16:17:27 INFO mapred.JobClient: map 82% reduce 15%
>>> 11/01/07 16:17:31 INFO mapred.JobClient: map 83% reduce 15%
>>> 11/01/07 16:17:33 INFO mapred.JobClient: map 84% reduce 15%
>>> 11/01/07 16:17:38 INFO mapred.JobClient: map 85% reduce 15%
>>> 11/01/07 16:17:41 INFO mapred.JobClient: map 86% reduce 15%
>>> 11/01/07 16:17:45 INFO mapred.JobClient: map 87% reduce 15%
>>> 11/01/07 16:17:50 INFO mapred.JobClient: map 88% reduce 15%
>>> 11/01/07 16:17:54 INFO mapred.JobClient: map 89% reduce 15%
>>> 11/01/07 16:18:00 INFO mapred.JobClient: map 90% reduce 15%
>>> 11/01/07 16:18:02 INFO mapred.JobClient: map 90% reduce 16%
>>> 11/01/07 16:18:03 INFO mapred.JobClient: map 91% reduce 16%
>>> 11/01/07 16:18:06 INFO mapred.JobClient: map 92% reduce 16%
>>> 11/01/07 16:18:09 INFO mapred.JobClient: map 93% reduce 16%
>>> 11/01/07 16:18:14 INFO mapred.JobClient: map 94% reduce 16%
>>> 11/01/07 16:18:17 INFO mapred.JobClient: map 95% reduce 16%
>>> 11/01/07 16:18:21 INFO mapred.JobClient: map 96% reduce 16%
>>> 11/01/07 16:18:27 INFO mapred.JobClient: map 97% reduce 16%
>>> 11/01/07 16:18:29 INFO mapred.JobClient: map 97% reduce 17%
>>> 11/01/07 16:18:35 INFO mapred.JobClient: map 98% reduce 17%
>>> 11/01/07 16:18:48 INFO mapred.JobClient: map 99% reduce 18%
>>> 11/01/07 16:18:55 INFO mapred.JobClient: map 99% reduce 19%
>>> 11/01/07 16:19:12 INFO mapred.JobClient: map 100% reduce 19%
>>> 11/01/07 16:19:13 INFO mapred.JobClient: map 100% reduce 20%
>>> 11/01/07 16:19:34 INFO mapred.JobClient: map 100% reduce 21%
>>> 11/01/07 16:22:54 INFO mapred.JobClient: Task Id :
>>> attempt_201101071129_0001_m_000012_0, Status : FAILED
>>> Too many fetch-failures
>>> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stdout
>>>
>>> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stderr
>>>
>>> 11/01/07 16:22:58 INFO mapred.JobClient: map 98% reduce 21%
>>> 11/01/07 16:23:16 INFO mapred.JobClient: map 99% reduce 21%
>>> 11/01/07 16:23:34 INFO mapred.JobClient: map 100% reduce 21%
>>> 11/01/07 16:25:28 INFO mapred.JobClient: map 100% reduce 23%
>>> 11/01/07 16:35:17 INFO mapred.JobClient: Task Id :
>>> attempt_201101071129_0001_m_000011_0, Status : FAILED
>>> Too many fetch-failures
>>> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stdout
>>>
>>> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stderr
>>>
>>> 11/01/07 16:35:21 INFO mapred.JobClient: map 98% reduce 23%
>>> 11/01/07 16:35:39 INFO mapred.JobClient: map 99% reduce 23%
>>> 11/01/07 16:35:54 INFO mapred.JobClient: map 100% reduce 23%
>>> 11/01/07 16:37:49 INFO mapred.JobClient: map 100% reduce 24%
>>> 11/01/07 16:47:32 INFO mapred.JobClient: Task Id :
>>> attempt_201101071129_0001_m_000015_0, Status : FAILED
>>> Too many fetch-failures
>>> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stdout
>>>
>>> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stderr
>>>
>>> 11/01/07 16:47:36 INFO mapred.JobClient: map 98% reduce 24%
>>> 11/01/07 16:47:54 INFO mapred.JobClient: map 99% reduce 24%
>>> 11/01/07 16:48:12 INFO mapred.JobClient: map 100% reduce 24%
>>> 11/01/07 16:50:03 INFO mapred.JobClient: map 100% reduce 25%
>>> 11/01/07 16:59:48 INFO mapred.JobClient: Task Id :
>>> attempt_201101071129_0001_m_000019_0, Status : FAILED
>>> Too many fetch-failures
>>> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stdout
>>>
>>> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
>>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stderr
>>>
>>> 11/01/07 16:59:52 INFO mapred.JobClient: map 98% reduce 25%
>>> 11/01/07 17:00:10 INFO mapred.JobClient: map 99% reduce 25%
>>> 11/01/07 17:00:26 INFO mapred.JobClient: map 100% reduce 25%
>>> 11/01/07 17:00:38 INFO mapred.JobClient: map 100% reduce 26%
>>>
>>>
>>> Would anyone Please send me the reason as I am interested to find
>>> the cause
>>> to clear my understanding.
>>> I configured properly Hadoop on 2 clusters. I am sure there is no
>>> configuration problem because in my other cluster it runs fine.
>>>
>>> One Cluster is on Standalone Servers ( 4 nodes ). Job is executed
>>> successfully.
>>>
>>> Other Cluster is on VM's (Cloud) .
>>>
>>> My tasktracker log says this
>>>
>>> 2011-01-07 16:20:23,234 WARN org.apache.hadoop.mapred.TaskTracker:
>>> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
>>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>>>
>>> in any of the configured local directories
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>>>
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>>>
>>> at
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>>>
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> at
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>>>
>>> at
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>
>>> at
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>>
>>> at
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>
>>> at
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>>> at
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>
>>> at
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>
>>> at org.mortbay.jetty.Server.handle(Server.java:324)
>>> at
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>> at
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>>
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>>> at
>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>>> at
>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>>> at
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>>
>>> at
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>>
>>> 2011-01-07 16:20:23,236 WARN org.apache.hadoop.mapred.TaskTracker:
>>> Unknown
>>> child with bad map output: attempt_201101071129_0001_m_000012_0.
>>> Ignored.
>>> 2011-01-07 16:20:23,239 INFO
>>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
>>> 172.16.1.3:50060,
>>> dest: 172.16.1.5:47135, bytes: 0, op: MAPRED_SHUFFLE, cliID:
>>> attempt_201101071129_0001_m_000012_0
>>> 2011-01-07 16:20:23,239 WARN org.mortbay.log: /mapOutput:
>>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>>>
>>> in any of the configured local directories
>>> 2011-01-07 16:22:53,266 WARN org.apache.hadoop.mapred.TaskTracker:
>>> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
>>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>>>
>>> in any of the configured local directories
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>>>
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>>>
>>> at
>>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>>>
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> at
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>>>
>>> at
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>
>>> at
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>>
>>> at
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>
>>> at
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>>> at
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>
>>> at
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>
>>> at org.mortbay.jetty.Server.handle(Server.java:324)
>>> at
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>> at
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>>
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>>> at
>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>>> at
>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>>> at
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>>
>>> at
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>>
>>>
>>>
>>>
>>> Let's have some discussion.
>>>
>>>
>>> Thanks & Regards
>>>
>>> Adarsh Sharma
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Too-many fetch failure Reduce Error
Posted by Adarsh Sharma <ad...@orkash.com>.
Esteban Gutierrez Moguel wrote:
> Adarsh,
>
> Dou you have in /etc/hosts the hostnames for masters and slaves?
>
Yes I know this issue. But did you think the error occurs while reading
the output of map.
I want to know the proper reason of below lines :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
> esteban.
>
> On Fri, Jan 7, 2011 at 06:47, Adarsh Sharma <ad...@orkash.com>wrote:
>
>
>> Dear all,
>>
>> I am researching about the below error and could not able to find the
>> reason :
>>
>> Data Size : 3.4 GB
>> Hadoop-0.20.0
>>
>> hadoop@ws32-test-lin:~/project/hadoop-0.20.2$ bin/hadoop jar
>> hadoop-0.20.2-examples.jar wordcount /user/hadoop/page_content.txt
>> page_content_output.txt
>> 11/01/07 16:11:14 INFO input.FileInputFormat: Total input paths to process
>> : 1
>> 11/01/07 16:11:15 INFO mapred.JobClient: Running job: job_201101071129_0001
>> 11/01/07 16:11:16 INFO mapred.JobClient: map 0% reduce 0%
>> 11/01/07 16:11:41 INFO mapred.JobClient: map 1% reduce 0%
>> 11/01/07 16:11:45 INFO mapred.JobClient: map 2% reduce 0%
>> 11/01/07 16:11:48 INFO mapred.JobClient: map 3% reduce 0%
>> 11/01/07 16:11:52 INFO mapred.JobClient: map 4% reduce 0%
>> 11/01/07 16:11:56 INFO mapred.JobClient: map 5% reduce 0%
>> 11/01/07 16:12:00 INFO mapred.JobClient: map 6% reduce 0%
>> 11/01/07 16:12:05 INFO mapred.JobClient: map 7% reduce 0%
>> 11/01/07 16:12:08 INFO mapred.JobClient: map 8% reduce 0%
>> 11/01/07 16:12:11 INFO mapred.JobClient: map 9% reduce 0%
>> 11/01/07 16:12:14 INFO mapred.JobClient: map 10% reduce 0%
>> 11/01/07 16:12:17 INFO mapred.JobClient: map 11% reduce 0%
>> 11/01/07 16:12:21 INFO mapred.JobClient: map 12% reduce 0%
>> 11/01/07 16:12:24 INFO mapred.JobClient: map 13% reduce 0%
>> 11/01/07 16:12:27 INFO mapred.JobClient: map 14% reduce 0%
>> 11/01/07 16:12:30 INFO mapred.JobClient: map 15% reduce 0%
>> 11/01/07 16:12:33 INFO mapred.JobClient: map 16% reduce 0%
>> 11/01/07 16:12:36 INFO mapred.JobClient: map 17% reduce 0%
>> 11/01/07 16:12:40 INFO mapred.JobClient: map 18% reduce 0%
>> 11/01/07 16:12:45 INFO mapred.JobClient: map 19% reduce 0%
>> 11/01/07 16:12:48 INFO mapred.JobClient: map 20% reduce 0%
>> 11/01/07 16:12:54 INFO mapred.JobClient: map 21% reduce 0%
>> 11/01/07 16:13:00 INFO mapred.JobClient: map 22% reduce 0%
>> 11/01/07 16:13:04 INFO mapred.JobClient: map 22% reduce 1%
>> 11/01/07 16:13:13 INFO mapred.JobClient: map 23% reduce 1%
>> 11/01/07 16:13:19 INFO mapred.JobClient: map 24% reduce 1%
>> 11/01/07 16:13:25 INFO mapred.JobClient: map 25% reduce 1%
>> 11/01/07 16:13:30 INFO mapred.JobClient: map 26% reduce 1%
>> 11/01/07 16:13:34 INFO mapred.JobClient: map 26% reduce 3%
>> 11/01/07 16:13:36 INFO mapred.JobClient: map 27% reduce 3%
>> 11/01/07 16:13:37 INFO mapred.JobClient: map 27% reduce 4%
>> 11/01/07 16:13:39 INFO mapred.JobClient: map 28% reduce 4%
>> 11/01/07 16:13:43 INFO mapred.JobClient: map 29% reduce 4%
>> 11/01/07 16:13:46 INFO mapred.JobClient: map 30% reduce 4%
>> 11/01/07 16:13:49 INFO mapred.JobClient: map 31% reduce 4%
>> 11/01/07 16:13:52 INFO mapred.JobClient: map 32% reduce 4%
>> 11/01/07 16:13:55 INFO mapred.JobClient: map 33% reduce 4%
>> 11/01/07 16:13:58 INFO mapred.JobClient: map 34% reduce 4%
>> 11/01/07 16:14:02 INFO mapred.JobClient: map 35% reduce 4%
>> 11/01/07 16:14:05 INFO mapred.JobClient: map 36% reduce 4%
>> 11/01/07 16:14:08 INFO mapred.JobClient: map 37% reduce 4%
>> 11/01/07 16:14:11 INFO mapred.JobClient: map 38% reduce 4%
>> 11/01/07 16:14:15 INFO mapred.JobClient: map 39% reduce 4%
>> 11/01/07 16:14:19 INFO mapred.JobClient: map 40% reduce 4%
>> 11/01/07 16:14:20 INFO mapred.JobClient: map 40% reduce 5%
>> 11/01/07 16:14:25 INFO mapred.JobClient: map 41% reduce 5%
>> 11/01/07 16:14:32 INFO mapred.JobClient: map 42% reduce 5%
>> 11/01/07 16:14:38 INFO mapred.JobClient: map 43% reduce 5%
>> 11/01/07 16:14:41 INFO mapred.JobClient: map 43% reduce 6%
>> 11/01/07 16:14:43 INFO mapred.JobClient: map 44% reduce 6%
>> 11/01/07 16:14:47 INFO mapred.JobClient: map 45% reduce 6%
>> 11/01/07 16:14:50 INFO mapred.JobClient: map 46% reduce 6%
>> 11/01/07 16:14:54 INFO mapred.JobClient: map 47% reduce 7%
>> 11/01/07 16:14:59 INFO mapred.JobClient: map 48% reduce 7%
>> 11/01/07 16:15:02 INFO mapred.JobClient: map 49% reduce 7%
>> 11/01/07 16:15:05 INFO mapred.JobClient: map 50% reduce 7%
>> 11/01/07 16:15:11 INFO mapred.JobClient: map 51% reduce 7%
>> 11/01/07 16:15:14 INFO mapred.JobClient: map 52% reduce 7%
>> 11/01/07 16:15:16 INFO mapred.JobClient: map 52% reduce 8%
>> 11/01/07 16:15:20 INFO mapred.JobClient: map 53% reduce 8%
>> 11/01/07 16:15:25 INFO mapred.JobClient: map 54% reduce 8%
>> 11/01/07 16:15:29 INFO mapred.JobClient: map 55% reduce 8%
>> 11/01/07 16:15:31 INFO mapred.JobClient: map 55% reduce 9%
>> 11/01/07 16:15:33 INFO mapred.JobClient: map 56% reduce 9%
>> 11/01/07 16:15:38 INFO mapred.JobClient: map 57% reduce 9%
>> 11/01/07 16:15:42 INFO mapred.JobClient: map 58% reduce 9%
>> 11/01/07 16:15:43 INFO mapred.JobClient: map 58% reduce 10%
>> 11/01/07 16:15:46 INFO mapred.JobClient: map 59% reduce 10%
>> 11/01/07 16:15:49 INFO mapred.JobClient: map 60% reduce 10%
>> 11/01/07 16:15:53 INFO mapred.JobClient: map 61% reduce 10%
>> 11/01/07 16:15:56 INFO mapred.JobClient: map 62% reduce 10%
>> 11/01/07 16:16:00 INFO mapred.JobClient: map 63% reduce 10%
>> 11/01/07 16:16:06 INFO mapred.JobClient: map 64% reduce 10%
>> 11/01/07 16:16:10 INFO mapred.JobClient: map 65% reduce 10%
>> 11/01/07 16:16:15 INFO mapred.JobClient: map 66% reduce 10%
>> 11/01/07 16:16:18 INFO mapred.JobClient: map 67% reduce 10%
>> 11/01/07 16:16:19 INFO mapred.JobClient: map 67% reduce 12%
>> 11/01/07 16:16:21 INFO mapred.JobClient: map 68% reduce 12%
>> 11/01/07 16:16:25 INFO mapred.JobClient: map 69% reduce 12%
>> 11/01/07 16:16:28 INFO mapred.JobClient: map 70% reduce 12%
>> 11/01/07 16:16:31 INFO mapred.JobClient: map 71% reduce 12%
>> 11/01/07 16:16:35 INFO mapred.JobClient: map 72% reduce 12%
>> 11/01/07 16:16:38 INFO mapred.JobClient: map 73% reduce 12%
>> 11/01/07 16:16:43 INFO mapred.JobClient: map 74% reduce 12%
>> 11/01/07 16:16:47 INFO mapred.JobClient: map 75% reduce 12%
>> 11/01/07 16:16:53 INFO mapred.JobClient: map 75% reduce 13%
>> 11/01/07 16:16:54 INFO mapred.JobClient: map 76% reduce 13%
>> 11/01/07 16:17:01 INFO mapred.JobClient: map 77% reduce 13%
>> 11/01/07 16:17:03 INFO mapred.JobClient: map 77% reduce 14%
>> 11/01/07 16:17:06 INFO mapred.JobClient: map 78% reduce 15%
>> 11/01/07 16:17:11 INFO mapred.JobClient: map 79% reduce 15%
>> 11/01/07 16:17:15 INFO mapred.JobClient: map 80% reduce 15%
>> 11/01/07 16:17:23 INFO mapred.JobClient: map 81% reduce 15%
>> 11/01/07 16:17:27 INFO mapred.JobClient: map 82% reduce 15%
>> 11/01/07 16:17:31 INFO mapred.JobClient: map 83% reduce 15%
>> 11/01/07 16:17:33 INFO mapred.JobClient: map 84% reduce 15%
>> 11/01/07 16:17:38 INFO mapred.JobClient: map 85% reduce 15%
>> 11/01/07 16:17:41 INFO mapred.JobClient: map 86% reduce 15%
>> 11/01/07 16:17:45 INFO mapred.JobClient: map 87% reduce 15%
>> 11/01/07 16:17:50 INFO mapred.JobClient: map 88% reduce 15%
>> 11/01/07 16:17:54 INFO mapred.JobClient: map 89% reduce 15%
>> 11/01/07 16:18:00 INFO mapred.JobClient: map 90% reduce 15%
>> 11/01/07 16:18:02 INFO mapred.JobClient: map 90% reduce 16%
>> 11/01/07 16:18:03 INFO mapred.JobClient: map 91% reduce 16%
>> 11/01/07 16:18:06 INFO mapred.JobClient: map 92% reduce 16%
>> 11/01/07 16:18:09 INFO mapred.JobClient: map 93% reduce 16%
>> 11/01/07 16:18:14 INFO mapred.JobClient: map 94% reduce 16%
>> 11/01/07 16:18:17 INFO mapred.JobClient: map 95% reduce 16%
>> 11/01/07 16:18:21 INFO mapred.JobClient: map 96% reduce 16%
>> 11/01/07 16:18:27 INFO mapred.JobClient: map 97% reduce 16%
>> 11/01/07 16:18:29 INFO mapred.JobClient: map 97% reduce 17%
>> 11/01/07 16:18:35 INFO mapred.JobClient: map 98% reduce 17%
>> 11/01/07 16:18:48 INFO mapred.JobClient: map 99% reduce 18%
>> 11/01/07 16:18:55 INFO mapred.JobClient: map 99% reduce 19%
>> 11/01/07 16:19:12 INFO mapred.JobClient: map 100% reduce 19%
>> 11/01/07 16:19:13 INFO mapred.JobClient: map 100% reduce 20%
>> 11/01/07 16:19:34 INFO mapred.JobClient: map 100% reduce 21%
>> 11/01/07 16:22:54 INFO mapred.JobClient: Task Id :
>> attempt_201101071129_0001_m_000012_0, Status : FAILED
>> Too many fetch-failures
>> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stdout
>> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stderr
>> 11/01/07 16:22:58 INFO mapred.JobClient: map 98% reduce 21%
>> 11/01/07 16:23:16 INFO mapred.JobClient: map 99% reduce 21%
>> 11/01/07 16:23:34 INFO mapred.JobClient: map 100% reduce 21%
>> 11/01/07 16:25:28 INFO mapred.JobClient: map 100% reduce 23%
>> 11/01/07 16:35:17 INFO mapred.JobClient: Task Id :
>> attempt_201101071129_0001_m_000011_0, Status : FAILED
>> Too many fetch-failures
>> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stdout
>> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stderr
>> 11/01/07 16:35:21 INFO mapred.JobClient: map 98% reduce 23%
>> 11/01/07 16:35:39 INFO mapred.JobClient: map 99% reduce 23%
>> 11/01/07 16:35:54 INFO mapred.JobClient: map 100% reduce 23%
>> 11/01/07 16:37:49 INFO mapred.JobClient: map 100% reduce 24%
>> 11/01/07 16:47:32 INFO mapred.JobClient: Task Id :
>> attempt_201101071129_0001_m_000015_0, Status : FAILED
>> Too many fetch-failures
>> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stdout
>> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stderr
>> 11/01/07 16:47:36 INFO mapred.JobClient: map 98% reduce 24%
>> 11/01/07 16:47:54 INFO mapred.JobClient: map 99% reduce 24%
>> 11/01/07 16:48:12 INFO mapred.JobClient: map 100% reduce 24%
>> 11/01/07 16:50:03 INFO mapred.JobClient: map 100% reduce 25%
>> 11/01/07 16:59:48 INFO mapred.JobClient: Task Id :
>> attempt_201101071129_0001_m_000019_0, Status : FAILED
>> Too many fetch-failures
>> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stdout
>> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
>> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stderr
>> 11/01/07 16:59:52 INFO mapred.JobClient: map 98% reduce 25%
>> 11/01/07 17:00:10 INFO mapred.JobClient: map 99% reduce 25%
>> 11/01/07 17:00:26 INFO mapred.JobClient: map 100% reduce 25%
>> 11/01/07 17:00:38 INFO mapred.JobClient: map 100% reduce 26%
>>
>>
>> Would anyone Please send me the reason as I am interested to find the cause
>> to clear my understanding.
>> I configured properly Hadoop on 2 clusters. I am sure there is no
>> configuration problem because in my other cluster it runs fine.
>>
>> One Cluster is on Standalone Servers ( 4 nodes ). Job is executed
>> successfully.
>>
>> Other Cluster is on VM's (Cloud) .
>>
>> My tasktracker log says this
>>
>> 2011-01-07 16:20:23,234 WARN org.apache.hadoop.mapred.TaskTracker:
>> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>> in any of the configured local directories
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>> at
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:324)
>> at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> 2011-01-07 16:20:23,236 WARN org.apache.hadoop.mapred.TaskTracker: Unknown
>> child with bad map output: attempt_201101071129_0001_m_000012_0. Ignored.
>> 2011-01-07 16:20:23,239 INFO
>> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 172.16.1.3:50060,
>> dest: 172.16.1.5:47135, bytes: 0, op: MAPRED_SHUFFLE, cliID:
>> attempt_201101071129_0001_m_000012_0
>> 2011-01-07 16:20:23,239 WARN org.mortbay.log: /mapOutput:
>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>> in any of the configured local directories
>> 2011-01-07 16:22:53,266 WARN org.apache.hadoop.mapred.TaskTracker:
>> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
>> in any of the configured local directories
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>> at
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:324)
>> at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>
>>
>>
>> Let's have some discussion.
>>
>>
>> Thanks & Regards
>>
>> Adarsh Sharma
>>
>>
>>
>>
>>
>>
>>
>
>
Re: Too-many fetch failure Reduce Error
Posted by Esteban Gutierrez Moguel <es...@gmail.com>.
Adarsh,
Dou you have in /etc/hosts the hostnames for masters and slaves?
esteban.
On Fri, Jan 7, 2011 at 06:47, Adarsh Sharma <ad...@orkash.com>wrote:
> Dear all,
>
> I am researching about the below error and could not able to find the
> reason :
>
> Data Size : 3.4 GB
> Hadoop-0.20.0
>
> hadoop@ws32-test-lin:~/project/hadoop-0.20.2$ bin/hadoop jar
> hadoop-0.20.2-examples.jar wordcount /user/hadoop/page_content.txt
> page_content_output.txt
> 11/01/07 16:11:14 INFO input.FileInputFormat: Total input paths to process
> : 1
> 11/01/07 16:11:15 INFO mapred.JobClient: Running job: job_201101071129_0001
> 11/01/07 16:11:16 INFO mapred.JobClient: map 0% reduce 0%
> 11/01/07 16:11:41 INFO mapred.JobClient: map 1% reduce 0%
> 11/01/07 16:11:45 INFO mapred.JobClient: map 2% reduce 0%
> 11/01/07 16:11:48 INFO mapred.JobClient: map 3% reduce 0%
> 11/01/07 16:11:52 INFO mapred.JobClient: map 4% reduce 0%
> 11/01/07 16:11:56 INFO mapred.JobClient: map 5% reduce 0%
> 11/01/07 16:12:00 INFO mapred.JobClient: map 6% reduce 0%
> 11/01/07 16:12:05 INFO mapred.JobClient: map 7% reduce 0%
> 11/01/07 16:12:08 INFO mapred.JobClient: map 8% reduce 0%
> 11/01/07 16:12:11 INFO mapred.JobClient: map 9% reduce 0%
> 11/01/07 16:12:14 INFO mapred.JobClient: map 10% reduce 0%
> 11/01/07 16:12:17 INFO mapred.JobClient: map 11% reduce 0%
> 11/01/07 16:12:21 INFO mapred.JobClient: map 12% reduce 0%
> 11/01/07 16:12:24 INFO mapred.JobClient: map 13% reduce 0%
> 11/01/07 16:12:27 INFO mapred.JobClient: map 14% reduce 0%
> 11/01/07 16:12:30 INFO mapred.JobClient: map 15% reduce 0%
> 11/01/07 16:12:33 INFO mapred.JobClient: map 16% reduce 0%
> 11/01/07 16:12:36 INFO mapred.JobClient: map 17% reduce 0%
> 11/01/07 16:12:40 INFO mapred.JobClient: map 18% reduce 0%
> 11/01/07 16:12:45 INFO mapred.JobClient: map 19% reduce 0%
> 11/01/07 16:12:48 INFO mapred.JobClient: map 20% reduce 0%
> 11/01/07 16:12:54 INFO mapred.JobClient: map 21% reduce 0%
> 11/01/07 16:13:00 INFO mapred.JobClient: map 22% reduce 0%
> 11/01/07 16:13:04 INFO mapred.JobClient: map 22% reduce 1%
> 11/01/07 16:13:13 INFO mapred.JobClient: map 23% reduce 1%
> 11/01/07 16:13:19 INFO mapred.JobClient: map 24% reduce 1%
> 11/01/07 16:13:25 INFO mapred.JobClient: map 25% reduce 1%
> 11/01/07 16:13:30 INFO mapred.JobClient: map 26% reduce 1%
> 11/01/07 16:13:34 INFO mapred.JobClient: map 26% reduce 3%
> 11/01/07 16:13:36 INFO mapred.JobClient: map 27% reduce 3%
> 11/01/07 16:13:37 INFO mapred.JobClient: map 27% reduce 4%
> 11/01/07 16:13:39 INFO mapred.JobClient: map 28% reduce 4%
> 11/01/07 16:13:43 INFO mapred.JobClient: map 29% reduce 4%
> 11/01/07 16:13:46 INFO mapred.JobClient: map 30% reduce 4%
> 11/01/07 16:13:49 INFO mapred.JobClient: map 31% reduce 4%
> 11/01/07 16:13:52 INFO mapred.JobClient: map 32% reduce 4%
> 11/01/07 16:13:55 INFO mapred.JobClient: map 33% reduce 4%
> 11/01/07 16:13:58 INFO mapred.JobClient: map 34% reduce 4%
> 11/01/07 16:14:02 INFO mapred.JobClient: map 35% reduce 4%
> 11/01/07 16:14:05 INFO mapred.JobClient: map 36% reduce 4%
> 11/01/07 16:14:08 INFO mapred.JobClient: map 37% reduce 4%
> 11/01/07 16:14:11 INFO mapred.JobClient: map 38% reduce 4%
> 11/01/07 16:14:15 INFO mapred.JobClient: map 39% reduce 4%
> 11/01/07 16:14:19 INFO mapred.JobClient: map 40% reduce 4%
> 11/01/07 16:14:20 INFO mapred.JobClient: map 40% reduce 5%
> 11/01/07 16:14:25 INFO mapred.JobClient: map 41% reduce 5%
> 11/01/07 16:14:32 INFO mapred.JobClient: map 42% reduce 5%
> 11/01/07 16:14:38 INFO mapred.JobClient: map 43% reduce 5%
> 11/01/07 16:14:41 INFO mapred.JobClient: map 43% reduce 6%
> 11/01/07 16:14:43 INFO mapred.JobClient: map 44% reduce 6%
> 11/01/07 16:14:47 INFO mapred.JobClient: map 45% reduce 6%
> 11/01/07 16:14:50 INFO mapred.JobClient: map 46% reduce 6%
> 11/01/07 16:14:54 INFO mapred.JobClient: map 47% reduce 7%
> 11/01/07 16:14:59 INFO mapred.JobClient: map 48% reduce 7%
> 11/01/07 16:15:02 INFO mapred.JobClient: map 49% reduce 7%
> 11/01/07 16:15:05 INFO mapred.JobClient: map 50% reduce 7%
> 11/01/07 16:15:11 INFO mapred.JobClient: map 51% reduce 7%
> 11/01/07 16:15:14 INFO mapred.JobClient: map 52% reduce 7%
> 11/01/07 16:15:16 INFO mapred.JobClient: map 52% reduce 8%
> 11/01/07 16:15:20 INFO mapred.JobClient: map 53% reduce 8%
> 11/01/07 16:15:25 INFO mapred.JobClient: map 54% reduce 8%
> 11/01/07 16:15:29 INFO mapred.JobClient: map 55% reduce 8%
> 11/01/07 16:15:31 INFO mapred.JobClient: map 55% reduce 9%
> 11/01/07 16:15:33 INFO mapred.JobClient: map 56% reduce 9%
> 11/01/07 16:15:38 INFO mapred.JobClient: map 57% reduce 9%
> 11/01/07 16:15:42 INFO mapred.JobClient: map 58% reduce 9%
> 11/01/07 16:15:43 INFO mapred.JobClient: map 58% reduce 10%
> 11/01/07 16:15:46 INFO mapred.JobClient: map 59% reduce 10%
> 11/01/07 16:15:49 INFO mapred.JobClient: map 60% reduce 10%
> 11/01/07 16:15:53 INFO mapred.JobClient: map 61% reduce 10%
> 11/01/07 16:15:56 INFO mapred.JobClient: map 62% reduce 10%
> 11/01/07 16:16:00 INFO mapred.JobClient: map 63% reduce 10%
> 11/01/07 16:16:06 INFO mapred.JobClient: map 64% reduce 10%
> 11/01/07 16:16:10 INFO mapred.JobClient: map 65% reduce 10%
> 11/01/07 16:16:15 INFO mapred.JobClient: map 66% reduce 10%
> 11/01/07 16:16:18 INFO mapred.JobClient: map 67% reduce 10%
> 11/01/07 16:16:19 INFO mapred.JobClient: map 67% reduce 12%
> 11/01/07 16:16:21 INFO mapred.JobClient: map 68% reduce 12%
> 11/01/07 16:16:25 INFO mapred.JobClient: map 69% reduce 12%
> 11/01/07 16:16:28 INFO mapred.JobClient: map 70% reduce 12%
> 11/01/07 16:16:31 INFO mapred.JobClient: map 71% reduce 12%
> 11/01/07 16:16:35 INFO mapred.JobClient: map 72% reduce 12%
> 11/01/07 16:16:38 INFO mapred.JobClient: map 73% reduce 12%
> 11/01/07 16:16:43 INFO mapred.JobClient: map 74% reduce 12%
> 11/01/07 16:16:47 INFO mapred.JobClient: map 75% reduce 12%
> 11/01/07 16:16:53 INFO mapred.JobClient: map 75% reduce 13%
> 11/01/07 16:16:54 INFO mapred.JobClient: map 76% reduce 13%
> 11/01/07 16:17:01 INFO mapred.JobClient: map 77% reduce 13%
> 11/01/07 16:17:03 INFO mapred.JobClient: map 77% reduce 14%
> 11/01/07 16:17:06 INFO mapred.JobClient: map 78% reduce 15%
> 11/01/07 16:17:11 INFO mapred.JobClient: map 79% reduce 15%
> 11/01/07 16:17:15 INFO mapred.JobClient: map 80% reduce 15%
> 11/01/07 16:17:23 INFO mapred.JobClient: map 81% reduce 15%
> 11/01/07 16:17:27 INFO mapred.JobClient: map 82% reduce 15%
> 11/01/07 16:17:31 INFO mapred.JobClient: map 83% reduce 15%
> 11/01/07 16:17:33 INFO mapred.JobClient: map 84% reduce 15%
> 11/01/07 16:17:38 INFO mapred.JobClient: map 85% reduce 15%
> 11/01/07 16:17:41 INFO mapred.JobClient: map 86% reduce 15%
> 11/01/07 16:17:45 INFO mapred.JobClient: map 87% reduce 15%
> 11/01/07 16:17:50 INFO mapred.JobClient: map 88% reduce 15%
> 11/01/07 16:17:54 INFO mapred.JobClient: map 89% reduce 15%
> 11/01/07 16:18:00 INFO mapred.JobClient: map 90% reduce 15%
> 11/01/07 16:18:02 INFO mapred.JobClient: map 90% reduce 16%
> 11/01/07 16:18:03 INFO mapred.JobClient: map 91% reduce 16%
> 11/01/07 16:18:06 INFO mapred.JobClient: map 92% reduce 16%
> 11/01/07 16:18:09 INFO mapred.JobClient: map 93% reduce 16%
> 11/01/07 16:18:14 INFO mapred.JobClient: map 94% reduce 16%
> 11/01/07 16:18:17 INFO mapred.JobClient: map 95% reduce 16%
> 11/01/07 16:18:21 INFO mapred.JobClient: map 96% reduce 16%
> 11/01/07 16:18:27 INFO mapred.JobClient: map 97% reduce 16%
> 11/01/07 16:18:29 INFO mapred.JobClient: map 97% reduce 17%
> 11/01/07 16:18:35 INFO mapred.JobClient: map 98% reduce 17%
> 11/01/07 16:18:48 INFO mapred.JobClient: map 99% reduce 18%
> 11/01/07 16:18:55 INFO mapred.JobClient: map 99% reduce 19%
> 11/01/07 16:19:12 INFO mapred.JobClient: map 100% reduce 19%
> 11/01/07 16:19:13 INFO mapred.JobClient: map 100% reduce 20%
> 11/01/07 16:19:34 INFO mapred.JobClient: map 100% reduce 21%
> 11/01/07 16:22:54 INFO mapred.JobClient: Task Id :
> attempt_201101071129_0001_m_000012_0, Status : FAILED
> Too many fetch-failures
> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stdout
> 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stderr
> 11/01/07 16:22:58 INFO mapred.JobClient: map 98% reduce 21%
> 11/01/07 16:23:16 INFO mapred.JobClient: map 99% reduce 21%
> 11/01/07 16:23:34 INFO mapred.JobClient: map 100% reduce 21%
> 11/01/07 16:25:28 INFO mapred.JobClient: map 100% reduce 23%
> 11/01/07 16:35:17 INFO mapred.JobClient: Task Id :
> attempt_201101071129_0001_m_000011_0, Status : FAILED
> Too many fetch-failures
> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stdout
> 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stderr
> 11/01/07 16:35:21 INFO mapred.JobClient: map 98% reduce 23%
> 11/01/07 16:35:39 INFO mapred.JobClient: map 99% reduce 23%
> 11/01/07 16:35:54 INFO mapred.JobClient: map 100% reduce 23%
> 11/01/07 16:37:49 INFO mapred.JobClient: map 100% reduce 24%
> 11/01/07 16:47:32 INFO mapred.JobClient: Task Id :
> attempt_201101071129_0001_m_000015_0, Status : FAILED
> Too many fetch-failures
> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stdout
> 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stderr
> 11/01/07 16:47:36 INFO mapred.JobClient: map 98% reduce 24%
> 11/01/07 16:47:54 INFO mapred.JobClient: map 99% reduce 24%
> 11/01/07 16:48:12 INFO mapred.JobClient: map 100% reduce 24%
> 11/01/07 16:50:03 INFO mapred.JobClient: map 100% reduce 25%
> 11/01/07 16:59:48 INFO mapred.JobClient: Task Id :
> attempt_201101071129_0001_m_000019_0, Status : FAILED
> Too many fetch-failures
> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stdout
> 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stderr
> 11/01/07 16:59:52 INFO mapred.JobClient: map 98% reduce 25%
> 11/01/07 17:00:10 INFO mapred.JobClient: map 99% reduce 25%
> 11/01/07 17:00:26 INFO mapred.JobClient: map 100% reduce 25%
> 11/01/07 17:00:38 INFO mapred.JobClient: map 100% reduce 26%
>
>
> Would anyone Please send me the reason as I am interested to find the cause
> to clear my understanding.
> I configured properly Hadoop on 2 clusters. I am sure there is no
> configuration problem because in my other cluster it runs fine.
>
> One Cluster is on Standalone Servers ( 4 nodes ). Job is executed
> successfully.
>
> Other Cluster is on VM's (Cloud) .
>
> My tasktracker log says this
>
> 2011-01-07 16:20:23,234 WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
> in any of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> 2011-01-07 16:20:23,236 WARN org.apache.hadoop.mapred.TaskTracker: Unknown
> child with bad map output: attempt_201101071129_0001_m_000012_0. Ignored.
> 2011-01-07 16:20:23,239 INFO
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 172.16.1.3:50060,
> dest: 172.16.1.5:47135, bytes: 0, op: MAPRED_SHUFFLE, cliID:
> attempt_201101071129_0001_m_000012_0
> 2011-01-07 16:20:23,239 WARN org.mortbay.log: /mapOutput:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
> in any of the configured local directories
> 2011-01-07 16:22:53,266 WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index
> in any of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
>
> Let's have some discussion.
>
>
> Thanks & Regards
>
> Adarsh Sharma
>
>
>
>
>
>