You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Ajay Srivastava <Aj...@guavus.com> on 2012/12/03 18:21:26 UTC
Too many fetch-failures
Hi,
I am getting following errors -
12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
Too many fetch-failures
attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
Too many fetch-failures
attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
I see following error in task tracker log :-
2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
Regards,
Ajay Srivastava
Re: Too many fetch-failures
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Harsh.
Problem is resolved. Entry of one of the datanodes was missing from /etc/hosts. After adding the entry, job finished without any problem.
Regards,
Ajay Srivastava
On 04-Dec-2012, at 2:50 AM, Harsh J wrote:
> What version/distribution of Hadoop is this?
>
> On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
> <Aj...@guavus.com> wrote:
>> Hi,
>>
>> I am getting following errors -
>>
>> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
>> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
>> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
>> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
>> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
>> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
>> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
>> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
>> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
>> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
>> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
>> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
>> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>>
>> I see following error in task tracker log :-
>>
>> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
>> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>> at org.mortbay.jetty.Response.sendError(Response.java:240)
>> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>
>> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>>
>>
>> Regards,
>> Ajay Srivastava
>>
>>
>>
>
>
>
> --
> Harsh J
Re: Too many fetch-failures
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Harsh.
Problem is resolved. Entry of one of the datanodes was missing from /etc/hosts. After adding the entry, job finished without any problem.
Regards,
Ajay Srivastava
On 04-Dec-2012, at 2:50 AM, Harsh J wrote:
> What version/distribution of Hadoop is this?
>
> On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
> <Aj...@guavus.com> wrote:
>> Hi,
>>
>> I am getting following errors -
>>
>> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
>> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
>> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
>> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
>> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
>> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
>> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
>> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
>> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
>> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
>> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
>> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
>> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>>
>> I see following error in task tracker log :-
>>
>> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
>> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>> at org.mortbay.jetty.Response.sendError(Response.java:240)
>> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>
>> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>>
>>
>> Regards,
>> Ajay Srivastava
>>
>>
>>
>
>
>
> --
> Harsh J
Re: Too many fetch-failures
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Harsh.
Problem is resolved. Entry of one of the datanodes was missing from /etc/hosts. After adding the entry, job finished without any problem.
Regards,
Ajay Srivastava
On 04-Dec-2012, at 2:50 AM, Harsh J wrote:
> What version/distribution of Hadoop is this?
>
> On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
> <Aj...@guavus.com> wrote:
>> Hi,
>>
>> I am getting following errors -
>>
>> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
>> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
>> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
>> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
>> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
>> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
>> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
>> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
>> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
>> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
>> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
>> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
>> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>>
>> I see following error in task tracker log :-
>>
>> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
>> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>> at org.mortbay.jetty.Response.sendError(Response.java:240)
>> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>
>> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>>
>>
>> Regards,
>> Ajay Srivastava
>>
>>
>>
>
>
>
> --
> Harsh J
Re: Too many fetch-failures
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Harsh.
Problem is resolved. Entry of one of the datanodes was missing from /etc/hosts. After adding the entry, job finished without any problem.
Regards,
Ajay Srivastava
On 04-Dec-2012, at 2:50 AM, Harsh J wrote:
> What version/distribution of Hadoop is this?
>
> On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
> <Aj...@guavus.com> wrote:
>> Hi,
>>
>> I am getting following errors -
>>
>> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
>> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
>> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
>> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
>> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
>> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
>> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
>> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
>> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
>> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
>> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
>> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
>> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
>> Too many fetch-failures
>> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
>> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>>
>> I see following error in task tracker log :-
>>
>> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
>> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>> at org.mortbay.jetty.Response.sendError(Response.java:240)
>> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>
>> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>>
>>
>> Regards,
>> Ajay Srivastava
>>
>>
>>
>
>
>
> --
> Harsh J
Re: Too many fetch-failures
Posted by Harsh J <ha...@cloudera.com>.
What version/distribution of Hadoop is this?
On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> I am getting following errors -
>
> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>
> I see following error in task tracker log :-
>
> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
> at org.mortbay.jetty.Response.sendError(Response.java:240)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>
>
> Regards,
> Ajay Srivastava
>
>
>
--
Harsh J
Re: Too many fetch-failures
Posted by Harsh J <ha...@cloudera.com>.
What version/distribution of Hadoop is this?
On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> I am getting following errors -
>
> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>
> I see following error in task tracker log :-
>
> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
> at org.mortbay.jetty.Response.sendError(Response.java:240)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>
>
> Regards,
> Ajay Srivastava
>
>
>
--
Harsh J
Re: Too many fetch-failures
Posted by Harsh J <ha...@cloudera.com>.
What version/distribution of Hadoop is this?
On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> I am getting following errors -
>
> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>
> I see following error in task tracker log :-
>
> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
> at org.mortbay.jetty.Response.sendError(Response.java:240)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>
>
> Regards,
> Ajay Srivastava
>
>
>
--
Harsh J
Re: Too many fetch-failures
Posted by Harsh J <ha...@cloudera.com>.
What version/distribution of Hadoop is this?
On Mon, Dec 3, 2012 at 10:51 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> I am getting following errors -
>
> 12/12/03 17:11:42 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000011_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000011_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:11:47 INFO mapred.JobClient: map 99% reduce 24%
> 12/12/03 17:12:04 INFO mapred.JobClient: map 99% reduce 32%
> 12/12/03 17:12:07 INFO mapred.JobClient: map 99% reduce 33%
> 12/12/03 17:12:13 INFO mapred.JobClient: map 99% reduce 34%
> 12/12/03 17:12:22 INFO mapred.JobClient: map 99% reduce 35%
> 12/12/03 17:12:28 INFO mapred.JobClient: map 99% reduce 36%
> 12/12/03 17:12:29 INFO mapred.JobClient: map 100% reduce 36%
> 12/12/03 17:12:39 INFO mapred.JobClient: map 100% reduce 37%
> 12/12/03 17:12:46 INFO mapred.JobClient: map 100% reduce 38%
> 12/12/03 17:12:55 INFO mapred.JobClient: map 100% reduce 39%
> 12/12/03 17:13:08 INFO mapred.JobClient: map 100% reduce 40%
> 12/12/03 17:13:23 INFO mapred.JobClient: map 100% reduce 41%
> 12/12/03 17:13:59 INFO mapred.JobClient: Task Id : attempt_201212031650_0001_m_000012_0, Status : FAILED
> Too many fetch-failures
> attempt_201212031650_0001_m_000012_0: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 12/12/03 17:14:04 INFO mapred.JobClient: map 99% reduce 41%
>
> I see following error in task tracker log :-
>
> 2012-12-03 17:09:40,969 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.156.22:50060, dest: 192.168.156.22:42666, bytes: 131072, op: MAPRED_SHUFFLE, cliID: attempt_201212031650_0001_m_000166_0, duration: 3343330
> 2012-12-03 17:09:40,970 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
> at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
> at org.mortbay.jetty.Response.sendError(Response.java:240)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3627)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> What could be the reason behind it ? The job should have finished in 10 min. but it's going on and on for hours.
>
>
> Regards,
> Ajay Srivastava
>
>
>
--
Harsh J
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Too many fetch-failures
Posted by Liuva Rosabal Valdes <lr...@estudiantes.uci.cu>.
HI
Forum in which you are enrolled
--
ALGUIEN QUE TE AMA , SIEMPRE HARA LO QUE CREE MEJOR PARA TI AUNQUE NO TODAS LAS VECES TENGA RAZON.
BESITOS LUV
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci