You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Alves <da...@student.dei.uc.pt> on 2008/11/20 19:29:29 UTC
Map merge part makes the task timeout
Hi all
I have a big map task that takes a long time to complete and produces
a lot of information.
The merge part makes the task timeout (you can see this in the end of
this email, the merge part is aborted after ten minutes, the default
time).
I've increased the mapred.tasks.timeout property to 30 min instead of
10, in hadoop-site.xml as follows:
<property> <name>mapred.task.timeout</name> <value>1800000</value> </
property>
But stiil the task fails with:
Task attempt_200811201704_0001_m_000000_0 failed to report status for
603 seconds. Killing!
Is there any other property I should change?.
Regards
David Alves
17:35:40,697 INFO [MapTask] Starting flush of map output
17:35:40,697 INFO [MapTask] bufstart = 17472260; bufend = 51354916;
bufvoid = 99614720
17:35:40,697 INFO [MapTask] kvstart = 39119; kvend = 39468; length =
327680
17:35:40,950 INFO [MapTask] Index: (0, 33884416, 33884416)
17:35:40,950 INFO [MapTask] Finished spill 152
17:35:45,333 INFO [Merger] Merging 153 sorted segments
17:35:46,337 INFO [Merger] Merging 9 intermediate segments out of a
total of 153
17:36:16,849 INFO [Merger] Merging 10 intermediate segments out of a
total of 145
17:36:47,615 INFO [Merger] Merging 10 intermediate segments out of a
total of 136
17:37:21,529 INFO [Merger] Merging 10 intermediate segments out of a
total of 127
17:37:59,883 INFO [Merger] Merging 10 intermediate segments out of a
total of 118
17:38:35,370 INFO [Merger] Merging 10 intermediate segments out of a
total of 109
17:39:14,795 INFO [Merger] Merging 10 intermediate segments out of a
total of 100
17:39:51,787 INFO [Merger] Merging 10 intermediate segments out of a
total of 91
17:40:28,721 INFO [Merger] Merging 10 intermediate segments out of a
total of 82
17:41:05,650 INFO [Merger] Merging 10 intermediate segments out of a
total of 73
17:41:43,285 INFO [Merger] Merging 10 intermediate segments out of a
total of 64
17:42:23,531 INFO [Merger] Merging 10 intermediate segments out of a
total of 55
17:43:01,709 INFO [Merger] Merging 10 intermediate segments out of a
total of 46
17:43:40,209 INFO [Merger] Merging 10 intermediate segments out of a
total of 37
17:44:20,707 INFO [Merger] Merging 10 intermediate segments out of a
total of 28
17:44:57,700 INFO [Merger] Merging 10 intermediate segments out of a
total of 19
Re: Map merge part makes the task timeout
Posted by David Alves <da...@student.dei.uc.pt>.
Hi again
Browsing the source code (Merger.class) I see that merger actually
call reporter.progress() so shouldn't this make the task be reported
as still working?
Regards
David Alves
On Nov 20, 2008, at 6:29 PM, David Alves wrote:
> Hi all
>
> I have a big map task that takes a long time to complete and
> produces a lot of information.
> The merge part makes the task timeout (you can see this in the end
> of this email, the merge part is aborted after ten minutes, the
> default time).
> I've increased the mapred.tasks.timeout property to 30 min instead
> of 10, in hadoop-site.xml as follows:
> <property> <name>mapred.task.timeout</name> <value>1800000</value>
> </property>
> But stiil the task fails with:
> Task attempt_200811201704_0001_m_000000_0 failed to report status
> for 603 seconds. Killing!
>
> Is there any other property I should change?.
>
> Regards
> David Alves
>
>
>
> 17:35:40,697 INFO [MapTask] Starting flush of map output
> 17:35:40,697 INFO [MapTask] bufstart = 17472260; bufend = 51354916;
> bufvoid = 99614720
> 17:35:40,697 INFO [MapTask] kvstart = 39119; kvend = 39468; length
> = 327680
> 17:35:40,950 INFO [MapTask] Index: (0, 33884416, 33884416)
> 17:35:40,950 INFO [MapTask] Finished spill 152
> 17:35:45,333 INFO [Merger] Merging 153 sorted segments
> 17:35:46,337 INFO [Merger] Merging 9 intermediate segments out of a
> total of 153
> 17:36:16,849 INFO [Merger] Merging 10 intermediate segments out of
> a total of 145
> 17:36:47,615 INFO [Merger] Merging 10 intermediate segments out of
> a total of 136
> 17:37:21,529 INFO [Merger] Merging 10 intermediate segments out of
> a total of 127
> 17:37:59,883 INFO [Merger] Merging 10 intermediate segments out of
> a total of 118
> 17:38:35,370 INFO [Merger] Merging 10 intermediate segments out of
> a total of 109
> 17:39:14,795 INFO [Merger] Merging 10 intermediate segments out of
> a total of 100
> 17:39:51,787 INFO [Merger] Merging 10 intermediate segments out of
> a total of 91
> 17:40:28,721 INFO [Merger] Merging 10 intermediate segments out of
> a total of 82
> 17:41:05,650 INFO [Merger] Merging 10 intermediate segments out of
> a total of 73
> 17:41:43,285 INFO [Merger] Merging 10 intermediate segments out of
> a total of 64
> 17:42:23,531 INFO [Merger] Merging 10 intermediate segments out of
> a total of 55
> 17:43:01,709 INFO [Merger] Merging 10 intermediate segments out of
> a total of 46
> 17:43:40,209 INFO [Merger] Merging 10 intermediate segments out of
> a total of 37
> 17:44:20,707 INFO [Merger] Merging 10 intermediate segments out of
> a total of 28
> 17:44:57,700 INFO [Merger] Merging 10 intermediate segments out of
> a total of 19