You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by EdwardKing <zh...@neusoft.com> on 2014/04/23 04:28:39 UTC
map execute twice
I use Hadoop 2.2.0, I know hadoop will execute map first,when map is 100%, it then execute reduce, after reduce is 100%,job will end. I execute a job,the map is from 0% to 100% and map is from 0% to 100% again, why map execute twice? Thanks.
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2014-04-22 19:08:49,118 Stage-1 map = 0%, reduce = 0%
2014-04-22 19:11:46,722 Stage-1 map = 100%, reduce = 0%
2014-04-22 19:12:27,633 Stage-1 map = 0%, reduce = 0%
2014-04-22 19:14:37,655 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 1.5 sec
2014-04-22 19:15:39,248 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 3.34 sec
2014-04-22 19:15:59,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.34 sec
2014-04-22 19:16:40,988 Stage-1 map = 0%, reduce = 0%
2014-04-22 19:18:56,845 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 2.57 sec
2014-04-22 19:19:46,574 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 2.73 sec
2014-04-22 19:20:30,718 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 2.82 sec
2014-04-22 19:20:35,007 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 3.57 sec
2014-04-22 19:20:55,280 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 3.76 sec
2014-04-22 19:21:27,247 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 4.41 sec
2014-04-22 19:22:28,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 4.41 sec
2014-04-22 19:22:49,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 4.41 sec
2014-04-22 19:22:52,995 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.69 sec
MapReduce Total cumulative CPU time: 5 seconds 690 msec
Ended Job = job_1398218615130_0001
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: map execute twice
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zh...@neusoft.com> wrote:
> I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice? Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%, reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%, reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%, reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%, reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: map execute twice
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zh...@neusoft.com> wrote:
> I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice? Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%, reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%, reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%, reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%, reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: map execute twice
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zh...@neusoft.com> wrote:
> I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice? Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%, reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%, reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%, reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%, reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: map execute twice
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This can happen when maps are marked as failed *after* they have
successfully completed the map operation. One common reason when this can
happen is reducers faiingl to fetch the map-outputs due to the node that
ran the mapper going down, the machine freezing up etc.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Apr 22, 2014 at 7:28 PM, EdwardKing <zh...@neusoft.com> wrote:
> I use Hadoop 2.2.0, I know hadoop will execute map first,when map is
> 100%, it then execute reduce, after reduce is 100%,job will end. I execute
> a job,the map is from 0% to 100% and map is from 0% to 100% again, why map
> execute twice? Thanks.
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2014-04-22 19:08:49,118 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:11:46,722 Stage-1 map = 100%, reduce = 0%
> 2014-04-22 19:12:27,633 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:14:37,655 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 1.5 sec
> 2014-04-22 19:15:39,248 Stage-1 map = 33%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:15:59,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 3.34 sec
> 2014-04-22 19:16:40,988 Stage-1 map = 0%, reduce = 0%
> 2014-04-22 19:18:56,845 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
> 2.57 sec
> 2014-04-22 19:19:46,574 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 2.73 sec
> 2014-04-22 19:20:30,718 Stage-1 map = 22%, reduce = 0%, Cumulative CPU
> 2.82 sec
> 2014-04-22 19:20:35,007 Stage-1 map = 41%, reduce = 0%, Cumulative CPU
> 3.57 sec
> 2014-04-22 19:20:55,280 Stage-1 map = 45%, reduce = 0%, Cumulative CPU
> 3.76 sec
> 2014-04-22 19:21:27,247 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:28,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:49,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU
> 4.41 sec
> 2014-04-22 19:22:52,995 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 5.69 sec
> MapReduce Total cumulative CPU time: 5 seconds 690 msec
> Ended Job = job_1398218615130_0001
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.