You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Efe Gencer <ef...@gmail.com> on 2013/09/24 23:53:17 UTC
Which Subphases Do Times on JobHistory Web UI Cover
Hi All,
In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
...
Started: <start time>
Finished: <finish time>
Elapsed: 12 mins, 5sec
Diagnostics:
*Average Map Time*: 1 mins, 40 sec
Average Reduce Time: 12 sec
Average Shuffle Time: 10 mins, 8 sec
Average Merge Time: 1 sec
...
1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
sort, spill, merge, read and transfer corresponding filesplit from HDFS)
2) Does Reduce time covers Shuffle Time? What else does it cover? (write to
hdfs, etc)
3) Is Average Merge time related with map or reduce? (since they both have
merge phases)
Best,
Efe
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Sandy Ryza <sa...@cloudera.com>.
Average map time includes everything the map task is doing, i.e. all the
things you mentioned. Reduce time does not cover shuffle time. Reduce
time is the time spent calling the reducer function and writing its output
to HDFS. Merge time is related to reduce, not map.
-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <ef...@gmail.com> wrote:
> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <ef...@gmail.com>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Sandy Ryza <sa...@cloudera.com>.
Average map time includes everything the map task is doing, i.e. all the
things you mentioned. Reduce time does not cover shuffle time. Reduce
time is the time spent calling the reducer function and writing its output
to HDFS. Merge time is related to reduce, not map.
-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <ef...@gmail.com> wrote:
> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <ef...@gmail.com>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Sandy Ryza <sa...@cloudera.com>.
Average map time includes everything the map task is doing, i.e. all the
things you mentioned. Reduce time does not cover shuffle time. Reduce
time is the time spent calling the reducer function and writing its output
to HDFS. Merge time is related to reduce, not map.
-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <ef...@gmail.com> wrote:
> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <ef...@gmail.com>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Sandy Ryza <sa...@cloudera.com>.
Average map time includes everything the map task is doing, i.e. all the
things you mentioned. Reduce time does not cover shuffle time. Reduce
time is the time spent calling the reducer function and writing its output
to HDFS. Merge time is related to reduce, not map.
-Sandy
On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <ef...@gmail.com> wrote:
> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <ef...@gmail.com>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Efe Gencer <ef...@gmail.com>.
*By the way this question is about Apache Hadoop Release 2.1.0-beta.
Thanks,
2013/9/24 Efe Gencer <ef...@gmail.com>
> Hi All,
>
> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
> ...
> Started: <start time>
> Finished: <finish time>
> Elapsed: 12 mins, 5sec
> Diagnostics:
> *Average Map Time*: 1 mins, 40 sec
> Average Reduce Time: 12 sec
> Average Shuffle Time: 10 mins, 8 sec
> Average Merge Time: 1 sec
> ...
>
> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
> to hdfs, etc)
> 3) Is Average Merge time related with map or reduce? (since they both have
> merge phases)
>
> Best,
> Efe
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Efe Gencer <ef...@gmail.com>.
*By the way this question is about Apache Hadoop Release 2.1.0-beta.
Thanks,
2013/9/24 Efe Gencer <ef...@gmail.com>
> Hi All,
>
> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
> ...
> Started: <start time>
> Finished: <finish time>
> Elapsed: 12 mins, 5sec
> Diagnostics:
> *Average Map Time*: 1 mins, 40 sec
> Average Reduce Time: 12 sec
> Average Shuffle Time: 10 mins, 8 sec
> Average Merge Time: 1 sec
> ...
>
> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
> to hdfs, etc)
> 3) Is Average Merge time related with map or reduce? (since they both have
> merge phases)
>
> Best,
> Efe
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Efe Gencer <ef...@gmail.com>.
*By the way this question is about Apache Hadoop Release 2.1.0-beta.
Thanks,
2013/9/24 Efe Gencer <ef...@gmail.com>
> Hi All,
>
> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
> ...
> Started: <start time>
> Finished: <finish time>
> Elapsed: 12 mins, 5sec
> Diagnostics:
> *Average Map Time*: 1 mins, 40 sec
> Average Reduce Time: 12 sec
> Average Shuffle Time: 10 mins, 8 sec
> Average Merge Time: 1 sec
> ...
>
> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
> to hdfs, etc)
> 3) Is Average Merge time related with map or reduce? (since they both have
> merge phases)
>
> Best,
> Efe
>
Re: Which Subphases Do Times on JobHistory Web UI Cover
Posted by Efe Gencer <ef...@gmail.com>.
*By the way this question is about Apache Hadoop Release 2.1.0-beta.
Thanks,
2013/9/24 Efe Gencer <ef...@gmail.com>
> Hi All,
>
> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
> ...
> Started: <start time>
> Finished: <finish time>
> Elapsed: 12 mins, 5sec
> Diagnostics:
> *Average Map Time*: 1 mins, 40 sec
> Average Reduce Time: 12 sec
> Average Shuffle Time: 10 mins, 8 sec
> Average Merge Time: 1 sec
> ...
>
> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
> to hdfs, etc)
> 3) Is Average Merge time related with map or reduce? (since they both have
> merge phases)
>
> Best,
> Efe
>