You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by xeonmailinglist-gmail <xe...@gmail.com> on 2015/03/09 15:56:23 UTC
How reduce tasks know which partition they should read?
Hi,
I am looking to the Yarn mapreduce internals to try to understand how
reduce tasks know which partition of the map output they should read.
Even, when they re-execute after a crash?
I am also looking to the mapreduce source code. Is there any class that
I should look to try to understand this question?
Any help?
Thanks
--
--
Re: How reduce tasks know which partition they should read?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.
Thanks,
+Vinod
On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:
> Hi,
>
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
>
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
>
> Any help?
>
> Thanks
>
>
> --
> --
>
Re: How reduce tasks know which partition they should read?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.
Thanks,
+Vinod
On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:
> Hi,
>
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
>
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
>
> Any help?
>
> Thanks
>
>
> --
> --
>
Re: How reduce tasks know which partition they should read?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.
Thanks,
+Vinod
On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:
> Hi,
>
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
>
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
>
> Any help?
>
> Thanks
>
>
> --
> --
>
Re: How reduce tasks know which partition they should read?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.
Thanks,
+Vinod
On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:
> Hi,
>
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
>
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
>
> Any help?
>
> Thanks
>
>
> --
> --
>