You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by xeonmailinglist-gmail <xe...@gmail.com> on 2015/03/09 15:56:23 UTC

How reduce tasks know which partition they should read?

Hi,

I am looking to the Yarn mapreduce internals to try to understand how 
reduce tasks know which partition of the map output they should read. 
Even, when they re-execute after a crash?

I am also looking to the mapreduce source code. Is there any class that 
I should look to try to understand this question?

Any help?

Thanks


-- 
--

Re: How reduce tasks know which partition they should read?

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.

Thanks,
+Vinod

On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:

> Hi,
> 
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
> 
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
> 
> Any help?
> 
> Thanks
> 
> 
> -- 
> --
>

Re: How reduce tasks know which partition they should read?

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.

Thanks,
+Vinod

On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:

> Hi,
> 
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
> 
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
> 
> Any help?
> 
> Thanks
> 
> 
> -- 
> --
>

Re: How reduce tasks know which partition they should read?

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.

Thanks,
+Vinod

On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:

> Hi,
> 
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
> 
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
> 
> Any help?
> 
> Thanks
> 
> 
> -- 
> --
>

Re: How reduce tasks know which partition they should read?

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them output corresponding to a specific map. The partitioning detail is hidden from the reducers.

Thanks,
+Vinod

On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xe...@gmail.com> wrote:

> Hi,
> 
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know which partition of the map output they should read. Even, when they re-execute after a crash?
> 
> I am also looking to the mapreduce source code. Is there any class that I should look to try to understand this question?
> 
> Any help?
> 
> Thanks
> 
> 
> -- 
> --
>