You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Pavan Kulkarni <pa...@gmail.com> on 2012/06/27 18:30:27 UTC
Doubt regarding use of HTTP during Shuffle phase
Hi,
I am working on a project where I am trying to create hardlinks for the
Reducer nodes to the intermediate output files after Map phase during the
Shuffle phase
instead of copying the files via HTTP which I assume would be more
efficient.
I had few doubts regarding this. Why is HTTP used during the Shuffle phase
instead of just creating a Hardlink?
and Can you give pointers towards how do I create these hardlinks? I am
looking for where exactly can the intermediate Mapoutput Filename be
fetched?
Any help is highly appreciated .Thanks
--
--With Regards
Pavan Kulkarni
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Pavan Kulkarni <pa...@gmail.com>.
Oh.Thanks a lot Owen.
I'll have a look into it.
On Wed, Jun 27, 2012 at 10:21 AM, Owen O'Malley <om...@apache.org> wrote:
> Pavan,
> This is a very big project. Look at the users of IFile.java. IFile is the
> format for storing the shuffle outputs.
>
> -- Owen
>
--
--With Regards
Pavan Kulkarni
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Owen O'Malley <om...@apache.org>.
Pavan,
This is a very big project. Look at the users of IFile.java. IFile is the
format for storing the shuffle outputs.
-- Owen
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Pavan Kulkarni <pa...@gmail.com>.
@Roman: Exactly these are the papers I referred before venturing into this
project.
Thanks a lot for the sailfish paper.Wasn't aware of that.
Also by any chance do you have any idea what classes do I need to tweak to
get the Filenames of Map Output files.?Thanks
On Wed, Jun 27, 2012 at 9:52 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <pa...@gmail.com>
> wrote:
> > Yes you are correct, but we can use Lustre FS and it does scale right?
> > I am new to this so please excuse if I am wrong in some assumptions.
>
> You can use hybrid approaches, but you'd be venturing into the unknown.
> Take a look at these projects first:
> http://code.google.com/p/sailfish/
>
> http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf
>
> Thanks,
> Roman.
>
--
--With Regards
Pavan Kulkarni
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <pa...@gmail.com> wrote:
> Yes you are correct, but we can use Lustre FS and it does scale right?
> I am new to this so please excuse if I am wrong in some assumptions.
You can use hybrid approaches, but you'd be venturing into the unknown.
Take a look at these projects first:
http://code.google.com/p/sailfish/
http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf
Thanks,
Roman.
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Pavan Kulkarni <pa...@gmail.com>.
Yes you are correct, but we can use Lustre FS and it does scale right?
I am new to this so please excuse if I am wrong in some assumptions.Thanks
On Wed, Jun 27, 2012 at 9:40 AM, Owen O'Malley <om...@apache.org> wrote:
> On Wed, Jun 27, 2012 at 9:33 AM, Pavan Kulkarni <pavan.baburao@gmail.com
> >wrote:
>
> Why is HTTP used during the Shuffle phase
> > instead of just creating a Hardlink?
> >
>
> The reduces run on a separate machine that the map. Hadoop doesn't use a
> shared NFS file system between the machines, because it doesn't scale.
>
> -- Owen
>
--
--With Regards
Pavan Kulkarni
Re: Doubt regarding use of HTTP during Shuffle phase
Posted by Owen O'Malley <om...@apache.org>.
On Wed, Jun 27, 2012 at 9:33 AM, Pavan Kulkarni <pa...@gmail.com>wrote:
Why is HTTP used during the Shuffle phase
> instead of just creating a Hardlink?
>
The reduces run on a separate machine that the map. Hadoop doesn't use a
shared NFS file system between the machines, because it doesn't scale.
-- Owen
Doubt regarding use of HTTP during Shuffle phase
Posted by Pavan Kulkarni <pa...@gmail.com>.
Hi,
I am working on a project where I am trying to create hardlinks for the
Reducer nodes to the intermediate output files after Map phase during the
Shuffle phase
instead of copying the files via HTTP which I assume would be more
efficient.
I had few doubts regarding this. Why is HTTP used during the Shuffle phase
instead of just creating a Hardlink?
and Can you give pointers towards how do I create these hardlinks? I am
looking for where exactly can the intermediate Mapoutput Filename be
fetched?
Any help is highly appreciated .Thanks
--
--With Regards
Pavan Kulkarni