You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Grandl Robert <rg...@yahoo.com> on 2013/06/11 02:46:06 UTC
In memory shuffle phase
Hi,
It is possible to run shuffle phase in memory ? I mean, the mapper to write the intermediate output in memory only and the reducer to read it from there. So no disk involved for intermediate output.
If so, what are the configuration parameters I need to tune ?
Thanks,
Robert
Re: In memory shuffle phase
Posted by Radim Kolar <hs...@filez.com>.
Dne 11.6.2013 2:46, Grandl Robert napsal(a):
> Hi,
>
>
> It is possible to run shuffle phase in memory ? I mean, the mapper to write the intermediate output in memory only and the reducer to read it from there. So no disk involved for intermediate output.
I do not think that this is possible in hadoop but its pretty common in
other mapreduce software. They can even chain map-reduce-map without
writing to disk and moving it across network to different node.
Re: In memory shuffle phase
Posted by 牛兆捷 <nz...@gmail.com>.
*import* *org**.apache.tez.dag.api.records*.DAGProtos.DAGPlan;
why this import errors in tez trunk ?
2013/6/14 Arun C Murthy <ac...@hortonworks.com>
> https://issues.apache.org/jira/browse/TEZ-65
>
> Arun
>
> On Jun 12, 2013, at 12:28 AM, 牛兆捷 wrote:
>
> > Is there any design document about tez?
> >
> > 2013/6/12 Arun C Murthy <ac...@hortonworks.com>
> >
> >> Take a look at Apache Tez (
> http://incubator.apache.org/projects/tez.html).
> >>
> >> I'll warn you that it's a little rough, but works.
> >>
> >> Arun
> >>
> >> On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:
> >>
> >>> Hi,
> >>>
> >>>
> >>> It is possible to run shuffle phase in memory ? I mean, the mapper to
> >> write the intermediate output in memory only and the reducer to read it
> >> from there. So no disk involved for intermediate output.
> >>>
> >>>
> >>> If so, what are the configuration parameters I need to tune ?
> >>>
> >>> Thanks,
> >>> Robert
> >>
> >> --
> >> Arun C. Murthy
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >>
> >>
> >
> >
> > --
> > *Sincerely,*
> > *Zhaojie*
> > *
> > *
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
--
*Sincerely,*
*Zhaojie*
*
*
Re: In memory shuffle phase
Posted by Arun C Murthy <ac...@hortonworks.com>.
https://issues.apache.org/jira/browse/TEZ-65
Arun
On Jun 12, 2013, at 12:28 AM, 牛兆捷 wrote:
> Is there any design document about tez?
>
> 2013/6/12 Arun C Murthy <ac...@hortonworks.com>
>
>> Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html).
>>
>> I'll warn you that it's a little rough, but works.
>>
>> Arun
>>
>> On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:
>>
>>> Hi,
>>>
>>>
>>> It is possible to run shuffle phase in memory ? I mean, the mapper to
>> write the intermediate output in memory only and the reducer to read it
>> from there. So no disk involved for intermediate output.
>>>
>>>
>>> If so, what are the configuration parameters I need to tune ?
>>>
>>> Thanks,
>>> Robert
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
>
> --
> *Sincerely,*
> *Zhaojie*
> *
> *
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: In memory shuffle phase
Posted by Zhijie Shen <zs...@hortonworks.com>.
You may want to have a look at the following webpage:
http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/
On Wed, Jun 12, 2013 at 12:28 AM, 牛兆捷 <nz...@gmail.com> wrote:
> Is there any design document about tez?
>
> 2013/6/12 Arun C Murthy <ac...@hortonworks.com>
>
> > Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html
> ).
> >
> > I'll warn you that it's a little rough, but works.
> >
> > Arun
> >
> > On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:
> >
> > > Hi,
> > >
> > >
> > > It is possible to run shuffle phase in memory ? I mean, the mapper to
> > write the intermediate output in memory only and the reducer to read it
> > from there. So no disk involved for intermediate output.
> > >
> > >
> > > If so, what are the configuration parameters I need to tune ?
> > >
> > > Thanks,
> > > Robert
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
>
> --
> *Sincerely,*
> *Zhaojie*
> *
> *
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
Re: In memory shuffle phase
Posted by 牛兆捷 <nz...@gmail.com>.
Is there any design document about tez?
2013/6/12 Arun C Murthy <ac...@hortonworks.com>
> Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html).
>
> I'll warn you that it's a little rough, but works.
>
> Arun
>
> On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:
>
> > Hi,
> >
> >
> > It is possible to run shuffle phase in memory ? I mean, the mapper to
> write the intermediate output in memory only and the reducer to read it
> from there. So no disk involved for intermediate output.
> >
> >
> > If so, what are the configuration parameters I need to tune ?
> >
> > Thanks,
> > Robert
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
--
*Sincerely,*
*Zhaojie*
*
*
Re: In memory shuffle phase
Posted by Arun C Murthy <ac...@hortonworks.com>.
Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html).
I'll warn you that it's a little rough, but works.
Arun
On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:
> Hi,
>
>
> It is possible to run shuffle phase in memory ? I mean, the mapper to write the intermediate output in memory only and the reducer to read it from there. So no disk involved for intermediate output.
>
>
> If so, what are the configuration parameters I need to tune ?
>
> Thanks,
> Robert
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/