You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Naama Kraus <na...@gmail.com> on 2008/03/06 14:03:47 UTC
Difference between local mode and distributed mode
Hi,
I ran a simple MapReduce job which defines 3 reducers:
*conf.setNumReduceTasks(3);*
When running on top of HDFS (distributed mode), I got 3 out files as I
expected.
When running on top of a local files system (local mode), I got 1 file and
not 3.
My question is whether the behavior in the local is the expected one, is a
bug, or maybe something needs to be configured in the local mode in order to
get the 3 files as well ?
Thanks for an insight, Naama
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: Difference between local mode and distributed mode
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Mar 6, 2008, at 5:23 AM, Naama Kraus wrote:
> Thanks, that's fine with me then. Naama
>
Please note that you can also run hadoop in a 'psuedo-distributed'
mode on a single node where you can mimic all characteristics (all?
except, err... performance of course!) of a large cluster:
http://hadoop.apache.org/core/docs/r0.16.0/
quickstart.html#SingleNodeSetup
Arun
> On Thu, Mar 6, 2008 at 3:21 PM, Enis Soztutar
> <en...@gmail.com>
> wrote:
>
>> Hi,
>>
>> LocalJobRunner uses just 0 or 1 reduce. This is because running in
>> local
>> mode is only supported for testing purposes.
>> Although you can simulate distribute mode in local, by using
>> MiniMRCluster and MiniDFSCluster under src/test.
>>
>> Best wishes
>> Enis
>>
>> Naama Kraus wrote:
>>> Hi,
>>>
>>> I ran a simple MapReduce job which defines 3 reducers:
>>>
>>> *conf.setNumReduceTasks(3);*
>>>
>>> When running on top of HDFS (distributed mode), I got 3 out files
>>> as I
>>> expected.
>>> When running on top of a local files system (local mode), I got 1
>>> file
>> and
>>> not 3.
>>>
>>> My question is whether the behavior in the local is the expected
>>> one, is
>> a
>>> bug, or maybe something needs to be configured in the local mode in
>> order to
>>> get the 3 files as well ?
>>>
>>> Thanks for an insight, Naama
>>>
>>>
>>
>
>
>
> --
> oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00
> oo 00 oo
> 00 oo 00 oo
> "If you want your children to be intelligent, read them fairy
> tales. If you
> want them to be more intelligent, read them more fairy tales." (Albert
> Einstein)
Re: Difference between local mode and distributed mode
Posted by Naama Kraus <na...@gmail.com>.
Thanks, that's fine with me then. Naama
On Thu, Mar 6, 2008 at 3:21 PM, Enis Soztutar <en...@gmail.com>
wrote:
> Hi,
>
> LocalJobRunner uses just 0 or 1 reduce. This is because running in local
> mode is only supported for testing purposes.
> Although you can simulate distribute mode in local, by using
> MiniMRCluster and MiniDFSCluster under src/test.
>
> Best wishes
> Enis
>
> Naama Kraus wrote:
> > Hi,
> >
> > I ran a simple MapReduce job which defines 3 reducers:
> >
> > *conf.setNumReduceTasks(3);*
> >
> > When running on top of HDFS (distributed mode), I got 3 out files as I
> > expected.
> > When running on top of a local files system (local mode), I got 1 file
> and
> > not 3.
> >
> > My question is whether the behavior in the local is the expected one, is
> a
> > bug, or maybe something needs to be configured in the local mode in
> order to
> > get the 3 files as well ?
> >
> > Thanks for an insight, Naama
> >
> >
>
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: Difference between local mode and distributed mode
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Mar 6, 2008, at 5:21 AM, Enis Soztutar wrote:
> Hi,
>
> LocalJobRunner uses just 0 or 1 reduce. This is because running in
> local mode is only supported for testing purposes.
This is a very long standing design bug. (HADOOP-206)
-- Owen
Re: Difference between local mode and distributed mode
Posted by Enis Soztutar <en...@gmail.com>.
Hi,
LocalJobRunner uses just 0 or 1 reduce. This is because running in local
mode is only supported for testing purposes.
Although you can simulate distribute mode in local, by using
MiniMRCluster and MiniDFSCluster under src/test.
Best wishes
Enis
Naama Kraus wrote:
> Hi,
>
> I ran a simple MapReduce job which defines 3 reducers:
>
> *conf.setNumReduceTasks(3);*
>
> When running on top of HDFS (distributed mode), I got 3 out files as I
> expected.
> When running on top of a local files system (local mode), I got 1 file and
> not 3.
>
> My question is whether the behavior in the local is the expected one, is a
> bug, or maybe something needs to be configured in the local mode in order to
> get the 3 files as well ?
>
> Thanks for an insight, Naama
>
>