You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Torsten Curdt <tc...@vafer.org> on 2010/06/06 19:33:50 UTC
number of reducers
When I set
job.setPartitionerClass(MyPartitioner.class);
job.setNumReduceTasks(4);
I would expect to see my MyParitioner get called with
getPartition(key, value, 4)
but still I see it only get called with 1.
If also tried setting
conf.set("mapred.map.tasks.speculative.execution", "false");
conf.set("mapred.reduce.tasks.speculative.execution", "false");
What am I missing here?
cheers
--
Torsten
Re: number of reducers
Posted by Torsten Curdt <tc...@vafer.org>.
Great. Just thought I'd be missing something :)
On Mon, Jun 7, 2010 at 01:55, Aaron Kimball <aa...@cloudera.com> wrote:
> Yes. LJR sets your number of reduce tasks to 1 if the number is >= 1.
> Subnote: I've posted a patch to fix this at MAPREDUCE-434, but it's not
> committed.
> - Aaron
>
> On Mon, Jun 7, 2010 at 1:42 AM, Torsten Curdt <tc...@vafer.org> wrote:
>>
>> I see only one.
>>
>> Could it be that using the LocalJobRunner interferes here?
>>
>> On Mon, Jun 7, 2010 at 01:31, Eric Sammer <es...@cloudera.com> wrote:
>> > Torsten:
>> >
>> > To clarify, how many reducers do you actually see? (i.e. Do you see 4
>> > reducers or 1?) It should work as you expect.
>> >
>> > On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tc...@vafer.org> wrote:
>> >> When I set
>> >>
>> >> job.setPartitionerClass(MyPartitioner.class);
>> >> job.setNumReduceTasks(4);
>> >>
>> >> I would expect to see my MyParitioner get called with
>> >>
>> >> getPartition(key, value, 4)
>> >>
>> >> but still I see it only get called with 1.
>> >>
>> >> If also tried setting
>> >>
>> >> conf.set("mapred.map.tasks.speculative.execution",
>> >> "false");
>> >> conf.set("mapred.reduce.tasks.speculative.execution",
>> >> "false");
>> >>
>> >> What am I missing here?
>> >>
>> >> cheers
>> >> --
>> >> Torsten
>> >>
>> >
>> >
>> >
>> > --
>> > Eric Sammer
>> > phone: +1-917-287-2675
>> > twitter: esammer
>> > data: www.cloudera.com
>> >
>
>
Re: number of reducers
Posted by Aaron Kimball <aa...@cloudera.com>.
Yes. LJR sets your number of reduce tasks to 1 if the number is >= 1.
Subnote: I've posted a patch to fix this at MAPREDUCE-434, but it's not
committed.
- Aaron
On Mon, Jun 7, 2010 at 1:42 AM, Torsten Curdt <tc...@vafer.org> wrote:
> I see only one.
>
> Could it be that using the LocalJobRunner interferes here?
>
> On Mon, Jun 7, 2010 at 01:31, Eric Sammer <es...@cloudera.com> wrote:
> > Torsten:
> >
> > To clarify, how many reducers do you actually see? (i.e. Do you see 4
> > reducers or 1?) It should work as you expect.
> >
> > On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tc...@vafer.org> wrote:
> >> When I set
> >>
> >> job.setPartitionerClass(MyPartitioner.class);
> >> job.setNumReduceTasks(4);
> >>
> >> I would expect to see my MyParitioner get called with
> >>
> >> getPartition(key, value, 4)
> >>
> >> but still I see it only get called with 1.
> >>
> >> If also tried setting
> >>
> >> conf.set("mapred.map.tasks.speculative.execution",
> "false");
> >> conf.set("mapred.reduce.tasks.speculative.execution",
> "false");
> >>
> >> What am I missing here?
> >>
> >> cheers
> >> --
> >> Torsten
> >>
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>
Re: number of reducers
Posted by Torsten Curdt <tc...@vafer.org>.
I see only one.
Could it be that using the LocalJobRunner interferes here?
On Mon, Jun 7, 2010 at 01:31, Eric Sammer <es...@cloudera.com> wrote:
> Torsten:
>
> To clarify, how many reducers do you actually see? (i.e. Do you see 4
> reducers or 1?) It should work as you expect.
>
> On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tc...@vafer.org> wrote:
>> When I set
>>
>> job.setPartitionerClass(MyPartitioner.class);
>> job.setNumReduceTasks(4);
>>
>> I would expect to see my MyParitioner get called with
>>
>> getPartition(key, value, 4)
>>
>> but still I see it only get called with 1.
>>
>> If also tried setting
>>
>> conf.set("mapred.map.tasks.speculative.execution", "false");
>> conf.set("mapred.reduce.tasks.speculative.execution", "false");
>>
>> What am I missing here?
>>
>> cheers
>> --
>> Torsten
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>
Re: number of reducers
Posted by Eric Sammer <es...@cloudera.com>.
Torsten:
To clarify, how many reducers do you actually see? (i.e. Do you see 4
reducers or 1?) It should work as you expect.
On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tc...@vafer.org> wrote:
> When I set
>
> job.setPartitionerClass(MyPartitioner.class);
> job.setNumReduceTasks(4);
>
> I would expect to see my MyParitioner get called with
>
> getPartition(key, value, 4)
>
> but still I see it only get called with 1.
>
> If also tried setting
>
> conf.set("mapred.map.tasks.speculative.execution", "false");
> conf.set("mapred.reduce.tasks.speculative.execution", "false");
>
> What am I missing here?
>
> cheers
> --
> Torsten
>
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com