You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Saurabh Bajaj <Sa...@mu-sigma.com> on 2012/08/02 12:25:13 UTC
All reducers are not being utilized
Hi everyone,
I was running a MR job in java and this scenario happened:
Case 1:
Number of distinct output keys from mapper = 3
Expected # of reducers = 3
Defined set # of reducers to be called = 2
Expected outcome:
# of reducers spawned = 2
# of keys processed under first reducer = 1
# of keys processed under second reducer = 2
Observed outcome:
# of keys processed under first reducer = 3
# of keys processed under second reducer = 0
Case 2:
Number of distinct output keys from mapper = 3
Expected # of reducers = 3
Defined set # of reducers to be called = 3
Expected outcome:
# of reducers spawned = 3
# of keys processed under first reducer = 1
# of keys processed under second reducer = 1
# of keys processed under third reducer = 1
Observed outcome:
# of reducers spawned = 3
# of keys processed under first reducer = 2
# of keys processed under second reducer = 0
# of keys processed under third reducer = 1
Any idea why all the reducers are not utilized?
Saurabh Bajaj | Senior Business Analyst | +91 9986588089 | www.mu-sigma.com<http://www.mu-sigma.com/> |
________________________________
This email message may contain proprietary, private and confidential information. The information transmitted is intended only for the person(s) or entities to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be illegal. If you received this in error, please contact the sender and delete the message from your system.
Mu Sigma takes all reasonable steps to ensure that its electronic communications are free from viruses. However, given Internet accessibility, the Company cannot accept liability for any virus introduced by this e-mail or any attachment and you are advised to use up-to-date virus checking software.
Re: All reducers are not being utilized
Posted by Steve Sonnenberg <st...@gmail.com>.
If I have 2 nodes, and 150 input files in a single 'input' directory to
search using the 'grep' example, isn't it reasonable that both nodes would
be involved?
Thanks
On Thu, Aug 2, 2012 at 3:31 PM, Bejoy Ks <be...@gmail.com> wrote:
> Hi Saurab/Steve
>
> From my understanding the schedulers in hadoop consider only data
> locality(for map tasks) and availability of slots for scheduling tasks on
> various nodes. Say if you have a 3 TT nodes with 2 reducer slots each
> (assume all slots are free) . If we execute a map reduce job with 3 reduce
> tasks there is no gaurentee that one task will be scheduled on each node.
> It can be like 2 in one node and 1 in another.
>
> Regards
> Bejoy KS
>
--
Steve Sonnenberg
Re: All reducers are not being utilized
Posted by Bejoy Ks <be...@gmail.com>.
Hi Saurab/Steve
>From my understanding the schedulers in hadoop consider only data
locality(for map tasks) and availability of slots for scheduling tasks on
various nodes. Say if you have a 3 TT nodes with 2 reducer slots each
(assume all slots are free) . If we execute a map reduce job with 3 reduce
tasks there is no gaurentee that one task will be scheduled on each node.
It can be like 2 in one node and 1 in another.
Regards
Bejoy KS
Re: All reducers are not being utilized
Posted by Steve Sonnenberg <st...@gmail.com>.
Harsh,
This may be similar to the case I was observing, when no tasks were created
on my 2nd node. It was the simple 'grep' example with only 6 input files.
-steve
On Thu, Aug 2, 2012 at 7:13 AM, Harsh J <ha...@cloudera.com> wrote:
> If you are speaking of Reduce tasks, then schedulers do try to spread them
> across (by attempting to schedule only 1 reduce task per heartbeat/round).
>
> If you speak of data distribution equality, that depends on your data
> (remember that reducers process key groups, and keys can carry skews
> depending on input data IRL) and your partitioner
> implementation/partitioning needs.
>
>
> On Thu, Aug 2, 2012 at 4:35 PM, Saurabh Bajaj <Sa...@mu-sigma.com>wrote:
>
>> Okay. One more question.****
>>
>> Can we somehow make sure that all the reducers are equally split across
>> all the nodes? ****
>>
>> ** **
>>
>> Saurabh****
>>
>> ** **
>>
>> *From:* Harsh J [mailto:harsh@cloudera.com]
>> *Sent:* Thursday, August 02, 2012 4:05 PM
>> *To:* mapreduce-user@hadoop.apache.org
>> *Subject:* Re: All reducers are not being utilized****
>>
>> ** **
>>
>> Saurabh,****
>>
>> ** **
>>
>> I do not see you talk about defining a custom Partitioner that can
>> guarantee such perfect key distribution. The default partitioner is the
>> HashPartitioner that can only guarantee randomized distribution (as it is
>> key data specific). Hence, your test here with just 3 keys is not really a
>> good way to test key distribution to reducers with a HashPartition. Try it
>> out with a large data set to see for real.****
>>
>> On Thu, Aug 2, 2012 at 3:55 PM, Saurabh Bajaj <Sa...@mu-sigma.com>
>> wrote:****
>>
>> Hi everyone, ****
>>
>> ****
>>
>> I was running a MR job in java and this scenario happened:****
>>
>> ****
>>
>> *Case 1:*****
>>
>> Number of distinct output keys from mapper = 3****
>>
>> Expected # of reducers = 3****
>>
>> Defined set # of reducers to be called = 2****
>>
>> *Expected outcome:*****
>>
>> # of reducers spawned = 2****
>>
>> # of keys processed under first reducer = 1****
>>
>> # of keys processed under second reducer = 2****
>>
>> *Observed outcome:*****
>>
>> # of keys processed under first reducer = 3****
>>
>> # of keys processed under second reducer = 0****
>>
>> ****
>>
>> ****
>>
>> ****
>>
>> *Case 2:*****
>>
>> Number of distinct output keys from mapper = 3****
>>
>> Expected # of reducers = 3****
>>
>> Defined set # of reducers to be called = 3****
>>
>> *Expected outcome:*****
>>
>> # of reducers spawned = 3****
>>
>> # of keys processed under first reducer = 1****
>>
>> # of keys processed under second reducer = 1****
>>
>> # of keys processed under third reducer = 1****
>>
>> * *****
>>
>> *Observed outcome:*****
>>
>> # of reducers spawned = 3****
>>
>> # of keys processed under first reducer = 2****
>>
>> # of keys processed under second reducer = 0****
>>
>> # of keys processed under third reducer = 1****
>>
>> ****
>>
>> ****
>>
>> Any idea why all the reducers are not utilized?****
>>
>> ****
>>
>> Saurabh Bajaj *|* Senior Business Analyst *|* +91 9986588089 *|*
>> www.mu-sigma.com* **|*****
>>
>> ****
>>
>> ** **
>> ------------------------------
>>
>> This email message may contain proprietary, private and confidential
>> information. The information transmitted is intended only for the person(s)
>> or entities to which it is addressed. Any review, retransmission,
>> dissemination or other use of, or taking of any action in reliance upon,
>> this information by persons or entities other than the intended recipient
>> is prohibited and may be illegal. If you received this in error, please
>> contact the sender and delete the message from your system.
>>
>> Mu Sigma takes all reasonable steps to ensure that its electronic
>> communications are free from viruses. However, given Internet
>> accessibility, the Company cannot accept liability for any virus introduced
>> by this e-mail or any attachment and you are advised to use up-to-date
>> virus checking software.****
>>
>>
>>
>> ****
>>
>> ** **
>>
>> --
>> Harsh J****
>>
>> ------------------------------
>> This email message may contain proprietary, private and confidential
>> information. The information transmitted is intended only for the person(s)
>> or entities to which it is addressed. Any review, retransmission,
>> dissemination or other use of, or taking of any action in reliance upon,
>> this information by persons or entities other than the intended recipient
>> is prohibited and may be illegal. If you received this in error, please
>> contact the sender and delete the message from your system.
>>
>> Mu Sigma takes all reasonable steps to ensure that its electronic
>> communications are free from viruses. However, given Internet
>> accessibility, the Company cannot accept liability for any virus introduced
>> by this e-mail or any attachment and you are advised to use up-to-date
>> virus checking software.
>>
>
>
>
> --
> Harsh J
>
--
Steve Sonnenberg
Re: All reducers are not being utilized
Posted by Harsh J <ha...@cloudera.com>.
If you are speaking of Reduce tasks, then schedulers do try to spread them
across (by attempting to schedule only 1 reduce task per heartbeat/round).
If you speak of data distribution equality, that depends on your data
(remember that reducers process key groups, and keys can carry skews
depending on input data IRL) and your partitioner
implementation/partitioning needs.
On Thu, Aug 2, 2012 at 4:35 PM, Saurabh Bajaj <Sa...@mu-sigma.com>wrote:
> Okay. One more question.****
>
> Can we somehow make sure that all the reducers are equally split across
> all the nodes? ****
>
> ** **
>
> Saurabh****
>
> ** **
>
> *From:* Harsh J [mailto:harsh@cloudera.com]
> *Sent:* Thursday, August 02, 2012 4:05 PM
> *To:* mapreduce-user@hadoop.apache.org
> *Subject:* Re: All reducers are not being utilized****
>
> ** **
>
> Saurabh,****
>
> ** **
>
> I do not see you talk about defining a custom Partitioner that can
> guarantee such perfect key distribution. The default partitioner is the
> HashPartitioner that can only guarantee randomized distribution (as it is
> key data specific). Hence, your test here with just 3 keys is not really a
> good way to test key distribution to reducers with a HashPartition. Try it
> out with a large data set to see for real.****
>
> On Thu, Aug 2, 2012 at 3:55 PM, Saurabh Bajaj <Sa...@mu-sigma.com>
> wrote:****
>
> Hi everyone, ****
>
> ****
>
> I was running a MR job in java and this scenario happened:****
>
> ****
>
> *Case 1:*****
>
> Number of distinct output keys from mapper = 3****
>
> Expected # of reducers = 3****
>
> Defined set # of reducers to be called = 2****
>
> *Expected outcome:*****
>
> # of reducers spawned = 2****
>
> # of keys processed under first reducer = 1****
>
> # of keys processed under second reducer = 2****
>
> *Observed outcome:*****
>
> # of keys processed under first reducer = 3****
>
> # of keys processed under second reducer = 0****
>
> ****
>
> ****
>
> ****
>
> *Case 2:*****
>
> Number of distinct output keys from mapper = 3****
>
> Expected # of reducers = 3****
>
> Defined set # of reducers to be called = 3****
>
> *Expected outcome:*****
>
> # of reducers spawned = 3****
>
> # of keys processed under first reducer = 1****
>
> # of keys processed under second reducer = 1****
>
> # of keys processed under third reducer = 1****
>
> * *****
>
> *Observed outcome:*****
>
> # of reducers spawned = 3****
>
> # of keys processed under first reducer = 2****
>
> # of keys processed under second reducer = 0****
>
> # of keys processed under third reducer = 1****
>
> ****
>
> ****
>
> Any idea why all the reducers are not utilized?****
>
> ****
>
> Saurabh Bajaj *|* Senior Business Analyst *|* +91 9986588089 *|*
> www.mu-sigma.com* **|*****
>
> ****
>
> ** **
> ------------------------------
>
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited and may be illegal. If you received this in error, please
> contact the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet
> accessibility, the Company cannot accept liability for any virus introduced
> by this e-mail or any attachment and you are advised to use up-to-date
> virus checking software.****
>
>
>
> ****
>
> ** **
>
> --
> Harsh J****
>
> ------------------------------
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited and may be illegal. If you received this in error, please
> contact the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet
> accessibility, the Company cannot accept liability for any virus introduced
> by this e-mail or any attachment and you are advised to use up-to-date
> virus checking software.
>
--
Harsh J
RE: All reducers are not being utilized
Posted by Saurabh Bajaj <Sa...@mu-sigma.com>.
Okay. One more question.
Can we somehow make sure that all the reducers are equally split across all the nodes?
Saurabh
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Thursday, August 02, 2012 4:05 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: All reducers are not being utilized
Saurabh,
I do not see you talk about defining a custom Partitioner that can guarantee such perfect key distribution. The default partitioner is the HashPartitioner that can only guarantee randomized distribution (as it is key data specific). Hence, your test here with just 3 keys is not really a good way to test key distribution to reducers with a HashPartition. Try it out with a large data set to see for real.
On Thu, Aug 2, 2012 at 3:55 PM, Saurabh Bajaj <Sa...@mu-sigma.com>> wrote:
Hi everyone,
I was running a MR job in java and this scenario happened:
Case 1:
Number of distinct output keys from mapper = 3
Expected # of reducers = 3
Defined set # of reducers to be called = 2
Expected outcome:
# of reducers spawned = 2
# of keys processed under first reducer = 1
# of keys processed under second reducer = 2
Observed outcome:
# of keys processed under first reducer = 3
# of keys processed under second reducer = 0
Case 2:
Number of distinct output keys from mapper = 3
Expected # of reducers = 3
Defined set # of reducers to be called = 3
Expected outcome:
# of reducers spawned = 3
# of keys processed under first reducer = 1
# of keys processed under second reducer = 1
# of keys processed under third reducer = 1
Observed outcome:
# of reducers spawned = 3
# of keys processed under first reducer = 2
# of keys processed under second reducer = 0
# of keys processed under third reducer = 1
Any idea why all the reducers are not utilized?
Saurabh Bajaj | Senior Business Analyst | +91 9986588089<tel:%2B91%209986588089> | www.mu-sigma.com<http://www.mu-sigma.com/> |
________________________________
This email message may contain proprietary, private and confidential information. The information transmitted is intended only for the person(s) or entities to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be illegal. If you received this in error, please contact the sender and delete the message from your system.
Mu Sigma takes all reasonable steps to ensure that its electronic communications are free from viruses. However, given Internet accessibility, the Company cannot accept liability for any virus introduced by this e-mail or any attachment and you are advised to use up-to-date virus checking software.
--
Harsh J
________________________________
This email message may contain proprietary, private and confidential information. The information transmitted is intended only for the person(s) or entities to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be illegal. If you received this in error, please contact the sender and delete the message from your system.
Mu Sigma takes all reasonable steps to ensure that its electronic communications are free from viruses. However, given Internet accessibility, the Company cannot accept liability for any virus introduced by this e-mail or any attachment and you are advised to use up-to-date virus checking software.
Re: All reducers are not being utilized
Posted by Harsh J <ha...@cloudera.com>.
Saurabh,
I do not see you talk about defining a custom Partitioner that can
guarantee such perfect key distribution. The default partitioner is the
HashPartitioner that can only guarantee randomized distribution (as it is
key data specific). Hence, your test here with just 3 keys is not really a
good way to test key distribution to reducers with a HashPartition. Try it
out with a large data set to see for real.
On Thu, Aug 2, 2012 at 3:55 PM, Saurabh Bajaj <Sa...@mu-sigma.com>wrote:
> Hi everyone, ****
>
> ** **
>
> I was running a MR job in java and this scenario happened:****
>
> ** **
>
> *Case 1:*
>
> Number of distinct output keys from mapper = 3****
>
> Expected # of reducers = 3****
>
> Defined set # of reducers to be called = 2****
>
> *Expected outcome:*
>
> # of reducers spawned = 2****
>
> # of keys processed under first reducer = 1****
>
> # of keys processed under second reducer = 2****
>
> *Observed outcome:*
>
> # of keys processed under first reducer = 3****
>
> # of keys processed under second reducer = 0****
>
> ** **
>
> ** **
>
> ** **
>
> *Case 2:*
>
> Number of distinct output keys from mapper = 3****
>
> Expected # of reducers = 3****
>
> Defined set # of reducers to be called = 3****
>
> *Expected outcome:*
>
> # of reducers spawned = 3****
>
> # of keys processed under first reducer = 1****
>
> # of keys processed under second reducer = 1****
>
> # of keys processed under third reducer = 1****
>
> * *
>
> *Observed outcome:*
>
> # of reducers spawned = 3****
>
> # of keys processed under first reducer = 2****
>
> # of keys processed under second reducer = 0****
>
> # of keys processed under third reducer = 1****
>
> ** **
>
> ** **
>
> Any idea why all the reducers are not utilized?****
>
> ** **
>
> Saurabh Bajaj *|* Senior Business Analyst *|* +91 9986588089 *|*
> www.mu-sigma.com* **|*
>
> ** **
>
> ------------------------------
> This email message may contain proprietary, private and confidential
> information. The information transmitted is intended only for the person(s)
> or entities to which it is addressed. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited and may be illegal. If you received this in error, please
> contact the sender and delete the message from your system.
>
> Mu Sigma takes all reasonable steps to ensure that its electronic
> communications are free from viruses. However, given Internet
> accessibility, the Company cannot accept liability for any virus introduced
> by this e-mail or any attachment and you are advised to use up-to-date
> virus checking software.
>
--
Harsh J