You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Garg, Rinku" <Ri...@fisglobal.com> on 2013/10/09 14:20:55 UTC
Can we reduce/fix number of mappers in Hive
Hi All
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have the following query:
Is there any option in Hive where mappers can be reduced/fixed?
For example if there are 8 mapping tasks defined in Hadoop configuration, then is it possible to use 4 hive setups that uses 2 mappers each?
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Can we reduce/fix number of mappers in Hive
Posted by Nitin Pawar <ni...@gmail.com>.
What is your understanding of "4 hive setups" ?
How many maps or reducers are needed for your query is determined by the
size of the data you got. You can alter the number of mappers needed for a
query by setting the max and min splits. But I am guessing that is not what
you want to achieve,
I guess what you are trying to achieve is resource sharing on jobtracker.
In other words you want to dedicate X number of resources to a particular
kind of work (In your case all of them are hive jobs)
If you want to do that way, then please look at hadoop job queues and
different schedulers like capacity scheduler.
Once you create different job queues, you can use the property
mapred.job.queue.name in your hive-site.conf to point to a particular queue
created on jobtracker.
On Thu, Oct 10, 2013 at 12:19 PM, Garg, Rinku <Ri...@fisglobal.com>wrote:
> Hi Nitin/Rajesh,
>
> Thanks for the reply...
>
> I tried with both the ways as you guys mentioned in the mail trail...
> but using that way hive does not fix the number of mappers for a hive query.
>
> My orginal question is :
>
> if there are 8 mapping tasks defined in Hadoop configuration, then is it
> possible to use 4 hive setups that uses 2 mappers each?
>
> Please reply.
>
> Thanks
> Rinku Garg
> ------------------------------
> *From:* Nitin Pawar [nitinpawar432@gmail.com]
> *Sent:* Wednesday, October 09, 2013 6:15 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Can we reduce/fix number of mappers in Hive
>
> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very
> old versions .
> You may want to move to hadoop 1.2.0 and hive-0.11 to have few more
> things.
>
> If you want to limit how many maps a particular job or particular user
> can occupy, take a look at hadoop's fair scheduler can capacity schedulers.
> If you want change the number of mapper and reducers for a hive query, you
> can change the input min and max split sizes as said by Rajesh.
>
> Here are few of the settings
> set mapred.min.split.size=1024000;
> set mapred.max.split.size=4096000;
> set hive.exec.reducers.max=2048;
>
> may be once you tell clearly what you want achieve we will be able to
> help better.
>
>
>
> On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan <
> rajesh.balamohan@gmail.com> wrote:
>
>> Did you try adjusting fileinputformat. Min and max size parameters?
>> On Oct 9, 2013 5:51 PM, "Garg, Rinku" <Ri...@fisglobal.com> wrote:
>>
>>> Hi All****
>>>
>>> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have
>>> the following query:****
>>>
>>> ** **
>>>
>>> Is there any option in Hive where mappers can be reduced/fixed?****
>>>
>>> ** **
>>>
>>> For example if there are 8 mapping tasks defined in Hadoop
>>> configuration, then is it possible to use 4 hive setups that uses 2 mappers
>>> each?****
>>>
>>> ** **
>>>
>>> Thanks & Regards,****
>>>
>>> Rinku Garg
>>>
>>>
>>> ****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> ** **
>>> _____________
>>> The information contained in this message is proprietary and/or
>>> confidential. If you are not the intended recipient, please: (i) delete the
>>> message and all copies; (ii) do not disclose, distribute or use the message
>>> in any manner; and (iii) notify the sender immediately. In addition, please
>>> be aware that any message addressed to our domain is subject to archiving
>>> and review by persons other than the intended recipient. Thank you.
>>>
>>
>
>
> --
> Nitin Pawar
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
--
Nitin Pawar
RE: Can we reduce/fix number of mappers in Hive
Posted by "Garg, Rinku" <Ri...@fisglobal.com>.
Hi Nitin/Rajesh,
Thanks for the reply...
I tried with both the ways as you guys mentioned in the mail trail... but using that way hive does not fix the number of mappers for a hive query.
My orginal question is :
if there are 8 mapping tasks defined in Hadoop configuration, then is it possible to use 4 hive setups that uses 2 mappers each?
Please reply.
Thanks
Rinku Garg
________________________________
From: Nitin Pawar [nitinpawar432@gmail.com]
Sent: Wednesday, October 09, 2013 6:15 PM
To: user@hive.apache.org
Subject: Re: Can we reduce/fix number of mappers in Hive
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very old versions .
You may want to move to hadoop 1.2.0 and hive-0.11 to have few more things.
If you want to limit how many maps a particular job or particular user can occupy, take a look at hadoop's fair scheduler can capacity schedulers.
If you want change the number of mapper and reducers for a hive query, you can change the input min and max split sizes as said by Rajesh.
Here are few of the settings
set mapred.min.split.size=1024000;
set mapred.max.split.size=4096000;
set hive.exec.reducers.max=2048;
may be once you tell clearly what you want achieve we will be able to help better.
On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan <ra...@gmail.com>> wrote:
Did you try adjusting fileinputformat. Min and max size parameters?
On Oct 9, 2013 5:51 PM, "Garg, Rinku" <Ri...@fisglobal.com>> wrote:
Hi All
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have the following query:
Is there any option in Hive where mappers can be reduced/fixed?
For example if there are 8 mapping tasks defined in Hadoop configuration, then is it possible to use 4 hive setups that uses 2 mappers each?
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Can we reduce/fix number of mappers in Hive
Posted by Nitin Pawar <ni...@gmail.com>.
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very old
versions .
You may want to move to hadoop 1.2.0 and hive-0.11 to have few more things.
If you want to limit how many maps a particular job or particular user can
occupy, take a look at hadoop's fair scheduler can capacity schedulers.
If you want change the number of mapper and reducers for a hive query, you
can change the input min and max split sizes as said by Rajesh.
Here are few of the settings
set mapred.min.split.size=1024000;
set mapred.max.split.size=4096000;
set hive.exec.reducers.max=2048;
may be once you tell clearly what you want achieve we will be able to help
better.
On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan <rajesh.balamohan@gmail.com
> wrote:
> Did you try adjusting fileinputformat. Min and max size parameters?
> On Oct 9, 2013 5:51 PM, "Garg, Rinku" <Ri...@fisglobal.com> wrote:
>
>> Hi All****
>>
>> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have
>> the following query:****
>>
>> ** **
>>
>> Is there any option in Hive where mappers can be reduced/fixed?****
>>
>> ** **
>>
>> For example if there are 8 mapping tasks defined in Hadoop configuration,
>> then is it possible to use 4 hive setups that uses 2 mappers each?****
>>
>> ** **
>>
>> Thanks & Regards,****
>>
>> Rinku Garg
>>
>>
>> ****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>> _____________
>> The information contained in this message is proprietary and/or
>> confidential. If you are not the intended recipient, please: (i) delete the
>> message and all copies; (ii) do not disclose, distribute or use the message
>> in any manner; and (iii) notify the sender immediately. In addition, please
>> be aware that any message addressed to our domain is subject to archiving
>> and review by persons other than the intended recipient. Thank you.
>>
>
--
Nitin Pawar
Re: Can we reduce/fix number of mappers in Hive
Posted by Rajesh Balamohan <ra...@gmail.com>.
Did you try adjusting fileinputformat. Min and max size parameters?
On Oct 9, 2013 5:51 PM, "Garg, Rinku" <Ri...@fisglobal.com> wrote:
> Hi All****
>
> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have
> the following query:****
>
> ** **
>
> Is there any option in Hive where mappers can be reduced/fixed?****
>
> ** **
>
> For example if there are 8 mapping tasks defined in Hadoop configuration,
> then is it possible to use 4 hive setups that uses 2 mappers each?****
>
> ** **
>
> Thanks & Regards,****
>
> Rinku Garg
>
>
> ****
>
> ** **
>
> ** **
>
> ** **
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>