You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by karthik Srivasthava <ka...@gmail.com> on 2014/08/16 17:15:42 UTC
Yarn Containers using all the space
I am using TEZ as Hive Execution Engine. When i run a query , Yarn
Containers are using all the space available in node. This doesnt allow
another query to run. Am i missing any configuration settings here. Please
let me know guys where the error might be.
Thanks in advance,
Karthik
Re: Yarn Containers using all the space
Posted by Arun Murthy <ac...@hortonworks.com>.
Karthik,
You might find https://github.com/hortonworks/hdp-configuration-utils
useful.
That should help provide all the right configs for YARN, Tez & Hive.
thanks,
Arun
On Sat, Aug 16, 2014 at 4:40 PM, karthik Srivasthava <
karthiksrivasthava@gmail.com> wrote:
> I have total Memory of 48 GB with active 8 nodes.
>
> Initially i set up
> set yarn.nodemanager.resource.memory-mb=6144;
> set yarn.scheduler.maximum-allocation-mb=6144;
> set yarn.scheduler.minimum-allocation-mb=512;
>
>
> 46 Containers are created using all the space available..
>
> Then i changed it to so that i can run other simple queries with remaining
> 8 GB.. But it still takes all the memory.
>
> set yarn.nodemanager.resource.memory-mb=5120;
> set yarn.scheduler.maximum-allocation-mb=5120;
> set yarn.scheduler.minimum-allocation-mb=512;
>
> Even now 46 containers are created using all space..
>
>
> Please accept apologies if it doesnt make any sense.. But please tell me
> where i made mistake
>
>
> On Sat, Aug 16, 2014 at 6:21 PM, Siddharth Tiwari <
> siddharth.tiwari@live.com> wrote:
>
>> Whats the value set in yarn-site ? Can you paste here ?
>>
>> Sent from my iPhone
>>
>> On Aug 16, 2014, at 8:19 PM, "karthik Srivasthava" <
>> karthiksrivasthava@gmail.com> wrote:
>>
>> So i tried using these
>>
>> set yarn.nodemanager.resource.memory-mb ;
>> set yarn.scheduler.maximum-allocation-mb;
>>
>> Even though, one yarn_tez application is using all the space available..
>> Do you think i didnt set these settings properly or
>> i should set different parameters.?
>>
>> Thanks in advance
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Aug 16, 2014 at 3:34 PM, Manoj <ma...@gmail.com> wrote:
>>
>>> Tweak parameters to control container memory allocation. Take a look at
>>> this:
>>> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>>>
>>> On Aug 16, 2014, at 12:58 PM, karthik Srivasthava <
>>> karthiksrivasthava@gmail.com> wrote:
>>>
>>> Thanks manoj for ur response.
>>>
>>> But i struggle to run another application in same node as first yarn
>>> application using all the available RAM .
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com>
>>> wrote:
>>>
>>>> Look into setting up yarn queues and quotas.
>>>>
>>>> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <
>>>> karthiksrivasthava@gmail.com> wrote:
>>>> >
>>>> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn
>>>> Containers are using all the space available in node. This doesnt allow
>>>> another query to run. Am i missing any configuration settings here. Please
>>>> let me know guys where the error might be.
>>>> >
>>>> > Thanks in advance,
>>>> >
>>>> > Karthik
>>>>
>>>
>>>
>>
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Yarn Containers using all the space
Posted by karthik Srivasthava <ka...@gmail.com>.
I have total Memory of 48 GB with active 8 nodes.
Initially i set up
set yarn.nodemanager.resource.memory-mb=6144;
set yarn.scheduler.maximum-allocation-mb=6144;
set yarn.scheduler.minimum-allocation-mb=512;
46 Containers are created using all the space available..
Then i changed it to so that i can run other simple queries with remaining
8 GB.. But it still takes all the memory.
set yarn.nodemanager.resource.memory-mb=5120;
set yarn.scheduler.maximum-allocation-mb=5120;
set yarn.scheduler.minimum-allocation-mb=512;
Even now 46 containers are created using all space..
Please accept apologies if it doesnt make any sense.. But please tell me
where i made mistake
On Sat, Aug 16, 2014 at 6:21 PM, Siddharth Tiwari <siddharth.tiwari@live.com
> wrote:
> Whats the value set in yarn-site ? Can you paste here ?
>
> Sent from my iPhone
>
> On Aug 16, 2014, at 8:19 PM, "karthik Srivasthava" <
> karthiksrivasthava@gmail.com> wrote:
>
> So i tried using these
>
> set yarn.nodemanager.resource.memory-mb ;
> set yarn.scheduler.maximum-allocation-mb;
>
> Even though, one yarn_tez application is using all the space available..
> Do you think i didnt set these settings properly or
> i should set different parameters.?
>
> Thanks in advance
>
>
>
>
>
>
>
>
>
> On Sat, Aug 16, 2014 at 3:34 PM, Manoj <ma...@gmail.com> wrote:
>
>> Tweak parameters to control container memory allocation. Take a look at
>> this:
>> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>>
>> On Aug 16, 2014, at 12:58 PM, karthik Srivasthava <
>> karthiksrivasthava@gmail.com> wrote:
>>
>> Thanks manoj for ur response.
>>
>> But i struggle to run another application in same node as first yarn
>> application using all the available RAM .
>>
>>
>>
>>
>>
>>
>> On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com> wrote:
>>
>>> Look into setting up yarn queues and quotas.
>>>
>>> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <
>>> karthiksrivasthava@gmail.com> wrote:
>>> >
>>> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn
>>> Containers are using all the space available in node. This doesnt allow
>>> another query to run. Am i missing any configuration settings here. Please
>>> let me know guys where the error might be.
>>> >
>>> > Thanks in advance,
>>> >
>>> > Karthik
>>>
>>
>>
>
Re: Yarn Containers using all the space
Posted by Siddharth Tiwari <si...@live.com>.
Whats the value set in yarn-site ? Can you paste here ?
Sent from my iPhone
> On Aug 16, 2014, at 8:19 PM, "karthik Srivasthava" <ka...@gmail.com> wrote:
>
> So i tried using these
>
> set yarn.nodemanager.resource.memory-mb ;
> set yarn.scheduler.maximum-allocation-mb;
>
> Even though, one yarn_tez application is using all the space available.. Do you think i didnt set these settings properly or
> i should set different parameters.?
>
> Thanks in advance
>
>
>
>
>
>
>
>
>
>> On Sat, Aug 16, 2014 at 3:34 PM, Manoj <ma...@gmail.com> wrote:
>> Tweak parameters to control container memory allocation. Take a look at this: http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>>
>>> On Aug 16, 2014, at 12:58 PM, karthik Srivasthava <ka...@gmail.com> wrote:
>>>
>>> Thanks manoj for ur response.
>>>
>>> But i struggle to run another application in same node as first yarn application using all the available RAM .
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com> wrote:
>>>> Look into setting up yarn queues and quotas.
>>>>
>>>> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <ka...@gmail.com> wrote:
>>>> >
>>>> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn Containers are using all the space available in node. This doesnt allow another query to run. Am i missing any configuration settings here. Please let me know guys where the error might be.
>>>> >
>>>> > Thanks in advance,
>>>> >
>>>> > Karthik
>
Re: Yarn Containers using all the space
Posted by karthik Srivasthava <ka...@gmail.com>.
So i tried using these
set yarn.nodemanager.resource.memory-mb ;
set yarn.scheduler.maximum-allocation-mb;
Even though, one yarn_tez application is using all the space available..
Do you think i didnt set these settings properly or
i should set different parameters.?
Thanks in advance
On Sat, Aug 16, 2014 at 3:34 PM, Manoj <ma...@gmail.com> wrote:
> Tweak parameters to control container memory allocation. Take a look at
> this:
> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>
> On Aug 16, 2014, at 12:58 PM, karthik Srivasthava <
> karthiksrivasthava@gmail.com> wrote:
>
> Thanks manoj for ur response.
>
> But i struggle to run another application in same node as first yarn
> application using all the available RAM .
>
>
>
>
>
>
> On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com> wrote:
>
>> Look into setting up yarn queues and quotas.
>>
>> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <
>> karthiksrivasthava@gmail.com> wrote:
>> >
>> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn
>> Containers are using all the space available in node. This doesnt allow
>> another query to run. Am i missing any configuration settings here. Please
>> let me know guys where the error might be.
>> >
>> > Thanks in advance,
>> >
>> > Karthik
>>
>
>
Re: Yarn Containers using all the space
Posted by Manoj <ma...@gmail.com>.
Tweak parameters to control container memory allocation. Take a look at this: http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
> On Aug 16, 2014, at 12:58 PM, karthik Srivasthava <ka...@gmail.com> wrote:
>
> Thanks manoj for ur response.
>
> But i struggle to run another application in same node as first yarn application using all the available RAM .
>
>
>
>
>
>
>> On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com> wrote:
>> Look into setting up yarn queues and quotas.
>>
>> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <ka...@gmail.com> wrote:
>> >
>> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn Containers are using all the space available in node. This doesnt allow another query to run. Am i missing any configuration settings here. Please let me know guys where the error might be.
>> >
>> > Thanks in advance,
>> >
>> > Karthik
>
Re: Yarn Containers using all the space
Posted by karthik Srivasthava <ka...@gmail.com>.
Thanks manoj for ur response.
But i struggle to run another application in same node as first yarn
application using all the available RAM .
On Sat, Aug 16, 2014 at 11:44 AM, Manoj <ma...@gmail.com> wrote:
> Look into setting up yarn queues and quotas.
>
> > On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <
> karthiksrivasthava@gmail.com> wrote:
> >
> > I am using TEZ as Hive Execution Engine. When i run a query , Yarn
> Containers are using all the space available in node. This doesnt allow
> another query to run. Am i missing any configuration settings here. Please
> let me know guys where the error might be.
> >
> > Thanks in advance,
> >
> > Karthik
>
Re: Yarn Containers using all the space
Posted by Manoj <ma...@gmail.com>.
Look into setting up yarn queues and quotas.
> On Aug 16, 2014, at 8:15 AM, karthik Srivasthava <ka...@gmail.com> wrote:
>
> I am using TEZ as Hive Execution Engine. When i run a query , Yarn Containers are using all the space available in node. This doesnt allow another query to run. Am i missing any configuration settings here. Please let me know guys where the error might be.
>
> Thanks in advance,
>
> Karthik
Re: Yarn Containers using all the space
Posted by Siddharth Tiwari <si...@live.com>.
Limit the maximum usable memory by Yarn
Sent from my iPhone
> On Aug 16, 2014, at 12:16 PM, "karthik Srivasthava" <ka...@gmail.com> wrote:
>
> I am using TEZ as Hive Execution Engine. When i run a query , Yarn Containers are using all the space available in node. This doesnt allow another query to run. Am i missing any configuration settings here. Please let me know guys where the error might be.
>
> Thanks in advance,
>
> Karthik