Sandbox Question - Set up more than 8 GIG RAM

[Jim <ji...@facility.supplies>]

Good afternoon,

I have a sandbox question.  I have three applications, that when run together currently take around 16 gig of ram.

On my laptop, I have 32 gig of ram, and have allocated 20 GIG to the virtual maching via virtual box.

However, when I look into the Hadoop console, I only see 8 GB ram available, which prevents some of my apps from starting.

What do I change, and where do I change it in the sandbox (version 3.4.0), to allow all the memory available to the virtual machine to be used for the datatorrent applications?

Thanks,

JIm

[cid:image001.png@01D1FECC.F006DC60]

RE: Sandbox Question - Set up more than 8 GIG RAM

[Jim <ji...@facility.supplies>]

Hi, I figured out the answer to this problem outlined below, and for those who may be looking for this:

In the file /etc/hadoop/conf/capacity-scheduler.xml I had to change the property yarn.scheduler.capacity.maximum-am-resource-percent from 0.1 to 0.5.

Changing this setting increases the fraction of the resources that is made available to be allocated to application masters, increasing the number of masters possible to run at once and hence increasing the number of possible concurrent applications.

Thanks,

Jim

From: Jim [mailto:jim@facility.supplies]
Sent: Friday, August 26, 2016 10:01 AM
To: users@apex.apache.org
Subject: RE: Sandbox Question - Set up more than 8 GIG RAM

Hi Brandon,

That worked in terms of getting the memory set up properly.

I actually have three applications, that kind of work together.

Is there anything in the sandbox that prevents me from launching multiple applications?

I can launch any of one of these applications, but themselves, and they all run fine; but when I try to launch a second application, after one is already running, it just sits in ‘ACCEPTED’ state, and never actually launches.

Thanks,

Jim

From: Feldkamp, Brandon (CONT) [mailto:Brandon.Feldkamp@capitalone.com]
Sent: Thursday, August 25, 2016 2:00 PM
To: users@apex.apache.org<ma...@apex.apache.org>
Subject: Re: Sandbox Question - Set up more than 8 GIG RAM

Hey Jim,

I had to make adjustments to the default memory settings which I accomplished by doing the following:

The setting is located in /sfw/hadoop/conf/yarn-site.xml. You should see:

1.      yarn.nodemanager.resource.memory-mb (Amount of physical memory, in MB, that can be allocated for containers.)

2.      yarn.scheduler.maximum-allocation-mb (The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value.)

Reference: https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

You’ll need to at least bounce the resource manager by executing “sudo /sfw/dtbox/dtbox-3.4.0/conf/init.d/hadoop-yarn-resourcemanager restart” to pick up the config change. I’m not sure if you need to restart any other services.

Hope this helps!
Brandon

On 2016-08-25 13:33 (-0400), Jim <j....@facility.supplies>> wrote:
> Good afternoon,>
>
> I have a sandbox question.  I have three applications, that when run together currently take around 16 gig of ram.>
>
> On my laptop, I have 32 gig of ram, and have allocated 20 GIG to the virtual maching via virtual box.>
>
> However, when I look into the Hadoop console, I only see 8 GB ram available, which prevents some of my apps from starting.>
>
> What do I change, and where do I change it in the sandbox (version 3.4.0), to allow all the memory available to the virtual machine to be used for the datatorrent applications?>
>
> Thanks,>
>
> JIm>
>
> [cid:image001.png@01D1FECC.F006DC60<ma...@01D1FECC.F006DC60>]>
>

________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

RE: Sandbox Question - Set up more than 8 GIG RAM

[Jim <ji...@facility.supplies>]

Hi Brandon,

That worked in terms of getting the memory set up properly.

I actually have three applications, that kind of work together.

Is there anything in the sandbox that prevents me from launching multiple applications?

I can launch any of one of these applications, but themselves, and they all run fine; but when I try to launch a second application, after one is already running, it just sits in ‘ACCEPTED’ state, and never actually launches.

Thanks,

Jim

From: Feldkamp, Brandon (CONT) [mailto:Brandon.Feldkamp@capitalone.com]
Sent: Thursday, August 25, 2016 2:00 PM
To: users@apex.apache.org
Subject: Re: Sandbox Question - Set up more than 8 GIG RAM

Hey Jim,

I had to make adjustments to the default memory settings which I accomplished by doing the following:

The setting is located in /sfw/hadoop/conf/yarn-site.xml. You should see:

1.      yarn.nodemanager.resource.memory-mb (Amount of physical memory, in MB, that can be allocated for containers.)

2.      yarn.scheduler.maximum-allocation-mb (The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value.)

Reference: https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

You’ll need to at least bounce the resource manager by executing “sudo /sfw/dtbox/dtbox-3.4.0/conf/init.d/hadoop-yarn-resourcemanager restart” to pick up the config change. I’m not sure if you need to restart any other services.

Hope this helps!
Brandon

On 2016-08-25 13:33 (-0400), Jim <j....@facility.supplies>> wrote:
> Good afternoon,>
>
> I have a sandbox question.  I have three applications, that when run together currently take around 16 gig of ram.>
>
> On my laptop, I have 32 gig of ram, and have allocated 20 GIG to the virtual maching via virtual box.>
>
> However, when I look into the Hadoop console, I only see 8 GB ram available, which prevents some of my apps from starting.>
>
> What do I change, and where do I change it in the sandbox (version 3.4.0), to allow all the memory available to the virtual machine to be used for the datatorrent applications?>
>
> Thanks,>
>
> JIm>
>
> [cid:image001.png@01D1FECC.F006DC60<ma...@01D1FECC.F006DC60>]>
>

________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.