You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Noelia Osés Fernández <no...@vicomtech.org> on 2017/12/13 12:08:50 UTC

How can I limit RAM usage?

Hi all,

I have a cloud server with 4GB RAM in which I have installed PIO
0.12.0-incubating and a few other things. However, I'm having trouble
running even the smallest examples as it runs out of RAM (plus the swap is
also full).

I have set a limit for ES in jvm.options as follows:
-Xms512m
-Xmx512m

Before training and deploying I limit java's RAM usage as follows:

export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8"
pio train  -- --driver-memory 1G --executor-memory 1G
nohup pio deploy > deploy.out &

Are there any more measures I can take to limit RAM usage?

I would like to know I've done absolutely everything possible before paying
to get more RAM on the server.

Thank you for your help!
Noelia

Re: How can I limit RAM usage?

Posted by Pat Ferrel <pa...@occamsmachete.com>.
Spark uses a minimum of 2g per executor with no data or work. There is always one executor and one driver with Spark. Welcome to Big Data.

A 4g server is not sufficient to run Spark let alone the rest of the PIO stack. 16g minimum is my recommendation—and I do mean minimum. Machine learning is not an application you can squeeze into small RAM machines.
 

On Dec 13, 2017, at 4:14 AM, Noelia Osés Fernández <no...@vicomtech.org> wrote:

I should mention that when I reboot the server and before I start PIO 3GB of RAM are already used, so there is only 1GB free RAM for PIO. 
 

On 13 December 2017 at 13:08, Noelia Osés Fernández <noses@vicomtech.org <ma...@vicomtech.org>> wrote:

Hi all,

I have a cloud server with 4GB RAM in which I have installed PIO 0.12.0-incubating and a few other things. However, I'm having trouble running even the smallest examples as it runs out of RAM (plus the swap is also full).

I have set a limit for ES in jvm.options as follows:
-Xms512m
-Xmx512m

Before training and deploying I limit java's RAM usage as follows:

export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8"
pio train  -- --driver-memory 1G --executor-memory 1G
nohup pio deploy > deploy.out &

Are there any more measures I can take to limit RAM usage?

I would like to know I've done absolutely everything possible before paying to get more RAM on the server.

Thank you for your help!
Noelia


 



-- 
 <http://www.vicomtech.org/>

Noelia Osés Fernández, PhD
Senior Researcher |
Investigadora Senior

noses@vicomtech.org <ma...@vicomtech.org>
+[34] 943 30 92 30
Data Intelligence for Energy and
Industrial Processes | Inteligencia
de Datos para Energía y Procesos
Industriales

 <https://www.linkedin.com/company/vicomtech>  <https://www.youtube.com/user/VICOMTech>  <ht...@Vicomtech_IK4>

member of:  <http://www.graphicsmedia.net/>     <http://www.ik4.es/>

Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>

Re: How can I limit RAM usage?

Posted by Noelia Osés Fernández <no...@vicomtech.org>.
I should mention that when I reboot the server and before I start PIO 3GB
of RAM are already used, so there is only 1GB free RAM for PIO.


On 13 December 2017 at 13:08, Noelia Osés Fernández <no...@vicomtech.org>
wrote:

>
> Hi all,
>
> I have a cloud server with 4GB RAM in which I have installed PIO
> 0.12.0-incubating and a few other things. However, I'm having trouble
> running even the smallest examples as it runs out of RAM (plus the swap is
> also full).
>
> I have set a limit for ES in jvm.options as follows:
> -Xms512m
> -Xmx512m
>
> Before training and deploying I limit java's RAM usage as follows:
>
> export JAVA_OPTS="-Xmx1g -Xms1g -Dfile.encoding=UTF-8"
> pio train  -- --driver-memory 1G --executor-memory 1G
> nohup pio deploy > deploy.out &
>
> Are there any more measures I can take to limit RAM usage?
>
> I would like to know I've done absolutely everything possible before
> paying to get more RAM on the server.
>
> Thank you for your help!
> Noelia
>
>
>
>



-- 
<http://www.vicomtech.org>

Noelia Osés Fernández, PhD
Senior Researcher |
Investigadora Senior

noses@vicomtech.org
+[34] 943 30 92 30
Data Intelligence for Energy and
Industrial Processes | Inteligencia
de Datos para Energía y Procesos
Industriales

<https://www.linkedin.com/company/vicomtech>
<https://www.youtube.com/user/VICOMTech>
<ht...@Vicomtech_IK4>

member of:  <http://www.graphicsmedia.net/>     <http://www.ik4.es>

Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>