You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by rv...@privaz.io.INVALID on 2021/01/20 07:43:59 UTC
4.15 Manager out of memory
Hi!
Congrats for the 4.15 release! UI makes a great difference.
I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
Anybody experiencing this?
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/cloudstack-management.service.d
└─mesh-dep.conf, mysql-dep.conf
Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
Main PID: 6641 (code=exited, status=1/FAILURE)
Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
Jan 19 11:02:44 manager java[6641]: #
Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
Re: 4.15 Manager out of memory
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
Hi Nicolas,
Thanks for asking.
I use KVM.
3G seems like a tremendous amount on memory to manage 3 hosts. I am resisting to increase for the time being in my test environment. In production there are 4G minimum, so there it is fine.
My test environment meant to catch issues, I run on low resources on purpose. That also allows me to run more tests in parallel.
If the problem reappears I would like to compare the memory behaviour between 4.14 and 4.15. Run the manager on exact same JVM settings and perform some black box stress test while recording the JVM behaviour.
Also, it is still early for me to start pointing fingers to any piece of software, I have run 4.15 for only a few days and fixed many issues.
Regards,
Rafael
On Thu, 2021-01-21 04:26 AM, Nicolas Vazquez <Ni...@shapeblue.com> wrote:
> Hi Rafael,
>
> Was this solved by increasing the RAM on the management server? Which hypervisor are you using in this test environment?
>
>
> Regards,
>
> Nicolas Vazquez
>
> ________________________________
> From: Rohit Yadav " target="_blank"><ro...@shapeblue.com>
> Sent: Wednesday, January 20, 2021 6:10 AM
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>; dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>; Rafael del Valle " target="_blank"><rv...@privaz.io>
> Subject: Re: 4.15 Manager out of memory
>
> This "Ram is 1.5GB + swap", please increase RAM to 3G and try again.
>
>
> Regards.
>
> ________________________________
> From: Rafael del Valle " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 14:39
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>; dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: Re: 4.15 Manager out of memory
>
> Hi Rohit
>
> This is not a production environment. This is a test environment run
> on minimum resources for my CD/CI cloud release process, so you have to
> take this with a pitch of salt:
>
>
> My environment is Ubuntu 18.04
> Ram is 1.5GB + swap
> DB is running on the same host.
>
> However, just in case, note that:
>
> With 4.14, 150 cloud environments were built and tested and this problem did not arise.
> with 4.15 the problem popped on the first environment.
>
> I can see the changes made from 4.14 to the JVM options and those are:
>
> -XX:+UseParallelGC
> -XX:MaxGCPauseMillis=500
>
> That could explain the difference in behaviour, but better tu remain suspicious.
> I am going to raise memory and continue testing and see if the problem persists.
>
> Is memory profiling done as part of ACS release/testing?
>
> Regards,
> Rafael
>
>
> On Wed, 2021-01-20 09:06 AM, Rohit Yadav " target="_blank"><ro...@shapeblue.com> wrote:
> > Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
> >
> >
> > [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
> >
> >
> > Regards.
> >
> > ________________________________
> > From: rvalle@privaz.io.INVALID " target="_blank">" target="_blank"><rv...@privaz.io.INVALID>
> > Sent: Wednesday, January 20, 2021 13:13
> > To: users@cloudstack.apache.org " target="_blank">" target="_blank"><us...@cloudstack.apache.org>
> > Cc: dev@cloudstack.apache.org " target="_blank">" target="_blank"><de...@cloudstack.apache.org>
> > Subject: 4.15 Manager out of memory
> >
> > Hi!
> >
> > Congrats for the 4.15 release! UI makes a great difference.
> >
> > I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
> >
> > This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
> >
> > Anybody experiencing this?
> >
> > ● cloudstack-management.service - CloudStack Management Server
> > Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> > Drop-In: /etc/systemd/system/cloudstack-management.service.d
> > └─mesh-dep.conf, mysql-dep.conf
> > Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> > Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> > Main PID: 6641 (code=exited, status=1/FAILURE)
> >
> > Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> > Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> > Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> > Jan 19 11:02:44 manager java[6641]: #
> > Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> > Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> > Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> > Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> > Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> > Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
> >
> >
> > rohit.yadav@shapeblue.com
> > http://www.shapeblue.com
> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > @shapeblue
> >
> >
> >
> >
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com<http://www.shapeblue.com>
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
> Nicolas.Vazquez@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
Re: 4.15 Manager out of memory
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi Rafael,
Was this solved by increasing the RAM on the management server? Which hypervisor are you using in this test environment?
Regards,
Nicolas Vazquez
________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: Wednesday, January 20, 2021 6:10 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Rafael del Valle <rv...@privaz.io>
Subject: Re: 4.15 Manager out of memory
This "Ram is 1.5GB + swap", please increase RAM to 3G and try again.
Regards.
________________________________
From: Rafael del Valle <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 14:39
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: 4.15 Manager out of memory
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Re: 4.15 Manager out of memory
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi Rafael,
Was this solved by increasing the RAM on the management server? Which hypervisor are you using in this test environment?
Regards,
Nicolas Vazquez
________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: Wednesday, January 20, 2021 6:10 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Rafael del Valle <rv...@privaz.io>
Subject: Re: 4.15 Manager out of memory
This "Ram is 1.5GB + swap", please increase RAM to 3G and try again.
Regards.
________________________________
From: Rafael del Valle <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 14:39
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: 4.15 Manager out of memory
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Re: 4.15 Manager out of memory
Posted by Rohit Yadav <ro...@shapeblue.com>.
This "Ram is 1.5GB + swap", please increase RAM to 3G and try again.
Regards.
________________________________
From: Rafael del Valle <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 14:39
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: 4.15 Manager out of memory
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Re: 4.15 Manager out of memory
Posted by Rohit Yadav <ro...@shapeblue.com>.
This "Ram is 1.5GB + swap", please increase RAM to 3G and try again.
Regards.
________________________________
From: Rafael del Valle <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 14:39
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: 4.15 Manager out of memory
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Re: 4.15 Manager out of memory
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
Re: 4.15 Manager out of memory
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
>
>
> [1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: rvalle@privaz.io.INVALID " target="_blank"><rv...@privaz.io.INVALID>
> Sent: Wednesday, January 20, 2021 13:13
> To: users@cloudstack.apache.org " target="_blank"><us...@cloudstack.apache.org>
> Cc: dev@cloudstack.apache.org " target="_blank"><de...@cloudstack.apache.org>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
> Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
>
>
> rohit.yadav@shapeblue.com
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
Re: 4.15 Manager out of memory
Posted by Rohit Yadav <ro...@shapeblue.com>.
Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
[1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
Regards.
________________________________
From: rvalle@privaz.io.INVALID <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 13:13
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: 4.15 Manager out of memory
Hi!
Congrats for the 4.15 release! UI makes a great difference.
I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
Anybody experiencing this?
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/cloudstack-management.service.d
└─mesh-dep.conf, mysql-dep.conf
Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
Main PID: 6641 (code=exited, status=1/FAILURE)
Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
Jan 19 11:02:44 manager java[6641]: #
Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
rohit.yadav@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
Re: 4.15 Manager out of memory
Posted by Rohit Yadav <ro...@shapeblue.com>.
Can you share your host memory, distro (CentOS or Ubuntu and which version) and describe your installation (hypervisor, storage, any special plugin etc you may be using). Are you running DB and usage server on the same host? The java process requires at least 2G RAM[1] for itself.
[1] https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
Regards.
________________________________
From: rvalle@privaz.io.INVALID <rv...@privaz.io.INVALID>
Sent: Wednesday, January 20, 2021 13:13
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: 4.15 Manager out of memory
Hi!
Congrats for the 4.15 release! UI makes a great difference.
I started launching deployment tests with 4.15 and I just noticed that the manager is running out of memory were 4.14 did not.
This is in a test environment where the manager host resources are constrained yet 100% functional in 4.14.
Anybody experiencing this?
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/cloudstack-management.service.d
└─mesh-dep.conf, mysql-dep.conf
Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h ago
Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
Main PID: 6641 (code=exited, status=1/FAILURE)
Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000009ea80000, 26738688, 0) fail
Jan 19 11:02:44 manager java[6641]: #
Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the Java Runtime Environment to continue.
Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed to map 26738688 bytes for committing reserved memory.
Jan 19 11:02:44 manager java[6641]: # An error report file with more information is saved as:
Jan 19 11:02:44 manager java[6641]: # /var/log/cloudstack/management/cloudstack-management.err
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main process exited, code=exited, status=1/FAILURE
Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.
rohit.yadav@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue