You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/09/22 12:16:43 UTC
[GitHub] [cloudstack] andrijapanicsb opened a new issue #4337: Memory leak in master
andrijapanicsb opened a new issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337
In Master, we have memory leak in the management server
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-701318496
@nvazquez can you advise if it's related to PR #4307, or an env issue or if you can propose a fix? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher edited a comment on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
GabrielBrascher edited a comment on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-701734127
Good catch, @andrijapanicsb.
Just for the record, I haven't seen a memory leak on the master. However, I tested _master_ only with KVM nodes + mgmt on Ubuntu 18.04.
Do you know if this happens on master + XenServer as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-697287657
@andrijapanicsb I'm trying to understand how you determine the memory size/leak exists and other env details (like hypervisor etc.) The details are missing in this issue.
`htop` shows three memory columns, I think you may be getting confused by the threads here. I'll however test latest master against visual VM and deploy some VMs with different networks on KVM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-696686880
@andrijapanicsb what are the steps of reproduction? What do we need to do or what did you did (what operations) when you observed this? To check memory leak did you check the `free -m` or are you getting confused with the top/htop output for virtual/resident/shared memory?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-699023326
[root@ref-trl-1479-v-M7-andrija-panic-mgmt1 ~]# systemctl status cloudstack-management
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-09-25 16:18:43 UTC; 1min 4s ago
Process: 47504 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
Main PID: 47504 (code=exited, status=1/FAILURE)
Sep 25 16:18:42 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: INFO [c.c.s.StorageManagerImpl] (API-Job-Executor-17:ctx-a5ad92c7 job-2153 ctx-7252aa30) (logid:a0dad257) Storage pool null (1) ...ugh capacity
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000eb500000, 344981504, 0) failed; error='Not enough space' (errno=12)
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: #
**Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # There is insufficient memory for the Java Runtime Environment to continue.**
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # Native memory allocation (mmap) failed to map 344981504 bytes for committing reserved memory.
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # An error report file with more information is saved as:
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # /var/log/cloudstack/management/cloudstack-management.err
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: cloudstack-management.service: main process exited, code=exited, status=1/FAILURE
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: Unit cloudstack-management.service entered failed state.
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: cloudstack-management.service failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-697989726
I see this in master with VMware 6.7
In my test env (like all Trillian test env), mgmt server has 4GB (no swap), and at some point I could not use yum due to not free memory - I've also seen in this env OOM killer kicking in and killing the mgmt server - this has not happene ever (to me) in any Trillian-deployed env - so I've used "ps aux | grep ......" to log the PS output to a file - see below a very short video, with RSS ever inreasing, even though no serios work was done in the env during 15h/day (while I'm not using it, as it's test env used only by me)
RSS increase video: https://screencast-o-matic.com/watch/cYQ0rCdYj4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-696686880
@andrijapanicsb what are the steps of reproduction? What do we need to do or what did you did (what operations) when you observed this? To check memory leak did you check the `free -m` or are you getting confused with the top/htop output for virtual/resident/shared memory?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-701919878
Haven't seen it elsewhere, only on VMware - but this needs reproduction by someone else - I will be reprorting back as I'm actively testing a few VMware stuff.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-697266723
install the env and let it do something (perhaps sping a few VRs) - I did use this env actively though.
"ps aux" and read the 6th column, or HTOP from time to time and observer the RAM consumed by the process..
...
or wait for few days, then try a simple yum install which will fail (assuming no swap mounted) with no memory error (this is what happened initially to me, so I've investigated)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd closed issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-701734127
Good catch, @andrijapanicsb.
Just for the record, I haven't seen a memory leak on the master. However, I have always been testing _master_ with KVM nodes + mgmt on Ubuntu 18.04.
Do you know if this happens on master + XenServer as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4337: Memory leak in master
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4337:
URL: https://github.com/apache/cloudstack/issues/4337#issuecomment-711772406
We couldn't reproduce after 94+hrs:
The issue was seen from pkgs of a specific PR and not master. With https://github.com/apache/cloudstack/pull/4407 fix ^^, based on this I'll close this issue. Let's reopen and re-investigate if it happens again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org