You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Balaji Mani (JIRA)" <ji...@apache.org> on 2016/01/29 07:20:39 UTC
[jira] [Comment Edited] (CLOUDSTACK-9250) Storage automation
results in java memory leaks.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123046#comment-15123046 ]
Balaji Mani edited comment on CLOUDSTACK-9250 at 1/29/16 6:20 AM:
------------------------------------------------------------------
We have been getting Java Heap Space Out of Memory Error in CloudStack Management Server frequently with CS release 4.5.2 and 4.6.0
We have providing load to CloudStack Management Server with below screnarios.
01. VM creation, VM Deletion, Volume Creation, Volume Deletion via Selenuim automated script. The cloudstack management server runs for a day and later mangment service gets down due to Java Heap Space Out of Memory Error.
02. External API calls to Cloudstack Management Server for VM creation, VM Deletion, Volume Creation, Volume Deletion. The cloudstack management server runs for few hours and later mangment service gets down due to Java Heap Space Out of Memory Error.
03. API Calls via jar file for VM creation, VM Deletion, VM Rename, Volume Creation, Volume Deletion. The cloudstack management server runs for few hours and later mangment service gets down due to Java Heap Space Out of Memory Error.
We have enclosed hprof dump file that was generated during Java Heap Space Out of Memory Error.
Also please see below log from cloudstack management server
2016-01-28 14:35:55,537 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5333ae5d) Management server heartbeat takes too long to finish. profiler: Done. Duration: 27789ms, profilerHeartbeatUpdate: Done. Duration: 11214ms, profilerPeerScan: Done. Duration: 16575ms
2016-01-28 14:41:37,336 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4148931f) AutoScaling Monitor is running...
2016-01-28 14:45:03,393 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-651ae6ea) HostStatsCollector is running...
2016-01-28 14:48:13,427 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-177a3b51) Checking if any host reservation can be released ...
2016-01-28 14:48:13,427 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Running Capacity Checker ...
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) recalculating system capacity
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Executing cpu/ram capacity update
2016-01-28 14:48:16,003 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-003e359d) checking health of usage server
2016-01-28 14:51:38,249 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-bcb1b8b3) StorageCollector is running...
2016-01-28 14:55:07,978 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Caught exception in recalculating capacity
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:14,423 WARN [c.c.s.VolumeApiServiceImpl] (catalina-exec-17:ctx-00709386 ctx-ea1c8ed4 ctx-5bc68d39) Failed to expunge volume:
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:851)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1327)
at com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:485)
at com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:55)
at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy62.updateState(Unknown Source)
at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:126)
at com.cloud.storage.VolumeApiServiceImpl.stateTransitTo(VolumeApiServiceImpl.java:1163)
at com.cloud.storage.VolumeApiServiceImpl.deleteVolume(VolumeApiServiceImpl.java:1130)
at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy185.deleteVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.DeleteVolumeCmd.execute(DeleteVolumeCmd.java:85)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:704)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:529)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:288)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:119)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:116)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:435)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:560)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:471)
at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:464)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:830)
... 60 more
2016-01-28 14:55:14,422 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-0a8d7999) Found the following agents behind on ping: [1]
2016-01-28 14:55:12,221 WARN [c.c.a.m.DirectAgentAttache] (DirectAgent-251:ctx-e7d4836a) Seq 1-1957939937999348093: Throwable caught while executing command
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:10,748 ERROR [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-79a685c7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
was (Author: balajimani):
We have been getting Java Heap Space Out of Memory Error in CloudStack Management Server frequently with CS release 4.5.2
We have providing load to CloudStack Management Server with below screnarios.
01. VM creation, VM Deletion, Volume Creation, Volume Deletion via Selenuim automated script. The cloudstack management server runs for a day and later mangment service gets down due to Java Heap Space Out of Memory Error.
02. External API calls to Cloudstack Management Server for VM creation, VM Deletion, Volume Creation, Volume Deletion. The cloudstack management server runs for few hours and later mangment service gets down due to Java Heap Space Out of Memory Error.
03. API Calls via jar file for VM creation, VM Deletion, VM Rename, Volume Creation, Volume Deletion. The cloudstack management server runs for few hours and later mangment service gets down due to Java Heap Space Out of Memory Error.
We have enclosed hprof dump file that was generated during Java Heap Space Out of Memory Error.
Also please see below log from cloudstack management server
2016-01-28 14:35:55,537 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5333ae5d) Management server heartbeat takes too long to finish. profiler: Done. Duration: 27789ms, profilerHeartbeatUpdate: Done. Duration: 11214ms, profilerPeerScan: Done. Duration: 16575ms
2016-01-28 14:41:37,336 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4148931f) AutoScaling Monitor is running...
2016-01-28 14:45:03,393 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-651ae6ea) HostStatsCollector is running...
2016-01-28 14:48:13,427 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-177a3b51) Checking if any host reservation can be released ...
2016-01-28 14:48:13,427 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Running Capacity Checker ...
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) recalculating system capacity
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Executing cpu/ram capacity update
2016-01-28 14:48:16,003 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-003e359d) checking health of usage server
2016-01-28 14:51:38,249 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-bcb1b8b3) StorageCollector is running...
2016-01-28 14:55:07,978 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-027f198c) Caught exception in recalculating capacity
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:14,423 WARN [c.c.s.VolumeApiServiceImpl] (catalina-exec-17:ctx-00709386 ctx-ea1c8ed4 ctx-5bc68d39) Failed to expunge volume:
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:851)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1327)
at com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:485)
at com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:55)
at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy62.updateState(Unknown Source)
at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:126)
at com.cloud.storage.VolumeApiServiceImpl.stateTransitTo(VolumeApiServiceImpl.java:1163)
at com.cloud.storage.VolumeApiServiceImpl.deleteVolume(VolumeApiServiceImpl.java:1130)
at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy185.deleteVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.DeleteVolumeCmd.execute(DeleteVolumeCmd.java:85)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:704)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:529)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:288)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:119)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:116)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:435)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:560)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:471)
at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:464)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:830)
... 60 more
2016-01-28 14:55:14,422 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-0a8d7999) Found the following agents behind on ping: [1]
2016-01-28 14:55:12,221 WARN [c.c.a.m.DirectAgentAttache] (DirectAgent-251:ctx-e7d4836a) Seq 1-1957939937999348093: Throwable caught while executing command
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:10,748 ERROR [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-79a685c7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Storage automation results in java memory leaks.
> ------------------------------------------------
>
> Key: CLOUDSTACK-9250
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9250
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Automation
> Affects Versions: 4.5.2, 4.6.0
> Reporter: Balaji Mani
> Priority: Critical
>
> While automating the storage creation and deletion in selenium, after 1000 storage creation and deletion, the CloudStack management server getting freezes due to JAVA Heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)