You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/02/06 07:54:44 UTC
[GitHub] [cloudstack] wido opened a new issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
wido opened a new issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
Management Server
MySQL
~~~
##### CLOUDSTACK VERSION
~~~
4.13.1
~~~
##### CONFIGURATION
This is a standard Management Server with a connection to a MariaDB server
##### OS / ENVIRONMENT
- Ubuntu 16.04.7
- 10.2.34-MariaDB-10.2.34+maria~xenial-log
- CloudStack 4.13.1
Settings in db.properties:
<pre>
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
db.cloud.maxIdle=30
db.cloud.maxActive=250
db.cloud.reconnectAtTxEnd=true
db.cloud.autoReconnectForPools=true
db.cloud.autoReconnect=tru
</pre>
##### SUMMARY
The Management Server seems to loose it's connection to MySQL while it is actually still up and running. Causing some components to fail inside the Management Server.
##### STEPS TO REPRODUCE
Not known yet
##### ACTUAL RESULTS
In the logs suddenly it shows:
~~~
2021-02-06 03:59:04,748 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 30, vm id: 1019, power state: PowerOn
2021-02-06 03:59:04,751 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 30, vm id: 187, power state: PowerOn
2021-02-06 03:59:04,754 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 30, vm id: 702, power state: PowerOn
2021-02-06 03:59:04,780 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) Done with process of VM state report. host: 30
2021-02-06 03:59:04,836 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Ping from 34(s-1072-VM)
2021-02-06 03:59:10,284 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-9527a5cd) (logid:2204af40) AutoScaling Monitor is running...
2021-02-06 03:59:13,074 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Ping from Routing host 3(xx.c01.p01.xx.xx)
2021-02-06 03:59:13,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Process host VM state report from ping process. host: 3
2021-02-06 03:59:13,369 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) (logid:) SeqA 35-21752: Processing Seq 35-21752: { Cmd , MgmtId: -1, via: 35, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1071,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
2021-02-06 03:59:14,662 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-dcc6eeba) (logid:098d3bcd) Seq 1-8767382574583524527: Received: { Ans: , MgmtId: 90520732637524, via: 1(xx.c01.p01.xx), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } }
2021-02-06 03:59:14,662 DEBUG [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-dcc6eeba) (logid:098d3bcd) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand:
2021-02-06 03:59:14,741 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-dcc6eeba) (logid:098d3bcd) Seq 1-8767382574583524528: Received: { Ans: , MgmtId: 90520732637524, via: 1(xx.c01.p01.xx), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } }
2021-02-06 03:59:14,741 DEBUG [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-dcc6eeba) (logid:098d3bcd) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand:
2021-02-06 03:59:14,782 ERROR [o.a.c.f.j.d.SyncQueueItemDaoImpl] (AsyncJobMgr-Heartbeat-1:ctx-6c122c53) (logid:f5164750) Unexpected sql exception,
java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:461)
at org.apache.cloudstack.framework.jobs.dao.SyncQueueItemDaoImpl.getNextQueueItems(SyncQueueItemDaoImpl.java:106)
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy195.getNextQueueItems(Unknown Source)
at org.apache.cloudstack.framework.jobs.impl.SyncQueueManagerImpl$3.doInTransactionWithoutResult(SyncQueueManagerImpl.java:140)
at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at org.apache.cloudstack.framework.jobs.impl.SyncQueueManagerImpl.dequeueFromAny(SyncQueueManagerImpl.java:137)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$6.reallyRun(AsyncJobManagerImpl.java:813)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$6.runInContext(AsyncJobManagerImpl.java:801)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:447)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
... 36 more
~~~
This then keeps repeating and the management server doesn't work properly until restarted.
Other log entries:
<pre>
2021-02-06 03:59:15,230 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-6b154ebc) (logid:4a081a66) Management server heartbeat takes too long to finish. profiler: Done. Duration: 10003ms, profilerHeartbeatUpdate: Started but not stopped, profilerPeerScan: Not Started
2021-02-06 03:59:15,230 ERROR [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-6b154ebc) (logid:4a081a66) Unexpected exception in cluster heartbeat
java.lang.RuntimeException: Cannot get a connection, pool error Timeout waiting for idle object
at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:148)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy176.update(Unknown Source)
at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:555)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 27 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:447)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
... 29 more
</pre>
<pre>
2021-02-06 03:59:15,227 WARN [c.c.c.d.ManagementServerHostDaoImpl] (Cluster-Heartbeat-1:ctx-6b154ebc) (logid:4a081a66) Unexpected exception,
java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:461)
at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:134)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy176.update(Unknown Source)
at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:555)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
</pre>
Increasing *db.cloud.maxActive=250* might help, but I would rather not increase this until we are sure that's the root-cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-791906447
Ping @wido @GabrielBrascher any update?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher closed issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher closed issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778128855
> > @wido what's the value of mysql variable "wait_timeout" ?
> > we had some issues caused by it in the past (it was 600 in our environment).
> > ```
> > MariaDB [cloud]> show variables like "wait_timeout";
> > ```
>
> MariaDB [(none)]> show variables like "wait_timeout";
> +---------------+-------+
> | Variable_name | Value |
> +---------------+-------+
> | wait_timeout | 28800 |
> +---------------+-------+
> 1 row in set (0.00 sec)
>
> MariaDB [(none)]>
> The server which this happened to had this issue now happen twice within 24h.
@wido can you check the global variables ?
`show global variables like "wait_timeout";`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] wido commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
wido commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-1006352990
@andrijapanicsb We ended up setting wait_timeout back to the default (3600) as this was set to 60 by some provisioning scripts unintended.
That solved the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-854112884
It looks like changing the MySQL global variables solved this issue.
Closing this issue for now; I may re-open if it happens again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-1006025595
ping @GabrielBrascher did you hit this issue again or was it resolved for good?
Can you specify (if you recall) what variables did you change on the MySQL side?
We are seeing this issue with 4.15.1 and Mysql 8.0.27, and with max_connections = 700 and wait_timeout = 43200.
Any reason to think this might be MySQL 8.x bug?
Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] wido commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
wido commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778429796
> > Thanks for the help @weizhouapache.
> > I think that `max_connections` was set to `28800` and then later to `60` but the errors kept coming either way: [#4657 (comment)](https://github.com/apache/cloudstack/issues/4657#issuecomment-774961189). I will re-check that; hope to find the root cause or a MySQL setup that handles it.
>
> @GabrielBrascher do you mean `wait_timeout` ?
>
> You should change the global value (from 60 to other value e.g. 28800).
>
> ```
> SET GLOBAL wait_timeout=28800;
> ```
We'll look into that and report back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher closed issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher closed issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-792731942
The MySQL variables have been updated recently and we are monitoring to see how it behaves.
Soon we should be able to update here @rhtyd.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-775081550
so that is 8 hours of waiting before mysql abandons stale connections, worth a shot (10 minutes is mayby a bit short)
It would explain things if ACS thinks it has only 20 connections and mysql thinks it has 250.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-774673554
@wido what's the value of mysql variable "wait_timeout" ?
we had some issues caused by it in the past (it was 600 in our environment).
```
MariaDB [cloud]> show variables like "wait_timeout";
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778193711
Thanks for the help @weizhouapache.
I think that `max_connections` was set to `28800` and then later to `60` but the errors kept coming either way: https://github.com/apache/cloudstack/issues/4657#issuecomment-774961189. I will re-check that; hope to find the root cause or a MySQL setup that handles it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778186019
@GabrielBrascher yes,
60 is ok if database is small.
But when db is getting bigger and bigger, you should increase the value.
refer to https://mariadb.com/docs/reference/es/system-variables/wait_timeout/
we use 28800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778232882
> Thanks for the help @weizhouapache.
>
> I think that `max_connections` was set to `28800` and then later to `60` but the errors kept coming either way: [#4657 (comment)](https://github.com/apache/cloudstack/issues/4657#issuecomment-774961189). I will re-check that; hope to find the root cause or a MySQL setup that handles it.
@GabrielBrascher do you mean `wait_timeout` ?
You should change the global value (from 60 to 28800).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher edited a comment on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher edited a comment on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778150967
@DaanHoogland unfortunately I don't have any due date to a patch on this one. So far we are in debugging process and we see it happening on 4.13.1.0; it might not even happen when we upgrade to 4.15.
What we know so far is that the MySQL connections are not growing much, we have 1/3 or less of maximum connections. Thus, a leak would probably be on the Java side.
@weizhouapache here follows some of our MySQL variables:
```
show global variables like "wait_timeout";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout | 60 |
+---------------+-------+
```
```
show status where `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_connected | 21 |
+-------------------+-------+
```
```
SHOW VARIABLES LIKE "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
```
Maybe we can increase the timeout, what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] wido commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
wido commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-774668613
> @wido, max active connections being reached shouldn't kill all connectivity, should it? open connections should still work and when closed new ones should be able to be opened. sound like a serious bug in production.
> Did corruptions occur in the DB?
No, it shouldn't indeed. I also doubt if we are reaching the max connections. Looking at this management server now it has 20 open connections from CloudStack. I doubt if it reaches the 250.
There were/are no known DB corruptions. MariaDB is running just fine. As stated, just restarting the mgmt server is enough to get everything working again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778150967
@DaanHoogland unfortunately I don't have any due date to a patch on this one. So far we are in debugging process and we see it happening on 4.13.1.0; it might not even happen when we upgrade to 4.15.
What we know so far is that the MySQL connections are not growing much, we have 1/3 or less of maximum connections. Thus, a leak would probably be on the Java side.
@weizhouapache here follows some of our MySQL variables:
```
show global variables like "wait_timeout";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout | 60 |
+---------------+-------+
```
```
show status where `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_connected | 21 |
+-------------------+-------+
```
```
SHOW VARIABLES LIKE "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] wido commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
wido commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-774961189
> @wido what's the value of mysql variable "wait_timeout" ?
>
> we had some issues caused by it in the past (it was 600 in our environment).
>
> ```
> MariaDB [cloud]> show variables like "wait_timeout";
> ```
<pre>
MariaDB [(none)]> show variables like "wait_timeout";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout | 28800 |
+---------------+-------+
1 row in set (0.00 sec)
MariaDB [(none)]>
</pre>
The server which this happened to had this issue now happen twice within 24h.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-774446017
@wido, max active connections being reached shouldn't kill all connectivity, should it? open connections should still work and when closed new ones should be able to be opened. sound like a serious bug in production.
Did corruptions occur in the DB?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778078424
@GabrielBrascher any plans for pr/prospect on this (i.e. can we put milestone 4.15.1.0 on this)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache edited a comment on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
weizhouapache edited a comment on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-778232882
> Thanks for the help @weizhouapache.
>
> I think that `max_connections` was set to `28800` and then later to `60` but the errors kept coming either way: [#4657 (comment)](https://github.com/apache/cloudstack/issues/4657#issuecomment-774961189). I will re-check that; hope to find the root cause or a MySQL setup that handles it.
@GabrielBrascher do you mean `wait_timeout` ?
You should change the global value (from 60 to other value e.g. 28800).
```
SET GLOBAL wait_timeout=28800;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4657: Management server looses MySQL connection: Timeout waiting for idle object
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4657:
URL: https://github.com/apache/cloudstack/issues/4657#issuecomment-854112884
It looks like changing the MySQL global variables solved this issue.
Closing this issue for now; I may re-open if it happens again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org