You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Slavka Peleva <sl...@storpool.com.INVALID> on 2021/12/01 08:56:22 UTC
Re: Zone not ready to launch system VMs though they are already running...
Hi Peter,
The error from your usage.log says that the `db.cloud.maxActive` from your
`db.properties` file is empty, and you have to provide a value for it.
Also, if those properties are empty too - `db.cloud.maxIdle` and
`db.cloud.maxWait` - the init of DB will fail again.
Best regards,
Slavka
On Wed, Dec 1, 2021 at 9:55 AM Peter Stine
<pe...@granddial.com.invalid> wrote:
> Wei,
>
> That got the Zone to recognize as up and ready.
> To the database point, in the usage log, I now see that is is failing to
> load the db configuration. As I am not using default logins, this means it
> won't correctly connect to the database. I am not sure what is stopping
> cloudstack from loading the database configuration.
>
> Here is the warning:
> 2021-11-30 18:09:34,747 WARN [db.Transaction.Transaction] (main:null)
> (logid:) Unable to load db configuration, using defaults with 5
> connections. Falling back on assumed datasource on localhost:3306 using
> username:password=cloud:cloud. Please check your configuration
> java.lang.NumberFormatException: null
> at java.base/java.lang.Integer.parseInt(Integer.java:614)
> at java.base/java.lang.Integer.parseInt(Integer.java:770)
> at
> com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1030)
> at
> com.cloud.utils.db.TransactionLegacy.<clinit>(TransactionLegacy.java:1008)
> at
> com.cloud.utils.db.TransactionContextBuilder.interceptStart(TransactionContextBuilder.java:49)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:116)
> at
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.createPartialSelectSql(<generated>)
> at com.cloud.utils.db.GenericDaoBase.<init>(GenericDaoBase.java:238)
> at
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl.<init>(UsagePortForwardingRuleDaoImpl.java:50)
> at
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.<init>(<generated>)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:291)
> at
> net.sf.cglib.proxy.Enhancer$EnhancerFactoryData.newInstance(Enhancer.java:448)
> at net.sf.cglib.proxy.Enhancer.nextInstance(Enhancer.java:700)
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:298)
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:480)
> at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:305)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor.postProcessBeforeInstantiation(ComponentInstantiationPostProcessor.java:92)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1144)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1119)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:923)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:588)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:85)
> at com.cloud.usage.UsageServer.start(UsageServer.java:52)
> at com.cloud.usage.UsageServer.main(UsageServer.java:41)
>
> ----------------------------------------
> From: "Wei ZHOU" <us...@gmail.com>
> Sent: 11/30/21 1:00 PM
> To: users <us...@cloudstack.apache.org>, peter.stine@granddial.com
> Subject: Re: Zone not ready to launch system VMs though they are already
> running...
>
> Hi Peter,
>
> This seems to be caused by database connection problems. Try to restart
> cloudstack management service, and also mariadb if it still doesn't work.
>
> -Wei
>
> On Tue, 30 Nov 2021 at 18:37, Peter Stine <pe...@granddial.com.invalid>
> wrote:
> Hello everyone,
>
> I was getting a warning for high CPU usage on my management servers and
> found that the spikes were in line with the cloudstack logs showing a
> message that the Zone is not prepared to launch the consoleproxy and
> secondary storage VMs.
>
> 2021-11-30 09:40:58,957 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 has no host available
> which is enabled and in Up state
> 2021-11-30 09:40:58,958 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 is not ready to launch
> secondary storage VM yet
> 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 has no host available
> which is enabled and in Up state
> 2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 is not ready to
> launch console proxy yet
>
> The odd thing is that both of those VMs are currently running, accessible,
> and can be modified from the webUI. The zone also has 2 hosts, both of
> which seem to be operating just fine.
> However, there seems to be a Lockmaster1 java exception on the management
> servers for the db connection. I think this might be part of the issue, but
> I cannot find anything about it.
>
> This post seems relevant (
> https://blog.widodh.nl/2013/04/cloudstack-zone-x-is-is-not-ready-to-launch-console-proxy-yet/),
> but seems only to refer to an issue when the VMs won't launch.
>
> Has anyone run across this before and found a solution?
>
> Cloudstack version: 4.15.2
> My systems are running Ubuntu 20.04
> Hypervisor is KVM.
> My database (MariaDB) is running in a galera cluster (I have confirmed
> that I can log into the database from each server)
> My 3 management servers are in a corosync with a HA IP (That IP is where
> the database connections are pointed)
>
> Management logs:
> Management 2 (main at this point):
> https://gist.github.com/PeterS-gd/10aff04f4204ebebf9066537dea06bac
> Management 1 (in cluster):
> https://gist.github.com/PeterS-gd/e9e29e030def574313a9168b581d71ed
> Management 3 (in cluster):
> https://gist.github.com/PeterS-gd/4effc4ddfe7d4a360cc518c121407ee3
>
> On the server that has the HA IP, the sql error log shows this:
> 2021-11-30 14:31:14 489 [Warning] Aborted connection 489 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 461 [Warning] Aborted connection 461 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 521 [Warning] Aborted connection 521 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 515 [Warning] Aborted connection 515 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 251 [Warning] Aborted connection 251 to db:
> 'cloud_usage' user: 'cloud' host: 'gd-cs-control-server' (Got an error
> reading communication packets)
> 2021-11-30 14:31:14 490 [Warning] Aborted connection 490 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 452 [Warning] Aborted connection 452 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 445 [Warning] Aborted connection 445 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 514 [Warning] Aborted connection 514 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
> 2021-11-30 14:31:14 502 [Warning] Aborted connection 502 to db: 'cloud'
> user: 'cloud' host: 'gd-cs-control-server' (Got an error reading
> communication packets)
>
> Thanks!
> Peter
>
Re: Zone not ready to launch system VMs though they are already running...
Posted by Peter Stine <pe...@granddial.com.INVALID>.
Hi Wei,
Apparently, my symlinks got deleted somehow. Recreating them fixed the issue. The CPUs are no longer spiking as high as before. Thanks for your help!
Peter
----------------------------------------
From: "Wei ZHOU" <us...@gmail.com>
Sent: 12/1/21 12:00 PM
To: peter.stine@granddial.com
Subject: Re: Zone not ready to launch system VMs though they are already running...
Hi Peter,
Could you please check /etc/cloudstack/usage/db.properties, it should be a symlink to /etc/cloudstack/management/db.properties
-Wei
On Wed, 1 Dec 2021 at 16:25, Peter Stine <pe...@granddial.com> wrote:
Hi Slavka,
I checked the db.properties on all the management servers and all of those parameters were filled.
db.cloud.maxActive=750
db.cloud.maxIdle=90
db.cloud.maxWait=10000
db.cloud.validationQuery=SELECT 1
db.cloud.testOnBorrow=true
db.cloud.testWhileIdle=true
db.cloud.timeBetweenEvictionRunsMillis=40000
db.cloud.minEvictableIdleTimeMillis=240000
db.cloud.poolPreparedStatements=false
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC
I did try raising the numbers for max active and max idle, but after restarting the service, it didn't seem to make a difference as the warning still persists.
Peter
----------------------------------------
From: "Slavka Peleva" <sl...@storpool.com>
Sent: 12/1/21 3:59 AM
To: users@cloudstack.apache.org, peter.stine@granddial.com
Cc: Wei ZHOU <us...@gmail.com>
Subject: Re: Zone not ready to launch system VMs though they are already running...
Hi Peter,
The error from your usage.log says that the `db.cloud.maxActive` from your `db.properties` file is empty, and you have to provide a value for it. Also, if those properties are empty too - `db.cloud.maxIdle` and `db.cloud.maxWait` - the init of DB will fail again.
Best regards,
Slavka
On Wed, Dec 1, 2021 at 9:55 AM Peter Stine <pe...@granddial.com.invalid> wrote:
Wei,
That got the Zone to recognize as up and ready.
To the database point, in the usage log, I now see that is is failing to load the db configuration. As I am not using default logins, this means it won't correctly connect to the database. I am not sure what is stopping cloudstack from loading the database configuration.
Here is the warning:
2021-11-30 18:09:34,747 WARN [db.Transaction.Transaction] (main:null) (logid:) Unable to load db configuration, using defaults with 5 connections. Falling back on assumed datasource on localhost:3306 using username:password=cloud:cloud. Please check your configuration
java.lang.NumberFormatException: null
at java.base/java.lang.Integer.parseInt(Integer.java:614)
at java.base/java.lang.Integer.parseInt(Integer.java:770)
at com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1030)
at com.cloud.utils.db.TransactionLegacy.<clinit>(TransactionLegacy.java:1008)
at com.cloud.utils.db.TransactionContextBuilder.interceptStart(TransactionContextBuilder.java:49)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:116)
at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.createPartialSelectSql(<generated>)
at com.cloud.utils.db.GenericDaoBase.<init>(GenericDaoBase.java:238)
at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl.<init>(UsagePortForwardingRuleDaoImpl.java:50)
at com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_56282efc.<init>(<generated>)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:291)
at net.sf.cglib.proxy.Enhancer$EnhancerFactoryData.newInstance(Enhancer.java:448)
at net.sf.cglib.proxy.Enhancer.nextInstance(Enhancer.java:700)
at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:298)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:480)
at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:305)
at com.cloud.utils.component.ComponentInstantiationPostProcessor.postProcessBeforeInstantiation(ComponentInstantiationPostProcessor.java:92)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1144)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1119)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:923)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:588)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:85)
at com.cloud.usage.UsageServer.start(UsageServer.java:52)
at com.cloud.usage.UsageServer.main(UsageServer.java:41)
----------------------------------------
From: "Wei ZHOU" <us...@gmail.com>
Sent: 11/30/21 1:00 PM
To: users <us...@cloudstack.apache.org>, peter.stine@granddial.com
Subject: Re: Zone not ready to launch system VMs though they are already running...
Hi Peter,
This seems to be caused by database connection problems. Try to restart cloudstack management service, and also mariadb if it still doesn't work.
-Wei
On Tue, 30 Nov 2021 at 18:37, Peter Stine <pe...@granddial.com.invalid> wrote:
Hello everyone,
I was getting a warning for high CPU usage on my management servers and found that the spikes were in line with the cloudstack logs showing a message that the Zone is not prepared to launch the consoleproxy and secondary storage VMs.
2021-11-30 09:40:58,957 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 has no host available which is enabled and in Up state
2021-11-30 09:40:58,958 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-ff6437c2) (logid:b30a1545) Zone 2 is not ready to launch secondary storage VM yet
2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 has no host available which is enabled and in Up state
2021-11-30 09:40:59,751 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-95609472) (logid:6b4db700) Zone 2 is not ready to launch console proxy yet
The odd thing is that both of those VMs are currently running, accessible, and can be modified from the webUI. The zone also has 2 hosts, both of which seem to be operating just fine.
However, there seems to be a Lockmaster1 java exception on the management servers for the db connection. I think this might be part of the issue, but I cannot find anything about it.
This post seems relevant (https://blog.widodh.nl/2013/04/cloudstack-zone-x-is-is-not-ready-to-launch-console-proxy-yet/), but seems only to refer to an issue when the VMs won't launch.
Has anyone run across this before and found a solution?
Cloudstack version: 4.15.2
My systems are running Ubuntu 20.04
Hypervisor is KVM.
My database (MariaDB) is running in a galera cluster (I have confirmed that I can log into the database from each server)
My 3 management servers are in a corosync with a HA IP (That IP is where the database connections are pointed)
Management logs:
Management 2 (main at this point): https://gist.github.com/PeterS-gd/10aff04f4204ebebf9066537dea06bac
Management 1 (in cluster): https://gist.github.com/PeterS-gd/e9e29e030def574313a9168b581d71ed
Management 3 (in cluster): https://gist.github.com/PeterS-gd/4effc4ddfe7d4a360cc518c121407ee3
On the server that has the HA IP, the sql error log shows this:
2021-11-30 14:31:14 489 [Warning] Aborted connection 489 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 461 [Warning] Aborted connection 461 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 521 [Warning] Aborted connection 521 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 515 [Warning] Aborted connection 515 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 251 [Warning] Aborted connection 251 to db: 'cloud_usage' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 490 [Warning] Aborted connection 490 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 452 [Warning] Aborted connection 452 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 445 [Warning] Aborted connection 445 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 514 [Warning] Aborted connection 514 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
2021-11-30 14:31:14 502 [Warning] Aborted connection 502 to db: 'cloud' user: 'cloud' host: 'gd-cs-control-server' (Got an error reading communication packets)
Thanks!
Peter