You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Francois Gaudreault <fg...@cloudops.com> on 2013/10/08 17:04:05 UTC

Latest Master DB issue

Hi,

I compiled Master this morning, and there is a small DB issue. One field 
is missing in the account table (field default). CS will not start 
because of that.

2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null) 
Exiting the system because we're unable to register the system call context.
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: 
com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id, 
account.account_name, account.type, account.domain_id, account.state, 
account.removed, account.cleanup_needed, account.network_domain, 
account.uuid, account.default_zone_id, account.default FROM account 
WHERE account.id = 1  AND account.removed IS NULL
     at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
     at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
     at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
     at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
     at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
     at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
     at com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
     at 
org.apache.cloudstack.context.CallContext.register(CallContext.java:166)
     at 
org.apache.cloudstack.context.CallContext.registerSystemCallContextOnceOnly(CallContext.java:141)
     at 
org.apache.cloudstack.context.CallContextListener.onEnterContext(CallContextListener.java:36)
     at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:83)
     at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
     at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
     at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
     at java.util.TimerThread.mainLoop(Timer.java:534)
     at java.util.TimerThread.run(Timer.java:484)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: 
Unknown column 'account.default' in 'field list'
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
     at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
     at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
     at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
     at com.mysql.jdbc.Util.getInstance(Util.java:386)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
     at 
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
     at 
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
     at 
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
     at 
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
     at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
     ... 27 more


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

Just throwing in a pitch for my spring-modularization branch.  The DB
initialization stuff should all be fixed in the spring-modularization
branch.  If it isn't we can better address it there anyhow.  The
current initialization of spring in master is pure madness.  When it
works, its because it just happens to work at that time.  Adding new
beans and dependencies can easily break the initialization order as
seen over and over again with the DB upgrades.

Darren

On Tue, Oct 8, 2013 at 10:47 AM, Alena Prokharchyk
<Al...@citrix.com> wrote:
> Ok, this is what going on - the DB upgrade procedure is different on
> developer's setup and when deployed using cloudstack-setup-databases
>
> On developers setup:
>
> 1) you deploy the code
> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
> part of this step, the DataBaseUpgradeChecker:
>
> * first deploys the base DB version - 4.0.0
> * then checks the current version of the code, and performs the db upgrade
> if needed. So on master, version table looks like this after the db is
> deployed:
>
> mysql> select * from version;
> +----+---------+---------------------+----------+
> | id | version | updated             | step     |
> +----+---------+---------------------+----------+
> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
> +----+---------+---------------------+----------+
> 4 rows in set (0.00 sec)
>
>
> 3) Start management server.
>
>
> When deployed from rpm:
>
> 1) you deploy the code
> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0 base
> version of the DB is deployed. Thats why you see only 4.0.0 record in the
> DB.
> 3) Start management server. DataBaseUpgradeChecker is being invoked as a
> part of it, and performs the db upgrade to the version of the current
> code. Only after that all the managers get invoked + system caller context
> get initialized.
>
> Looks like the load order for step 3) got broken recently, and system
> context gets initialized before the db upgrade is finished. So we either
> need to fix the order, or invoke DataBaseUpgradeChecker as a part of
> cloudstack-setup-databases so at the point when management server starts
> up, the DB already upgraded to the latest version.
>
> -Alena.
>
>
> On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
> wrote:
>
>>Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>>no?
>>
>>mysql> select * from version;
>>+----+---------+---------------------+----------+
>>| id | version | updated             | step     |
>>+----+---------+---------------------+----------+
>>|  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>+----+---------+---------------------+----------+
>>1 row in set (0.00 sec)
>>
>>I installed cloudstack-management-4.3.0:
>>[root@eng-testing-cstack_master ~]# rpm -qf
>>/usr/bin/cloudstack-setup-databases
>>cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>
>>Francois
>>
>>On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>> It's a fresh master RPM install.
>>>
>>> Francois
>>>
>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>> It is not a small issue. is_default filed was added to the table as a
>>>> part
>>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>>> user
>>>> before the db upgrade is completed.
>>>>
>>>> DB upgrade is a major part of system integrity check; no queries to
>>>> the DB
>>>> should be made before its completed. Francois, did you start seeing
>>>>this
>>>> problem just recently?
>>>>
>>>> -Alena.
>>>>
>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I compiled Master this morning, and there is a small DB issue. One
>>>>> field
>>>>> is missing in the account table (field default). CS will not start
>>>>> because of that.
>>>>>
>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>>> Exiting the system because we're unable to register the system call
>>>>> context.
>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>>> account.account_name, account.type, account.domain_id, account.state,
>>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>>      at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>      at
>>>>>
>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>      at
>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>      at
>>>>>
>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>      at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>      at
>>>>>
>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>      at
>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.context.CallContext.register(CallContext.java:166
>>>>>)
>>>>>
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.context.CallContext.registerSystemCallContextOnce
>>>>>Onl
>>>>>
>>>>> y(CallContext.java:141)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.context.CallContextListener.onEnterContext(CallCo
>>>>>nte
>>>>>
>>>>> xtListener.java:36)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>thC
>>>>>
>>>>> ontext(DefaultManagedContext.java:83)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>hCo
>>>>>
>>>>> ntext(DefaultManagedContext.java:53)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>dCo
>>>>>
>>>>> ntextRunnable.java:46)
>>>>>      at
>>>>>
>>>>>org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Manag
>>>>>edC
>>>>>
>>>>> ontextTimerTask.java:27)
>>>>>      at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>      at java.util.TimerThread.run(Timer.java:484)
>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>> Unknown column 'account.default' in 'field list'
>>>>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>      at
>>>>>
>>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
>>>>>Acc
>>>>>
>>>>> essorImpl.java:57)
>>>>>      at
>>>>>
>>>>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
>>>>>str
>>>>>
>>>>> uctorAccessorImpl.java:45)
>>>>>      at
>>>>>java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>      at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>      at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>      at
>>>>>com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>      at
>>>>>
>>>>>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
>>>>>:21
>>>>>
>>>>> 55)
>>>>>      at
>>>>>
>>>>>com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:23
>>>>>18)
>>>>>
>>>>>      at
>>>>>
>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>tin
>>>>>
>>>>> gPreparedStatement.java:96)
>>>>>      at
>>>>>
>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>tin
>>>>>
>>>>> gPreparedStatement.java:96)
>>>>>      at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>      ... 27 more
>>>>>
>>>>>
>>>>> --
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>> fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>--
>>Francois Gaudreault
>>Architecte de Solution Cloud | Cloud Solutions Architect
>>fgaudreault@cloudops.com
>>514-629-6775
>>- - -
>>CloudOps
>>420 rue Guy
>>Montréal QC  H3J 1S6
>>www.cloudops.com
>>@CloudOps_
>>
>>
>
>

RE: Latest Master DB issue

Posted by Rayees Namathponnan <ra...@citrix.com>.

Any idea when we can get fix for https://issues.apache.org/jira/browse/CLOUDSTACK-4825 ? 

Regards,
Rayees 

-----Original Message-----
From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com] 
Sent: Wednesday, October 09, 2013 7:42 AM
To: dev@cloudstack.apache.org
Subject: Re: Latest Master DB issue

I think I'll solve this with a fourth option.  I don't particularly like two thing.  Threads that never access the DB before are now doing it (albeit just once) and if there is any error calling the DB it does exit(1).  I'm going to make the CallingContext instance that gets registered for the system account do lazy loading of the User and Account objects.  So the system CallContext will still be available on all thread, but if you never access it, it will never hit the database.  Second, I'm going to remove the exit(1) that was already in the code.  Its just a bad idea.  If you were to ever exhauster your DB connection pool (or the number of connections on mysql), your entire mgmt stack will kill itself and by default we don't run ACS under process supervision.

Darren

On Wed, Oct 9, 2013 at 1:59 AM, Darren Shepherd <da...@gmail.com> wrote:
> Kelven,
>
> So the issue is the combination of my code with VmwareContextPool.
> With the ManagedContext framework, what I've done is replace every 
> Runnable and TimerTask with ManagedContextRunnable and 
> ManagedContextTimerTask.  Those classes will run the onEnter, onLeave 
> logic which will setup CallContext as a result.  I purposely changed 
> every reference so that everything was consistent.  I didn't want to 
> have developers have to consider when or when not to use 
> ManagedContext, so just always use it.  So as a result even though 
> your code has nothing to do with the DB, the ManageContextListener for 
> CallContext does.
>
> So I'm sure your thinking that Resources shouldn't call the DB.  The 
> ManagedContext framework only does things when deployed in a managed 
> JVM.  The only managed JVM is the mgmt server.  If it's AWSAPI, Usage, 
> or an Agent JVM, then that framework does nothing.
>
> So there are three possible solutions I see
>
> 1) Change VmwareContextPool to be initialized from the @PostConstruct 
> or start() method.
> 2) Revert the change to VmwareContextPool to use TimerTask and not 
> ManageContextTimerTask
> 3) Merge spring modularization.
>
> The simplest stop gap would be option 2.
>
> Darren
>
> On Tue, Oct 8, 2013 at 5:35 PM, Kelven Yang <ke...@citrix.com> wrote:
>> The problem seems to me is whether or not a background job that 
>> touches with database respects the bootstrap initialization order. As 
>> of VmwareContextPool itself, its background job does something fully 
>> within its own territory (no database, no reference outside). and 
>> vmware-base package was originally designed to be running on its own 
>> without assuming any container that offers unified lifecycle 
>> management. I don't think this type of background job has anything to 
>> do with the failure in this particular case.
>>
>> However, I do agree that we need to clean up and unify a few things 
>> inside the CloudStack, especially on life-cycle management and all 
>> background-jobs that their execution path touches with component 
>> life-cyle, auto-wiring, AOP etc.
>>
>> To live with the time before the spring modularization merge, we just 
>> need to figure out which background job that triggers all these and 
>> get it fixed, it used to work before even it is fragile, I don't 
>> think the fix of the problem is impossible. Is anyone working on this issue?
>>
>> Kelven
>>
>>
>>
>>
>> On 10/8/13 2:35 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>>
>>>Some more info about this.  What specifically is happening is that 
>>>the VmwareContextPool call is creating a Timer during the constructor 
>>>of the class which is being constructed in a static block from 
>>>VmwareContextFactory.  So when the VmwareContextFactory class is 
>>>loaded by the class loader, the background thread is created.  Which 
>>>is way, way before the Database upgrade happens.  This will still be 
>>>fixed if we merge the spring modularization, but this vmware code 
>>>should change regardless.  Background threads should only be launched 
>>>from a @PostConstruct or ComponentLifecycle.start() method.  They 
>>>should not be started when a class is constructed or loaded.
>>>
>>>Darren
>>>
>>>
>>>On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd 
>>><da...@gmail.com> wrote:
>>>> Hey, I half way introduced this issue in a really long and round 
>>>> about way.  I don't think there's a good simple fix unless we merge 
>>>> the spring-modularization branch.  I'm going to look further into 
>>>> it.  But here's the background of why we are seeing this.
>>>>
>>>> I introduced "Managed Context" framework that will wrap all the 
>>>> background threads and manage the thread locals.  This was the 
>>>> union of CallContext, ServerContext, and AsyncJob*Context into one 
>>>> simple framework.  The problem with ACS though is that A LOT of 
>>>> background threads are spawned at all different random times of the 
>>>> initialization.  So what is happening is that during the 
>>>> initialization of some bean its kicking off a background thread 
>>>> that tries to access the database before the database upgrade has 
>>>> ran.  Now the CallContext has a strange suicidal behaviour (this 
>>>> was already there, I didn't change this), if it can't find account 
>>>> 1, it does a System.exit(1).  So since this one background thread 
>>>> is failing, the whole JVM shuts down.  Before CallContext only 
>>>> existed on some threads, but the addition of the Managed Context 
>>>> framework, it is now on almost all threads.
>>>>
>>>> Now in the spring-modularization branch there is a very strict and
>>>> (mostly) deterministic initialization order.  The database upgrade 
>>>> class will be initialized and ran before any other bean in 
>>>> CloudStack is even initiated.  So this works around all these DB 
>>>> problems.  The current spring setup in master is very, very 
>>>> fragile.  As I said before, it is really difficult to ensure 
>>>> certain aspects are initialized before others, and since we moved 
>>>> to (which I don't really agree with) doing DB schema upgrades 
>>>> purely on startup of the mgmt server, we now have to be extra careful about initialization order.
>>>>
>>>> Darren
>>>>
>>>> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan 
>>>> <ra...@citrix.com> wrote:
>>>>> Here the defect created for this issue
>>>>>
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>>>>>
>>>>>
>>>>> Regards,
>>>>> Rayees
>>>>>
>>>>> -----Original Message-----
>>>>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>>>>> Sent: Tuesday, October 08, 2013 11:15 AM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: Latest Master DB issue
>>>>>
>>>>> I guess in my case, it's fine. It was a fresh install...
>>>>>
>>>>> Francois
>>>>>
>>>>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>>>>>> Deploy db from maven will drop all the tables.  Not sure if this 
>>>>>> is fresh install or not.  For master, running mvn will be your best bet.
>>>>>> Otherwise you can look at running 
>>>>>> com.cloud.upgrade.DatabaseCreator manually if your adventurous.
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault 
>>>>>> <fg...@cloudops.com> wrote:
>>>>>>> Thanks Alena for the explaination.
>>>>>>>
>>>>>>> What is the better path to fix this on our setup? Should I wait 
>>>>>>> for a fix in master or should I manually run the deploydb with 
>>>>>>> mvn? I guess the second option won't work since I used RPM?
>>>>>>>
>>>>>>> Francois
>>>>>>>
>>>>>>>
>>>>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>>>>>> Ok, this is what going on - the DB upgrade procedure is 
>>>>>>>> different on developer's setup and when deployed using 
>>>>>>>> cloudstack-setup-databases
>>>>>>>>
>>>>>>>> On developers setup:
>>>>>>>>
>>>>>>>> 1) you deploy the code
>>>>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>>>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>>>>>
>>>>>>>> * first deploys the base DB version - 4.0.0
>>>>>>>> * then checks the current version of the code, and performs the 
>>>>>>>> db upgrade if needed. So on master, version table looks like 
>>>>>>>> this after the db is
>>>>>>>> deployed:
>>>>>>>>
>>>>>>>> mysql> select * from version;
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> | id | version | updated             | step     |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>>>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> 4 rows in set (0.00 sec)
>>>>>>>>
>>>>>>>>
>>>>>>>> 3) Start management server.
>>>>>>>>
>>>>>>>>
>>>>>>>> When deployed from rpm:
>>>>>>>>
>>>>>>>> 1) you deploy the code
>>>>>>>> 2) run cloudstack-setup-databases. As the result of this step, 
>>>>>>>> 4.0.0 base version of the DB is deployed. Thats why you see 
>>>>>>>> only 4.0.0 record in the DB.
>>>>>>>> 3) Start management server. DataBaseUpgradeChecker is being 
>>>>>>>> invoked as a part of it, and performs the db upgrade to the 
>>>>>>>> version of the current code. Only after that all the managers 
>>>>>>>> get invoked + system caller context get initialized.
>>>>>>>>
>>>>>>>> Looks like the load order for step 3) got broken recently, and  
>>>>>>>>system context gets initialized before the db upgrade is finished.
>>>>>>>> So we either need to fix the order, or invoke 
>>>>>>>>DataBaseUpgradeChecker  as a part of cloudstack-setup-databases 
>>>>>>>>so at the point when  management server starts up, the DB 
>>>>>>>>already upgraded to the latest version.
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>>>>>> <fg...@cloudops.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should 
>>>>>>>>> be
>>>>>>>>> 4.3.0 no?
>>>>>>>>>
>>>>>>>>> mysql> select * from version;
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> | id | version | updated             | step     |
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>
>>>>>>>>> I installed cloudstack-management-4.3.0:
>>>>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf 
>>>>>>>>> /usr/bin/cloudstack-setup-databases
>>>>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>>>>>
>>>>>>>>> Francois
>>>>>>>>>
>>>>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>>>>>> It's a fresh master RPM install.
>>>>>>>>>>
>>>>>>>>>> Francois
>>>>>>>>>>
>>>>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>>>>>> It is not a small issue. is_default filed was added to the 
>>>>>>>>>>> table as a part of the 41-42 db upgrade. Looks like the code 
>>>>>>>>>>> tries to retrieve system user before the db upgrade is completed.
>>>>>>>>>>>
>>>>>>>>>>> DB upgrade is a major part of system integrity check; no 
>>>>>>>>>>> queries to the DB should be made before its completed. 
>>>>>>>>>>> Francois, did you start seeing this problem just recently?
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>>>>>> <fg...@cloudops.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>>>>>> One field is missing in the account table (field default). 
>>>>>>>>>>>> CS will not start because of that.
>>>>>>>>>>>>
>>>>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to  
>>>>>>>>>>>>register the system call context.
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB 
>>>>>>>>>>>>Exception
>>>>>>>>>>>>on:
>>>>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT  
>>>>>>>>>>>>account.id, account.account_name, account.type,  
>>>>>>>>>>>>account.domain_id, account.state, account.removed,  
>>>>>>>>>>>>account.cleanup_needed, account.network_domain, 
>>>>>>>>>>>>account.uuid,  account.default_zone_id, account.default FROM 
>>>>>>>>>>>>account WHERE account.id = 1  AND account.removed IS NULL
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja
>>>>>>>>>>>>va:98
>>>>>>>>>>>>6)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>> or$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.jav
>>>>>>>>>>>>a:963
>>>>>>>>>>>>)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>> or$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja
>>>>>>>>>>>>va:92
>>>>>>>>>>>>6)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>> or$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.j
>>>>>>>>>>>>ava:4
>>>>>>>>>>>>5)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallCont
>>>>>>>>>>>> ext.j
>>>>>>>>>>>> ava:166
>>>>>>>>>>>> )
>>>>>>>>>>>>
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCal
>>>>>>>>>>>> lCont
>>>>>>>>>>>> extOnce
>>>>>>>>>>>> Onl
>>>>>>>>>>>>
>>>>>>>>>>>> y(CallContext.java:141)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterCo
>>>>>>>>>>>> ntext
>>>>>>>>>>>> (CallCo
>>>>>>>>>>>> nte
>>>>>>>>>>>>
>>>>>>>>>>>> xtListener.java:36)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedCo
>>>>>>>>>>>> ntext
>>>>>>>>>>>> .callWi
>>>>>>>>>>>> thC
>>>>>>>>>>>>
>>>>>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedCo
>>>>>>>>>>>> ntext
>>>>>>>>>>>> .runWit
>>>>>>>>>>>> hCo
>>>>>>>>>>>>
>>>>>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnabl
>>>>>>>>>>>> e.run
>>>>>>>>>>>> (Manage
>>>>>>>>>>>> dCo
>>>>>>>>>>>>
>>>>>>>>>>>> ntextRunnable.java:46)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTa
>>>>>>>>>>>> sk.ru
>>>>>>>>>>>> n(Manag
>>>>>>>>>>>> edC
>>>>>>>>>>>>
>>>>>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance0(Nativ
>>>>>>>>>>>>e
>>>>>>>>>>>> Method)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(Nativ
>>>>>>>>>>>> eCons
>>>>>>>>>>>> tructor
>>>>>>>>>>>> Acc
>>>>>>>>>>>>
>>>>>>>>>>>> essorImpl.java:57)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(D
>>>>>>>>>>>> elega
>>>>>>>>>>>> tingCon
>>>>>>>>>>>> str
>>>>>>>>>>>>
>>>>>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>>>>>        at
>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>>>>>        at
>>>>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>>>>>        at
>>>>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedSt
>>>>>>>>>>>> ateme
>>>>>>>>>>>> nt.java
>>>>>>>>>>>> :21
>>>>>>>>>>>>
>>>>>>>>>>>> 55)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>>>>>> java:23
>>>>>>>>>>>> 18)
>>>>>>>>>>>>
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute
>>>>>>>>>>>> Query
>>>>>>>>>>>> (Delega
>>>>>>>>>>>> tin
>>>>>>>>>>>>
>>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute
>>>>>>>>>>>> Query
>>>>>>>>>>>> (Delega
>>>>>>>>>>>> tin
>>>>>>>>>>>>
>>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja
>>>>>>>>>>>>va:98
>>>>>>>>>>>>3)
>>>>>>>>>>>>        ... 27 more
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Francois Gaudreault
>>>>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>>>>> 514-629-6775
>>>>>>>>>>>> - - -
>>>>>>>>>>>> CloudOps
>>>>>>>>>>>> 420 rue Guy
>>>>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>>>>> www.cloudops.com
>>>>>>>>>>>> @CloudOps_
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Francois Gaudreault
>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>> 514-629-6775
>>>>>>>>> - - -
>>>>>>>>> CloudOps
>>>>>>>>> 420 rue Guy
>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>> www.cloudops.com
>>>>>>>>> @CloudOps_
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Francois Gaudreault
>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>>>>> fgaudreault@cloudops.com
>>>>>>> 514-629-6775
>>>>>>> - - -
>>>>>>> CloudOps
>>>>>>> 420 rue Guy
>>>>>>> Montréal QC  H3J 1S6
>>>>>>> www.cloudops.com
>>>>>>> @CloudOps_
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>>>fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

I think I'll solve this with a fourth option.  I don't particularly
like two thing.  Threads that never access the DB before are now doing
it (albeit just once) and if there is any error calling the DB it does
exit(1).  I'm going to make the CallingContext instance that gets
registered for the system account do lazy loading of the User and
Account objects.  So the system CallContext will still be available on
all thread, but if you never access it, it will never hit the
database.  Second, I'm going to remove the exit(1) that was already in
the code.  Its just a bad idea.  If you were to ever exhauster your DB
connection pool (or the number of connections on mysql), your entire
mgmt stack will kill itself and by default we don't run ACS under
process supervision.

Darren

On Wed, Oct 9, 2013 at 1:59 AM, Darren Shepherd
<da...@gmail.com> wrote:
> Kelven,
>
> So the issue is the combination of my code with VmwareContextPool.
> With the ManagedContext framework, what I've done is replace every
> Runnable and TimerTask with ManagedContextRunnable and
> ManagedContextTimerTask.  Those classes will run the onEnter, onLeave
> logic which will setup CallContext as a result.  I purposely changed
> every reference so that everything was consistent.  I didn't want to
> have developers have to consider when or when not to use
> ManagedContext, so just always use it.  So as a result even though
> your code has nothing to do with the DB, the ManageContextListener for
> CallContext does.
>
> So I'm sure your thinking that Resources shouldn't call the DB.  The
> ManagedContext framework only does things when deployed in a managed
> JVM.  The only managed JVM is the mgmt server.  If it's AWSAPI, Usage,
> or an Agent JVM, then that framework does nothing.
>
> So there are three possible solutions I see
>
> 1) Change VmwareContextPool to be initialized from the @PostConstruct
> or start() method.
> 2) Revert the change to VmwareContextPool to use TimerTask and not
> ManageContextTimerTask
> 3) Merge spring modularization.
>
> The simplest stop gap would be option 2.
>
> Darren
>
> On Tue, Oct 8, 2013 at 5:35 PM, Kelven Yang <ke...@citrix.com> wrote:
>> The problem seems to me is whether or not a background job that touches
>> with database respects the bootstrap initialization order. As of
>> VmwareContextPool itself, its background job does something fully within
>> its own territory (no database, no reference outside). and vmware-base
>> package was originally designed to be running on its own without assuming
>> any container that offers unified lifecycle management. I don't think this
>> type of background job has anything to do with the failure in this
>> particular case.
>>
>> However, I do agree that we need to clean up and unify a few things inside
>> the CloudStack, especially on life-cycle management and all
>> background-jobs that their execution path touches with component
>> life-cyle, auto-wiring, AOP etc.
>>
>> To live with the time before the spring modularization merge, we just need
>> to figure out which background job that triggers all these and get it
>> fixed, it used to work before even it is fragile, I don't think the fix of
>> the problem is impossible. Is anyone working on this issue?
>>
>> Kelven
>>
>>
>>
>>
>> On 10/8/13 2:35 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>>
>>>Some more info about this.  What specifically is happening is that the
>>>VmwareContextPool call is creating a Timer during the constructor of
>>>the class which is being constructed in a static block from
>>>VmwareContextFactory.  So when the VmwareContextFactory class is
>>>loaded by the class loader, the background thread is created.  Which
>>>is way, way before the Database upgrade happens.  This will still be
>>>fixed if we merge the spring modularization, but this vmware code
>>>should change regardless.  Background threads should only be launched
>>>from a @PostConstruct or ComponentLifecycle.start() method.  They
>>>should not be started when a class is constructed or loaded.
>>>
>>>Darren
>>>
>>>
>>>On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd
>>><da...@gmail.com> wrote:
>>>> Hey, I half way introduced this issue in a really long and round about
>>>> way.  I don't think there's a good simple fix unless we merge the
>>>> spring-modularization branch.  I'm going to look further into it.  But
>>>> here's the background of why we are seeing this.
>>>>
>>>> I introduced "Managed Context" framework that will wrap all the
>>>> background threads and manage the thread locals.  This was the union
>>>> of CallContext, ServerContext, and AsyncJob*Context into one simple
>>>> framework.  The problem with ACS though is that A LOT of background
>>>> threads are spawned at all different random times of the
>>>> initialization.  So what is happening is that during the
>>>> initialization of some bean its kicking off a background thread that
>>>> tries to access the database before the database upgrade has ran.  Now
>>>> the CallContext has a strange suicidal behaviour (this was already
>>>> there, I didn't change this), if it can't find account 1, it does a
>>>> System.exit(1).  So since this one background thread is failing, the
>>>> whole JVM shuts down.  Before CallContext only existed on some
>>>> threads, but the addition of the Managed Context framework, it is now
>>>> on almost all threads.
>>>>
>>>> Now in the spring-modularization branch there is a very strict and
>>>> (mostly) deterministic initialization order.  The database upgrade
>>>> class will be initialized and ran before any other bean in CloudStack
>>>> is even initiated.  So this works around all these DB problems.  The
>>>> current spring setup in master is very, very fragile.  As I said
>>>> before, it is really difficult to ensure certain aspects are
>>>> initialized before others, and since we moved to (which I don't really
>>>> agree with) doing DB schema upgrades purely on startup of the mgmt
>>>> server, we now have to be extra careful about initialization order.
>>>>
>>>> Darren
>>>>
>>>> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan
>>>> <ra...@citrix.com> wrote:
>>>>> Here the defect created for this issue
>>>>>
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>>>>>
>>>>>
>>>>> Regards,
>>>>> Rayees
>>>>>
>>>>> -----Original Message-----
>>>>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>>>>> Sent: Tuesday, October 08, 2013 11:15 AM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: Latest Master DB issue
>>>>>
>>>>> I guess in my case, it's fine. It was a fresh install...
>>>>>
>>>>> Francois
>>>>>
>>>>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>>>>>> Deploy db from maven will drop all the tables.  Not sure if this is
>>>>>> fresh install or not.  For master, running mvn will be your best bet.
>>>>>> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
>>>>>> manually if your adventurous.
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
>>>>>> <fg...@cloudops.com> wrote:
>>>>>>> Thanks Alena for the explaination.
>>>>>>>
>>>>>>> What is the better path to fix this on our setup? Should I wait for a
>>>>>>> fix in master or should I manually run the deploydb with mvn? I guess
>>>>>>> the second option won't work since I used RPM?
>>>>>>>
>>>>>>> Francois
>>>>>>>
>>>>>>>
>>>>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>>>>>> Ok, this is what going on - the DB upgrade procedure is different on
>>>>>>>> developer's setup and when deployed using cloudstack-setup-databases
>>>>>>>>
>>>>>>>> On developers setup:
>>>>>>>>
>>>>>>>> 1) you deploy the code
>>>>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>>>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>>>>>
>>>>>>>> * first deploys the base DB version - 4.0.0
>>>>>>>> * then checks the current version of the code, and performs the db
>>>>>>>> upgrade if needed. So on master, version table looks like this after
>>>>>>>> the db is
>>>>>>>> deployed:
>>>>>>>>
>>>>>>>> mysql> select * from version;
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> | id | version | updated             | step     |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>>>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> 4 rows in set (0.00 sec)
>>>>>>>>
>>>>>>>>
>>>>>>>> 3) Start management server.
>>>>>>>>
>>>>>>>>
>>>>>>>> When deployed from rpm:
>>>>>>>>
>>>>>>>> 1) you deploy the code
>>>>>>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>>>>>>> base version of the DB is deployed. Thats why you see only 4.0.0
>>>>>>>> record in the DB.
>>>>>>>> 3) Start management server. DataBaseUpgradeChecker is being invoked
>>>>>>>> as a part of it, and performs the db upgrade to the version of the
>>>>>>>> current code. Only after that all the managers get invoked + system
>>>>>>>> caller context get initialized.
>>>>>>>>
>>>>>>>> Looks like the load order for step 3) got broken recently, and
>>>>>>>> system context gets initialized before the db upgrade is finished.
>>>>>>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker
>>>>>>>> as a part of cloudstack-setup-databases so at the point when
>>>>>>>> management server starts up, the DB already upgraded to the latest
>>>>>>>>version.
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>>>>>> <fg...@cloudops.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should be
>>>>>>>>> 4.3.0 no?
>>>>>>>>>
>>>>>>>>> mysql> select * from version;
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> | id | version | updated             | step     |
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>
>>>>>>>>> I installed cloudstack-management-4.3.0:
>>>>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>>>>>>> /usr/bin/cloudstack-setup-databases
>>>>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>>>>>
>>>>>>>>> Francois
>>>>>>>>>
>>>>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>>>>>> It's a fresh master RPM install.
>>>>>>>>>>
>>>>>>>>>> Francois
>>>>>>>>>>
>>>>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>>>>>> It is not a small issue. is_default filed was added to the table
>>>>>>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to
>>>>>>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>>>>>>
>>>>>>>>>>> DB upgrade is a major part of system integrity check; no queries
>>>>>>>>>>> to the DB should be made before its completed. Francois, did you
>>>>>>>>>>> start seeing this problem just recently?
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>>>>>> <fg...@cloudops.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>>>>>> One field is missing in the account table (field default). CS
>>>>>>>>>>>> will not start because of that.
>>>>>>>>>>>>
>>>>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to
>>>>>>>>>>>> register the system call context.
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception
>>>>>>>>>>>>on:
>>>>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT
>>>>>>>>>>>> account.id, account.account_name, account.type,
>>>>>>>>>>>> account.domain_id, account.state, account.removed,
>>>>>>>>>>>> account.cleanup_needed, account.network_domain, account.uuid,
>>>>>>>>>>>> account.default_zone_id, account.default FROM account WHERE
>>>>>>>>>>>>account.id = 1  AND account.removed IS NULL
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>>>6)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963
>>>>>>>>>>>>)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:92
>>>>>>>>>>>>6)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>>> tercept
>>>>>>>>>>>> orD
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:4
>>>>>>>>>>>>5)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>>>>>>> ava:166
>>>>>>>>>>>> )
>>>>>>>>>>>>
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>>>>>>> extOnce
>>>>>>>>>>>> Onl
>>>>>>>>>>>>
>>>>>>>>>>>> y(CallContext.java:141)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>>>>>>> (CallCo
>>>>>>>>>>>> nte
>>>>>>>>>>>>
>>>>>>>>>>>> xtListener.java:36)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>>>> .callWi
>>>>>>>>>>>> thC
>>>>>>>>>>>>
>>>>>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>>>> .runWit
>>>>>>>>>>>> hCo
>>>>>>>>>>>>
>>>>>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>>>>>>> (Manage
>>>>>>>>>>>> dCo
>>>>>>>>>>>>
>>>>>>>>>>>> ntextRunnable.java:46)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>>>>>>> n(Manag
>>>>>>>>>>>> edC
>>>>>>>>>>>>
>>>>>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>>>>>        at
>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>>>>>>> tructor
>>>>>>>>>>>> Acc
>>>>>>>>>>>>
>>>>>>>>>>>> essorImpl.java:57)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>>>>>>> tingCon
>>>>>>>>>>>> str
>>>>>>>>>>>>
>>>>>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>>>>>        at
>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>>>>>        at
>>>>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>>>>>        at
>>>>>>>>>>>>com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>>>>>        at
>>>>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>>>>>>> nt.java
>>>>>>>>>>>> :21
>>>>>>>>>>>>
>>>>>>>>>>>> 55)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>>>>>> java:23
>>>>>>>>>>>> 18)
>>>>>>>>>>>>
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>>>> (Delega
>>>>>>>>>>>> tin
>>>>>>>>>>>>
>>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>>>> (Delega
>>>>>>>>>>>> tin
>>>>>>>>>>>>
>>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>>        at
>>>>>>>>>>>>
>>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>>>3)
>>>>>>>>>>>>        ... 27 more
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Francois Gaudreault
>>>>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>>>>> 514-629-6775
>>>>>>>>>>>> - - -
>>>>>>>>>>>> CloudOps
>>>>>>>>>>>> 420 rue Guy
>>>>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>>>>> www.cloudops.com
>>>>>>>>>>>> @CloudOps_
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Francois Gaudreault
>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>> 514-629-6775
>>>>>>>>> - - -
>>>>>>>>> CloudOps
>>>>>>>>> 420 rue Guy
>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>> www.cloudops.com
>>>>>>>>> @CloudOps_
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Francois Gaudreault
>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>> fgaudreault@cloudops.com
>>>>>>> 514-629-6775
>>>>>>> - - -
>>>>>>> CloudOps
>>>>>>> 420 rue Guy
>>>>>>> Montréal QC  H3J 1S6
>>>>>>> www.cloudops.com
>>>>>>> @CloudOps_
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

Kelven,

So the issue is the combination of my code with VmwareContextPool.
With the ManagedContext framework, what I've done is replace every
Runnable and TimerTask with ManagedContextRunnable and
ManagedContextTimerTask.  Those classes will run the onEnter, onLeave
logic which will setup CallContext as a result.  I purposely changed
every reference so that everything was consistent.  I didn't want to
have developers have to consider when or when not to use
ManagedContext, so just always use it.  So as a result even though
your code has nothing to do with the DB, the ManageContextListener for
CallContext does.

So I'm sure your thinking that Resources shouldn't call the DB.  The
ManagedContext framework only does things when deployed in a managed
JVM.  The only managed JVM is the mgmt server.  If it's AWSAPI, Usage,
or an Agent JVM, then that framework does nothing.

So there are three possible solutions I see

1) Change VmwareContextPool to be initialized from the @PostConstruct
or start() method.
2) Revert the change to VmwareContextPool to use TimerTask and not
ManageContextTimerTask
3) Merge spring modularization.

The simplest stop gap would be option 2.

Darren

On Tue, Oct 8, 2013 at 5:35 PM, Kelven Yang <ke...@citrix.com> wrote:
> The problem seems to me is whether or not a background job that touches
> with database respects the bootstrap initialization order. As of
> VmwareContextPool itself, its background job does something fully within
> its own territory (no database, no reference outside). and vmware-base
> package was originally designed to be running on its own without assuming
> any container that offers unified lifecycle management. I don't think this
> type of background job has anything to do with the failure in this
> particular case.
>
> However, I do agree that we need to clean up and unify a few things inside
> the CloudStack, especially on life-cycle management and all
> background-jobs that their execution path touches with component
> life-cyle, auto-wiring, AOP etc.
>
> To live with the time before the spring modularization merge, we just need
> to figure out which background job that triggers all these and get it
> fixed, it used to work before even it is fragile, I don't think the fix of
> the problem is impossible. Is anyone working on this issue?
>
> Kelven
>
>
>
>
> On 10/8/13 2:35 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>
>>Some more info about this.  What specifically is happening is that the
>>VmwareContextPool call is creating a Timer during the constructor of
>>the class which is being constructed in a static block from
>>VmwareContextFactory.  So when the VmwareContextFactory class is
>>loaded by the class loader, the background thread is created.  Which
>>is way, way before the Database upgrade happens.  This will still be
>>fixed if we merge the spring modularization, but this vmware code
>>should change regardless.  Background threads should only be launched
>>from a @PostConstruct or ComponentLifecycle.start() method.  They
>>should not be started when a class is constructed or loaded.
>>
>>Darren
>>
>>
>>On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd
>><da...@gmail.com> wrote:
>>> Hey, I half way introduced this issue in a really long and round about
>>> way.  I don't think there's a good simple fix unless we merge the
>>> spring-modularization branch.  I'm going to look further into it.  But
>>> here's the background of why we are seeing this.
>>>
>>> I introduced "Managed Context" framework that will wrap all the
>>> background threads and manage the thread locals.  This was the union
>>> of CallContext, ServerContext, and AsyncJob*Context into one simple
>>> framework.  The problem with ACS though is that A LOT of background
>>> threads are spawned at all different random times of the
>>> initialization.  So what is happening is that during the
>>> initialization of some bean its kicking off a background thread that
>>> tries to access the database before the database upgrade has ran.  Now
>>> the CallContext has a strange suicidal behaviour (this was already
>>> there, I didn't change this), if it can't find account 1, it does a
>>> System.exit(1).  So since this one background thread is failing, the
>>> whole JVM shuts down.  Before CallContext only existed on some
>>> threads, but the addition of the Managed Context framework, it is now
>>> on almost all threads.
>>>
>>> Now in the spring-modularization branch there is a very strict and
>>> (mostly) deterministic initialization order.  The database upgrade
>>> class will be initialized and ran before any other bean in CloudStack
>>> is even initiated.  So this works around all these DB problems.  The
>>> current spring setup in master is very, very fragile.  As I said
>>> before, it is really difficult to ensure certain aspects are
>>> initialized before others, and since we moved to (which I don't really
>>> agree with) doing DB schema upgrades purely on startup of the mgmt
>>> server, we now have to be extra careful about initialization order.
>>>
>>> Darren
>>>
>>> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan
>>> <ra...@citrix.com> wrote:
>>>> Here the defect created for this issue
>>>>
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>>>>
>>>>
>>>> Regards,
>>>> Rayees
>>>>
>>>> -----Original Message-----
>>>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>>>> Sent: Tuesday, October 08, 2013 11:15 AM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: Latest Master DB issue
>>>>
>>>> I guess in my case, it's fine. It was a fresh install...
>>>>
>>>> Francois
>>>>
>>>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>>>>> Deploy db from maven will drop all the tables.  Not sure if this is
>>>>> fresh install or not.  For master, running mvn will be your best bet.
>>>>> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
>>>>> manually if your adventurous.
>>>>>
>>>>> Darren
>>>>>
>>>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
>>>>> <fg...@cloudops.com> wrote:
>>>>>> Thanks Alena for the explaination.
>>>>>>
>>>>>> What is the better path to fix this on our setup? Should I wait for a
>>>>>> fix in master or should I manually run the deploydb with mvn? I guess
>>>>>> the second option won't work since I used RPM?
>>>>>>
>>>>>> Francois
>>>>>>
>>>>>>
>>>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>>>>> Ok, this is what going on - the DB upgrade procedure is different on
>>>>>>> developer's setup and when deployed using cloudstack-setup-databases
>>>>>>>
>>>>>>> On developers setup:
>>>>>>>
>>>>>>> 1) you deploy the code
>>>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>>>>
>>>>>>> * first deploys the base DB version - 4.0.0
>>>>>>> * then checks the current version of the code, and performs the db
>>>>>>> upgrade if needed. So on master, version table looks like this after
>>>>>>> the db is
>>>>>>> deployed:
>>>>>>>
>>>>>>> mysql> select * from version;
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> | id | version | updated             | step     |
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> 4 rows in set (0.00 sec)
>>>>>>>
>>>>>>>
>>>>>>> 3) Start management server.
>>>>>>>
>>>>>>>
>>>>>>> When deployed from rpm:
>>>>>>>
>>>>>>> 1) you deploy the code
>>>>>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>>>>>> base version of the DB is deployed. Thats why you see only 4.0.0
>>>>>>> record in the DB.
>>>>>>> 3) Start management server. DataBaseUpgradeChecker is being invoked
>>>>>>> as a part of it, and performs the db upgrade to the version of the
>>>>>>> current code. Only after that all the managers get invoked + system
>>>>>>> caller context get initialized.
>>>>>>>
>>>>>>> Looks like the load order for step 3) got broken recently, and
>>>>>>> system context gets initialized before the db upgrade is finished.
>>>>>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker
>>>>>>> as a part of cloudstack-setup-databases so at the point when
>>>>>>> management server starts up, the DB already upgraded to the latest
>>>>>>>version.
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>>
>>>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>>>>> <fg...@cloudops.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should be
>>>>>>>> 4.3.0 no?
>>>>>>>>
>>>>>>>> mysql> select * from version;
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> | id | version | updated             | step     |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>>>>> +----+---------+---------------------+----------+
>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>
>>>>>>>> I installed cloudstack-management-4.3.0:
>>>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>>>>>> /usr/bin/cloudstack-setup-databases
>>>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>>>>
>>>>>>>> Francois
>>>>>>>>
>>>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>>>>> It's a fresh master RPM install.
>>>>>>>>>
>>>>>>>>> Francois
>>>>>>>>>
>>>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>>>>> It is not a small issue. is_default filed was added to the table
>>>>>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to
>>>>>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>>>>>
>>>>>>>>>> DB upgrade is a major part of system integrity check; no queries
>>>>>>>>>> to the DB should be made before its completed. Francois, did you
>>>>>>>>>> start seeing this problem just recently?
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>>>>> <fg...@cloudops.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>>>>> One field is missing in the account table (field default). CS
>>>>>>>>>>> will not start because of that.
>>>>>>>>>>>
>>>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to
>>>>>>>>>>> register the system call context.
>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception
>>>>>>>>>>>on:
>>>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT
>>>>>>>>>>> account.id, account.account_name, account.type,
>>>>>>>>>>> account.domain_id, account.state, account.removed,
>>>>>>>>>>> account.cleanup_needed, account.network_domain, account.uuid,
>>>>>>>>>>> account.default_zone_id, account.default FROM account WHERE
>>>>>>>>>>>account.id = 1  AND account.removed IS NULL
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>>6)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>> tercept
>>>>>>>>>>> orD
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963
>>>>>>>>>>>)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>> tercept
>>>>>>>>>>> orD
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:92
>>>>>>>>>>>6)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>>> tercept
>>>>>>>>>>> orD
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:4
>>>>>>>>>>>5)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>>>>>> ava:166
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>>>>>> extOnce
>>>>>>>>>>> Onl
>>>>>>>>>>>
>>>>>>>>>>> y(CallContext.java:141)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>>>>>> (CallCo
>>>>>>>>>>> nte
>>>>>>>>>>>
>>>>>>>>>>> xtListener.java:36)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>>> .callWi
>>>>>>>>>>> thC
>>>>>>>>>>>
>>>>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>>> .runWit
>>>>>>>>>>> hCo
>>>>>>>>>>>
>>>>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>>>>>> (Manage
>>>>>>>>>>> dCo
>>>>>>>>>>>
>>>>>>>>>>> ntextRunnable.java:46)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>>>>>> n(Manag
>>>>>>>>>>> edC
>>>>>>>>>>>
>>>>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>>>>> Caused by:
>>>>>>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>>>>        at
>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>>> Method)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>>>>>> tructor
>>>>>>>>>>> Acc
>>>>>>>>>>>
>>>>>>>>>>> essorImpl.java:57)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>>>>>> tingCon
>>>>>>>>>>> str
>>>>>>>>>>>
>>>>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>>>>        at
>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>>>>        at
>>>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>>>>        at
>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>>>>        at
>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>>>>        at
>>>>>>>>>>>com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>>>>        at
>>>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>>>>>> nt.java
>>>>>>>>>>> :21
>>>>>>>>>>>
>>>>>>>>>>> 55)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>>>>> java:23
>>>>>>>>>>> 18)
>>>>>>>>>>>
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>>> (Delega
>>>>>>>>>>> tin
>>>>>>>>>>>
>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>>> (Delega
>>>>>>>>>>> tin
>>>>>>>>>>>
>>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>>        at
>>>>>>>>>>>
>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>>3)
>>>>>>>>>>>        ... 27 more
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Francois Gaudreault
>>>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>>>> 514-629-6775
>>>>>>>>>>> - - -
>>>>>>>>>>> CloudOps
>>>>>>>>>>> 420 rue Guy
>>>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>>>> www.cloudops.com
>>>>>>>>>>> @CloudOps_
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Francois Gaudreault
>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>> fgaudreault@cloudops.com
>>>>>>>> 514-629-6775
>>>>>>>> - - -
>>>>>>>> CloudOps
>>>>>>>> 420 rue Guy
>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>> www.cloudops.com
>>>>>>>> @CloudOps_
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Francois Gaudreault
>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>> fgaudreault@cloudops.com
>>>>>> 514-629-6775
>>>>>> - - -
>>>>>> CloudOps
>>>>>> 420 rue Guy
>>>>>> Montréal QC  H3J 1S6
>>>>>> www.cloudops.com
>>>>>> @CloudOps_
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>

Re: Latest Master DB issue

Posted by Kelven Yang <ke...@citrix.com>.

The problem seems to me is whether or not a background job that touches
with database respects the bootstrap initialization order. As of
VmwareContextPool itself, its background job does something fully within
its own territory (no database, no reference outside). and vmware-base
package was originally designed to be running on its own without assuming
any container that offers unified lifecycle management. I don't think this
type of background job has anything to do with the failure in this
particular case.

However, I do agree that we need to clean up and unify a few things inside
the CloudStack, especially on life-cycle management and all
background-jobs that their execution path touches with component
life-cyle, auto-wiring, AOP etc.

To live with the time before the spring modularization merge, we just need
to figure out which background job that triggers all these and get it
fixed, it used to work before even it is fragile, I don't think the fix of
the problem is impossible. Is anyone working on this issue?

Kelven




On 10/8/13 2:35 PM, "Darren Shepherd" <da...@gmail.com> wrote:

>Some more info about this.  What specifically is happening is that the
>VmwareContextPool call is creating a Timer during the constructor of
>the class which is being constructed in a static block from
>VmwareContextFactory.  So when the VmwareContextFactory class is
>loaded by the class loader, the background thread is created.  Which
>is way, way before the Database upgrade happens.  This will still be
>fixed if we merge the spring modularization, but this vmware code
>should change regardless.  Background threads should only be launched
>from a @PostConstruct or ComponentLifecycle.start() method.  They
>should not be started when a class is constructed or loaded.
>
>Darren
>
>
>On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd
><da...@gmail.com> wrote:
>> Hey, I half way introduced this issue in a really long and round about
>> way.  I don't think there's a good simple fix unless we merge the
>> spring-modularization branch.  I'm going to look further into it.  But
>> here's the background of why we are seeing this.
>>
>> I introduced "Managed Context" framework that will wrap all the
>> background threads and manage the thread locals.  This was the union
>> of CallContext, ServerContext, and AsyncJob*Context into one simple
>> framework.  The problem with ACS though is that A LOT of background
>> threads are spawned at all different random times of the
>> initialization.  So what is happening is that during the
>> initialization of some bean its kicking off a background thread that
>> tries to access the database before the database upgrade has ran.  Now
>> the CallContext has a strange suicidal behaviour (this was already
>> there, I didn't change this), if it can't find account 1, it does a
>> System.exit(1).  So since this one background thread is failing, the
>> whole JVM shuts down.  Before CallContext only existed on some
>> threads, but the addition of the Managed Context framework, it is now
>> on almost all threads.
>>
>> Now in the spring-modularization branch there is a very strict and
>> (mostly) deterministic initialization order.  The database upgrade
>> class will be initialized and ran before any other bean in CloudStack
>> is even initiated.  So this works around all these DB problems.  The
>> current spring setup in master is very, very fragile.  As I said
>> before, it is really difficult to ensure certain aspects are
>> initialized before others, and since we moved to (which I don't really
>> agree with) doing DB schema upgrades purely on startup of the mgmt
>> server, we now have to be extra careful about initialization order.
>>
>> Darren
>>
>> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan
>> <ra...@citrix.com> wrote:
>>> Here the defect created for this issue
>>>
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>>>
>>>
>>> Regards,
>>> Rayees
>>>
>>> -----Original Message-----
>>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>>> Sent: Tuesday, October 08, 2013 11:15 AM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: Latest Master DB issue
>>>
>>> I guess in my case, it's fine. It was a fresh install...
>>>
>>> Francois
>>>
>>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>>>> Deploy db from maven will drop all the tables.  Not sure if this is
>>>> fresh install or not.  For master, running mvn will be your best bet.
>>>> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
>>>> manually if your adventurous.
>>>>
>>>> Darren
>>>>
>>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
>>>> <fg...@cloudops.com> wrote:
>>>>> Thanks Alena for the explaination.
>>>>>
>>>>> What is the better path to fix this on our setup? Should I wait for a
>>>>> fix in master or should I manually run the deploydb with mvn? I guess
>>>>> the second option won't work since I used RPM?
>>>>>
>>>>> Francois
>>>>>
>>>>>
>>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>>>> Ok, this is what going on - the DB upgrade procedure is different on
>>>>>> developer's setup and when deployed using cloudstack-setup-databases
>>>>>>
>>>>>> On developers setup:
>>>>>>
>>>>>> 1) you deploy the code
>>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>>>
>>>>>> * first deploys the base DB version - 4.0.0
>>>>>> * then checks the current version of the code, and performs the db
>>>>>> upgrade if needed. So on master, version table looks like this after
>>>>>> the db is
>>>>>> deployed:
>>>>>>
>>>>>> mysql> select * from version;
>>>>>> +----+---------+---------------------+----------+
>>>>>> | id | version | updated             | step     |
>>>>>> +----+---------+---------------------+----------+
>>>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>>>> +----+---------+---------------------+----------+
>>>>>> 4 rows in set (0.00 sec)
>>>>>>
>>>>>>
>>>>>> 3) Start management server.
>>>>>>
>>>>>>
>>>>>> When deployed from rpm:
>>>>>>
>>>>>> 1) you deploy the code
>>>>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>>>>> base version of the DB is deployed. Thats why you see only 4.0.0
>>>>>> record in the DB.
>>>>>> 3) Start management server. DataBaseUpgradeChecker is being invoked
>>>>>> as a part of it, and performs the db upgrade to the version of the
>>>>>> current code. Only after that all the managers get invoked + system
>>>>>> caller context get initialized.
>>>>>>
>>>>>> Looks like the load order for step 3) got broken recently, and
>>>>>> system context gets initialized before the db upgrade is finished.
>>>>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker
>>>>>> as a part of cloudstack-setup-databases so at the point when
>>>>>> management server starts up, the DB already upgraded to the latest
>>>>>>version.
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>>
>>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>>>> <fg...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should be
>>>>>>> 4.3.0 no?
>>>>>>>
>>>>>>> mysql> select * from version;
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> | id | version | updated             | step     |
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>>>> +----+---------+---------------------+----------+
>>>>>>> 1 row in set (0.00 sec)
>>>>>>>
>>>>>>> I installed cloudstack-management-4.3.0:
>>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>>>>> /usr/bin/cloudstack-setup-databases
>>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>>>
>>>>>>> Francois
>>>>>>>
>>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>>>> It's a fresh master RPM install.
>>>>>>>>
>>>>>>>> Francois
>>>>>>>>
>>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>>>> It is not a small issue. is_default filed was added to the table
>>>>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to
>>>>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>>>>
>>>>>>>>> DB upgrade is a major part of system integrity check; no queries
>>>>>>>>> to the DB should be made before its completed. Francois, did you
>>>>>>>>> start seeing this problem just recently?
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>>>> <fg...@cloudops.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>>>> One field is missing in the account table (field default). CS
>>>>>>>>>> will not start because of that.
>>>>>>>>>>
>>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to
>>>>>>>>>> register the system call context.
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception
>>>>>>>>>>on:
>>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT
>>>>>>>>>> account.id, account.account_name, account.type,
>>>>>>>>>> account.domain_id, account.state, account.removed,
>>>>>>>>>> account.cleanup_needed, account.network_domain, account.uuid,
>>>>>>>>>> account.default_zone_id, account.default FROM account WHERE
>>>>>>>>>>account.id = 1  AND account.removed IS NULL
>>>>>>>>>>        at
>>>>>>>>>> 
>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>6)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>> tercept
>>>>>>>>>> orD
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>        at
>>>>>>>>>> 
>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963
>>>>>>>>>>)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>> tercept
>>>>>>>>>> orD
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>        at
>>>>>>>>>> 
>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:92
>>>>>>>>>>6)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>>> tercept
>>>>>>>>>> orD
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>        at
>>>>>>>>>> 
>>>>>>>>>>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:4
>>>>>>>>>>5)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>>>>> ava:166
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>>>>> extOnce
>>>>>>>>>> Onl
>>>>>>>>>>
>>>>>>>>>> y(CallContext.java:141)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>>>>> (CallCo
>>>>>>>>>> nte
>>>>>>>>>>
>>>>>>>>>> xtListener.java:36)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>> .callWi
>>>>>>>>>> thC
>>>>>>>>>>
>>>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>>> .runWit
>>>>>>>>>> hCo
>>>>>>>>>>
>>>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>>>>> (Manage
>>>>>>>>>> dCo
>>>>>>>>>>
>>>>>>>>>> ntextRunnable.java:46)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>>>>> n(Manag
>>>>>>>>>> edC
>>>>>>>>>>
>>>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>>>> Caused by:
>>>>>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>>>        at
>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>> Method)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>>>>> tructor
>>>>>>>>>> Acc
>>>>>>>>>>
>>>>>>>>>> essorImpl.java:57)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>>>>> tingCon
>>>>>>>>>> str
>>>>>>>>>>
>>>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>>>        at
>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>>>        at
>>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>>>        at
>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>>>        at
>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>>>        at
>>>>>>>>>>com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>>>        at
>>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>>>>> nt.java
>>>>>>>>>> :21
>>>>>>>>>>
>>>>>>>>>> 55)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>>>> java:23
>>>>>>>>>> 18)
>>>>>>>>>>
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>> (Delega
>>>>>>>>>> tin
>>>>>>>>>>
>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>        at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>>> (Delega
>>>>>>>>>> tin
>>>>>>>>>>
>>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>>        at
>>>>>>>>>> 
>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:98
>>>>>>>>>>3)
>>>>>>>>>>        ... 27 more
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Francois Gaudreault
>>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>>> 514-629-6775
>>>>>>>>>> - - -
>>>>>>>>>> CloudOps
>>>>>>>>>> 420 rue Guy
>>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>>> www.cloudops.com
>>>>>>>>>> @CloudOps_
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>>> Francois Gaudreault
>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>> fgaudreault@cloudops.com
>>>>>>> 514-629-6775
>>>>>>> - - -
>>>>>>> CloudOps
>>>>>>> 420 rue Guy
>>>>>>> Montréal QC  H3J 1S6
>>>>>>> www.cloudops.com
>>>>>>> @CloudOps_
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>> fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>>>
>>>
>>>
>>> --
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>fgaudreault@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

Some more info about this.  What specifically is happening is that the
VmwareContextPool call is creating a Timer during the constructor of
the class which is being constructed in a static block from
VmwareContextFactory.  So when the VmwareContextFactory class is
loaded by the class loader, the background thread is created.  Which
is way, way before the Database upgrade happens.  This will still be
fixed if we merge the spring modularization, but this vmware code
should change regardless.  Background threads should only be launched
from a @PostConstruct or ComponentLifecycle.start() method.  They
should not be started when a class is constructed or loaded.

Darren


On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd
<da...@gmail.com> wrote:
> Hey, I half way introduced this issue in a really long and round about
> way.  I don't think there's a good simple fix unless we merge the
> spring-modularization branch.  I'm going to look further into it.  But
> here's the background of why we are seeing this.
>
> I introduced "Managed Context" framework that will wrap all the
> background threads and manage the thread locals.  This was the union
> of CallContext, ServerContext, and AsyncJob*Context into one simple
> framework.  The problem with ACS though is that A LOT of background
> threads are spawned at all different random times of the
> initialization.  So what is happening is that during the
> initialization of some bean its kicking off a background thread that
> tries to access the database before the database upgrade has ran.  Now
> the CallContext has a strange suicidal behaviour (this was already
> there, I didn't change this), if it can't find account 1, it does a
> System.exit(1).  So since this one background thread is failing, the
> whole JVM shuts down.  Before CallContext only existed on some
> threads, but the addition of the Managed Context framework, it is now
> on almost all threads.
>
> Now in the spring-modularization branch there is a very strict and
> (mostly) deterministic initialization order.  The database upgrade
> class will be initialized and ran before any other bean in CloudStack
> is even initiated.  So this works around all these DB problems.  The
> current spring setup in master is very, very fragile.  As I said
> before, it is really difficult to ensure certain aspects are
> initialized before others, and since we moved to (which I don't really
> agree with) doing DB schema upgrades purely on startup of the mgmt
> server, we now have to be extra careful about initialization order.
>
> Darren
>
> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan
> <ra...@citrix.com> wrote:
>> Here the defect created for this issue
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>>
>>
>> Regards,
>> Rayees
>>
>> -----Original Message-----
>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>> Sent: Tuesday, October 08, 2013 11:15 AM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Latest Master DB issue
>>
>> I guess in my case, it's fine. It was a fresh install...
>>
>> Francois
>>
>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>>> Deploy db from maven will drop all the tables.  Not sure if this is
>>> fresh install or not.  For master, running mvn will be your best bet.
>>> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
>>> manually if your adventurous.
>>>
>>> Darren
>>>
>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
>>> <fg...@cloudops.com> wrote:
>>>> Thanks Alena for the explaination.
>>>>
>>>> What is the better path to fix this on our setup? Should I wait for a
>>>> fix in master or should I manually run the deploydb with mvn? I guess
>>>> the second option won't work since I used RPM?
>>>>
>>>> Francois
>>>>
>>>>
>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>>> Ok, this is what going on - the DB upgrade procedure is different on
>>>>> developer's setup and when deployed using cloudstack-setup-databases
>>>>>
>>>>> On developers setup:
>>>>>
>>>>> 1) you deploy the code
>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>>
>>>>> * first deploys the base DB version - 4.0.0
>>>>> * then checks the current version of the code, and performs the db
>>>>> upgrade if needed. So on master, version table looks like this after
>>>>> the db is
>>>>> deployed:
>>>>>
>>>>> mysql> select * from version;
>>>>> +----+---------+---------------------+----------+
>>>>> | id | version | updated             | step     |
>>>>> +----+---------+---------------------+----------+
>>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>>> +----+---------+---------------------+----------+
>>>>> 4 rows in set (0.00 sec)
>>>>>
>>>>>
>>>>> 3) Start management server.
>>>>>
>>>>>
>>>>> When deployed from rpm:
>>>>>
>>>>> 1) you deploy the code
>>>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>>>> base version of the DB is deployed. Thats why you see only 4.0.0
>>>>> record in the DB.
>>>>> 3) Start management server. DataBaseUpgradeChecker is being invoked
>>>>> as a part of it, and performs the db upgrade to the version of the
>>>>> current code. Only after that all the managers get invoked + system
>>>>> caller context get initialized.
>>>>>
>>>>> Looks like the load order for step 3) got broken recently, and
>>>>> system context gets initialized before the db upgrade is finished.
>>>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker
>>>>> as a part of cloudstack-setup-databases so at the point when
>>>>> management server starts up, the DB already upgraded to the latest version.
>>>>>
>>>>> -Alena.
>>>>>
>>>>>
>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>>> <fg...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should be
>>>>>> 4.3.0 no?
>>>>>>
>>>>>> mysql> select * from version;
>>>>>> +----+---------+---------------------+----------+
>>>>>> | id | version | updated             | step     |
>>>>>> +----+---------+---------------------+----------+
>>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>>> +----+---------+---------------------+----------+
>>>>>> 1 row in set (0.00 sec)
>>>>>>
>>>>>> I installed cloudstack-management-4.3.0:
>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>>>> /usr/bin/cloudstack-setup-databases
>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>>
>>>>>> Francois
>>>>>>
>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>>> It's a fresh master RPM install.
>>>>>>>
>>>>>>> Francois
>>>>>>>
>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>>> It is not a small issue. is_default filed was added to the table
>>>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to
>>>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>>>
>>>>>>>> DB upgrade is a major part of system integrity check; no queries
>>>>>>>> to the DB should be made before its completed. Francois, did you
>>>>>>>> start seeing this problem just recently?
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>>> <fg...@cloudops.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>>> One field is missing in the account table (field default). CS
>>>>>>>>> will not start because of that.
>>>>>>>>>
>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to
>>>>>>>>> register the system call context.
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT
>>>>>>>>> account.id, account.account_name, account.type,
>>>>>>>>> account.domain_id, account.state, account.removed,
>>>>>>>>> account.cleanup_needed, account.network_domain, account.uuid,
>>>>>>>>> account.default_zone_id, account.default FROM account WHERE account.id = 1  AND account.removed IS NULL
>>>>>>>>>        at
>>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>> tercept
>>>>>>>>> orD
>>>>>>>>>
>>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>        at
>>>>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>> tercept
>>>>>>>>> orD
>>>>>>>>>
>>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>        at
>>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>>> tercept
>>>>>>>>> orD
>>>>>>>>>
>>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>        at
>>>>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>>>> ava:166
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>>>> extOnce
>>>>>>>>> Onl
>>>>>>>>>
>>>>>>>>> y(CallContext.java:141)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>>>> (CallCo
>>>>>>>>> nte
>>>>>>>>>
>>>>>>>>> xtListener.java:36)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>> .callWi
>>>>>>>>> thC
>>>>>>>>>
>>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>>> .runWit
>>>>>>>>> hCo
>>>>>>>>>
>>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>>>> (Manage
>>>>>>>>> dCo
>>>>>>>>>
>>>>>>>>> ntextRunnable.java:46)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>>>> n(Manag
>>>>>>>>> edC
>>>>>>>>>
>>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>>        at
>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>> Method)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>>>> tructor
>>>>>>>>> Acc
>>>>>>>>>
>>>>>>>>> essorImpl.java:57)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>>>> tingCon
>>>>>>>>> str
>>>>>>>>>
>>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>>        at
>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>>        at
>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>>        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>>        at
>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>>>> nt.java
>>>>>>>>> :21
>>>>>>>>>
>>>>>>>>> 55)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>>> java:23
>>>>>>>>> 18)
>>>>>>>>>
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>> (Delega
>>>>>>>>> tin
>>>>>>>>>
>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>        at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>>> (Delega
>>>>>>>>> tin
>>>>>>>>>
>>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>>        at
>>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>>>>        ... 27 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Francois Gaudreault
>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>>> fgaudreault@cloudops.com
>>>>>>>>> 514-629-6775
>>>>>>>>> - - -
>>>>>>>>> CloudOps
>>>>>>>>> 420 rue Guy
>>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>>> www.cloudops.com
>>>>>>>>> @CloudOps_
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> Francois Gaudreault
>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>> fgaudreault@cloudops.com
>>>>>> 514-629-6775
>>>>>> - - -
>>>>>> CloudOps
>>>>>> 420 rue Guy
>>>>>> Montréal QC  H3J 1S6
>>>>>> www.cloudops.com
>>>>>> @CloudOps_
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>> fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>>>
>>
>>
>> --
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

Hey, I half way introduced this issue in a really long and round about
way.  I don't think there's a good simple fix unless we merge the
spring-modularization branch.  I'm going to look further into it.  But
here's the background of why we are seeing this.

I introduced "Managed Context" framework that will wrap all the
background threads and manage the thread locals.  This was the union
of CallContext, ServerContext, and AsyncJob*Context into one simple
framework.  The problem with ACS though is that A LOT of background
threads are spawned at all different random times of the
initialization.  So what is happening is that during the
initialization of some bean its kicking off a background thread that
tries to access the database before the database upgrade has ran.  Now
the CallContext has a strange suicidal behaviour (this was already
there, I didn't change this), if it can't find account 1, it does a
System.exit(1).  So since this one background thread is failing, the
whole JVM shuts down.  Before CallContext only existed on some
threads, but the addition of the Managed Context framework, it is now
on almost all threads.

Now in the spring-modularization branch there is a very strict and
(mostly) deterministic initialization order.  The database upgrade
class will be initialized and ran before any other bean in CloudStack
is even initiated.  So this works around all these DB problems.  The
current spring setup in master is very, very fragile.  As I said
before, it is really difficult to ensure certain aspects are
initialized before others, and since we moved to (which I don't really
agree with) doing DB schema upgrades purely on startup of the mgmt
server, we now have to be extra careful about initialization order.

Darren

On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan
<ra...@citrix.com> wrote:
> Here the defect created for this issue
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-4825
>
>
> Regards,
> Rayees
>
> -----Original Message-----
> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
> Sent: Tuesday, October 08, 2013 11:15 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Latest Master DB issue
>
> I guess in my case, it's fine. It was a fresh install...
>
> Francois
>
> On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
>> Deploy db from maven will drop all the tables.  Not sure if this is
>> fresh install or not.  For master, running mvn will be your best bet.
>> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
>> manually if your adventurous.
>>
>> Darren
>>
>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
>> <fg...@cloudops.com> wrote:
>>> Thanks Alena for the explaination.
>>>
>>> What is the better path to fix this on our setup? Should I wait for a
>>> fix in master or should I manually run the deploydb with mvn? I guess
>>> the second option won't work since I used RPM?
>>>
>>> Francois
>>>
>>>
>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>>> Ok, this is what going on - the DB upgrade procedure is different on
>>>> developer's setup and when deployed using cloudstack-setup-databases
>>>>
>>>> On developers setup:
>>>>
>>>> 1) you deploy the code
>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'.
>>>> As a part of this step, the DataBaseUpgradeChecker:
>>>>
>>>> * first deploys the base DB version - 4.0.0
>>>> * then checks the current version of the code, and performs the db
>>>> upgrade if needed. So on master, version table looks like this after
>>>> the db is
>>>> deployed:
>>>>
>>>> mysql> select * from version;
>>>> +----+---------+---------------------+----------+
>>>> | id | version | updated             | step     |
>>>> +----+---------+---------------------+----------+
>>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>>> +----+---------+---------------------+----------+
>>>> 4 rows in set (0.00 sec)
>>>>
>>>>
>>>> 3) Start management server.
>>>>
>>>>
>>>> When deployed from rpm:
>>>>
>>>> 1) you deploy the code
>>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>>> base version of the DB is deployed. Thats why you see only 4.0.0
>>>> record in the DB.
>>>> 3) Start management server. DataBaseUpgradeChecker is being invoked
>>>> as a part of it, and performs the db upgrade to the version of the
>>>> current code. Only after that all the managers get invoked + system
>>>> caller context get initialized.
>>>>
>>>> Looks like the load order for step 3) got broken recently, and
>>>> system context gets initialized before the db upgrade is finished.
>>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker
>>>> as a part of cloudstack-setup-databases so at the point when
>>>> management server starts up, the DB already upgraded to the latest version.
>>>>
>>>> -Alena.
>>>>
>>>>
>>>> On 10/8/13 10:25 AM, "Francois Gaudreault"
>>>> <fg...@cloudops.com>
>>>> wrote:
>>>>
>>>>> Hmm... I just checked the DB version and it's 4.0??? It should be
>>>>> 4.3.0 no?
>>>>>
>>>>> mysql> select * from version;
>>>>> +----+---------+---------------------+----------+
>>>>> | id | version | updated             | step     |
>>>>> +----+---------+---------------------+----------+
>>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>>> +----+---------+---------------------+----------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>> I installed cloudstack-management-4.3.0:
>>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>>> /usr/bin/cloudstack-setup-databases
>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>>
>>>>> Francois
>>>>>
>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>>> It's a fresh master RPM install.
>>>>>>
>>>>>> Francois
>>>>>>
>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>>> It is not a small issue. is_default filed was added to the table
>>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to
>>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>>
>>>>>>> DB upgrade is a major part of system integrity check; no queries
>>>>>>> to the DB should be made before its completed. Francois, did you
>>>>>>> start seeing this problem just recently?
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault"
>>>>>>> <fg...@cloudops.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I compiled Master this morning, and there is a small DB issue.
>>>>>>>> One field is missing in the account table (field default). CS
>>>>>>>> will not start because of that.
>>>>>>>>
>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext]
>>>>>>>> (Timer-2:null) Exiting the system because we're unable to
>>>>>>>> register the system call context.
>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT
>>>>>>>> account.id, account.account_name, account.type,
>>>>>>>> account.domain_id, account.state, account.removed,
>>>>>>>> account.cleanup_needed, account.network_domain, account.uuid,
>>>>>>>> account.default_zone_id, account.default FROM account WHERE account.id = 1  AND account.removed IS NULL
>>>>>>>>        at
>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>> tercept
>>>>>>>> orD
>>>>>>>>
>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>        at
>>>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>> tercept
>>>>>>>> orD
>>>>>>>>
>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>        at
>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>>> tercept
>>>>>>>> orD
>>>>>>>>
>>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>        at
>>>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>>> ava:166
>>>>>>>> )
>>>>>>>>
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>>> extOnce
>>>>>>>> Onl
>>>>>>>>
>>>>>>>> y(CallContext.java:141)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>>> (CallCo
>>>>>>>> nte
>>>>>>>>
>>>>>>>> xtListener.java:36)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>> .callWi
>>>>>>>> thC
>>>>>>>>
>>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>>> .runWit
>>>>>>>> hCo
>>>>>>>>
>>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>>> (Manage
>>>>>>>> dCo
>>>>>>>>
>>>>>>>> ntextRunnable.java:46)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>>> n(Manag
>>>>>>>> edC
>>>>>>>>
>>>>>>>> ontextTimerTask.java:27)
>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>>        at
>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>> Method)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>>> tructor
>>>>>>>> Acc
>>>>>>>>
>>>>>>>> essorImpl.java:57)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>>> tingCon
>>>>>>>> str
>>>>>>>>
>>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>>        at
>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>>        at
>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>>        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>>        at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>>> nt.java
>>>>>>>> :21
>>>>>>>>
>>>>>>>> 55)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>>> java:23
>>>>>>>> 18)
>>>>>>>>
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>> (Delega
>>>>>>>> tin
>>>>>>>>
>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>>> (Delega
>>>>>>>> tin
>>>>>>>>
>>>>>>>> gPreparedStatement.java:96)
>>>>>>>>        at
>>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>>>        ... 27 more
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Francois Gaudreault
>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>>> fgaudreault@cloudops.com
>>>>>>>> 514-629-6775
>>>>>>>> - - -
>>>>>>>> CloudOps
>>>>>>>> 420 rue Guy
>>>>>>>> Montréal QC  H3J 1S6
>>>>>>>> www.cloudops.com
>>>>>>>> @CloudOps_
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>> --
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>> fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>> fgaudreault@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>
>>
>
>
> --
> Francois Gaudreault
> Architecte de Solution Cloud | Cloud Solutions Architect fgaudreault@cloudops.com
> 514-629-6775
> - - -
> CloudOps
> 420 rue Guy
> Montréal QC  H3J 1S6
> www.cloudops.com
> @CloudOps_
>

RE: Latest Master DB issue

Posted by Rayees Namathponnan <ra...@citrix.com>.

Here the defect created for this issue 

https://issues.apache.org/jira/browse/CLOUDSTACK-4825


Regards,
Rayees 

-----Original Message-----
From: Francois Gaudreault [mailto:fgaudreault@cloudops.com] 
Sent: Tuesday, October 08, 2013 11:15 AM
To: dev@cloudstack.apache.org
Subject: Re: Latest Master DB issue

I guess in my case, it's fine. It was a fresh install...

Francois

On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
> Deploy db from maven will drop all the tables.  Not sure if this is 
> fresh install or not.  For master, running mvn will be your best bet.
> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator 
> manually if your adventurous.
>
> Darren
>
> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault 
> <fg...@cloudops.com> wrote:
>> Thanks Alena for the explaination.
>>
>> What is the better path to fix this on our setup? Should I wait for a 
>> fix in master or should I manually run the deploydb with mvn? I guess 
>> the second option won't work since I used RPM?
>>
>> Francois
>>
>>
>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>> Ok, this is what going on - the DB upgrade procedure is different on 
>>> developer's setup and when deployed using cloudstack-setup-databases
>>>
>>> On developers setup:
>>>
>>> 1) you deploy the code
>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. 
>>> As a part of this step, the DataBaseUpgradeChecker:
>>>
>>> * first deploys the base DB version - 4.0.0
>>> * then checks the current version of the code, and performs the db 
>>> upgrade if needed. So on master, version table looks like this after 
>>> the db is
>>> deployed:
>>>
>>> mysql> select * from version;
>>> +----+---------+---------------------+----------+
>>> | id | version | updated             | step     |
>>> +----+---------+---------------------+----------+
>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>> +----+---------+---------------------+----------+
>>> 4 rows in set (0.00 sec)
>>>
>>>
>>> 3) Start management server.
>>>
>>>
>>> When deployed from rpm:
>>>
>>> 1) you deploy the code
>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0 
>>> base version of the DB is deployed. Thats why you see only 4.0.0 
>>> record in the DB.
>>> 3) Start management server. DataBaseUpgradeChecker is being invoked 
>>> as a part of it, and performs the db upgrade to the version of the 
>>> current code. Only after that all the managers get invoked + system 
>>> caller context get initialized.
>>>
>>> Looks like the load order for step 3) got broken recently, and 
>>> system context gets initialized before the db upgrade is finished. 
>>> So we either need to fix the order, or invoke DataBaseUpgradeChecker 
>>> as a part of cloudstack-setup-databases so at the point when 
>>> management server starts up, the DB already upgraded to the latest version.
>>>
>>> -Alena.
>>>
>>>
>>> On 10/8/13 10:25 AM, "Francois Gaudreault" 
>>> <fg...@cloudops.com>
>>> wrote:
>>>
>>>> Hmm... I just checked the DB version and it's 4.0??? It should be 
>>>> 4.3.0 no?
>>>>
>>>> mysql> select * from version;
>>>> +----+---------+---------------------+----------+
>>>> | id | version | updated             | step     |
>>>> +----+---------+---------------------+----------+
>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>> +----+---------+---------------------+----------+
>>>> 1 row in set (0.00 sec)
>>>>
>>>> I installed cloudstack-management-4.3.0:
>>>> [root@eng-testing-cstack_master ~]# rpm -qf 
>>>> /usr/bin/cloudstack-setup-databases
>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>
>>>> Francois
>>>>
>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>> It's a fresh master RPM install.
>>>>>
>>>>> Francois
>>>>>
>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>> It is not a small issue. is_default filed was added to the table 
>>>>>> as a part of the 41-42 db upgrade. Looks like the code tries to 
>>>>>> retrieve system user before the db upgrade is completed.
>>>>>>
>>>>>> DB upgrade is a major part of system integrity check; no queries 
>>>>>> to the DB should be made before its completed. Francois, did you 
>>>>>> start seeing this problem just recently?
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" 
>>>>>> <fg...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I compiled Master this morning, and there is a small DB issue. 
>>>>>>> One field is missing in the account table (field default). CS 
>>>>>>> will not start because of that.
>>>>>>>
>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] 
>>>>>>> (Timer-2:null) Exiting the system because we're unable to 
>>>>>>> register the system call context.
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT 
>>>>>>> account.id, account.account_name, account.type, 
>>>>>>> account.domain_id, account.state, account.removed, 
>>>>>>> account.cleanup_needed, account.network_domain, account.uuid, 
>>>>>>> account.default_zone_id, account.default FROM account WHERE account.id = 1  AND account.removed IS NULL
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>> tercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>> tercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$In
>>>>>>> tercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.j
>>>>>>> ava:166
>>>>>>> )
>>>>>>>
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallCont
>>>>>>> extOnce
>>>>>>> Onl
>>>>>>>
>>>>>>> y(CallContext.java:141)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext
>>>>>>> (CallCo
>>>>>>> nte
>>>>>>>
>>>>>>> xtListener.java:36)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>> .callWi
>>>>>>> thC
>>>>>>>
>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext
>>>>>>> .runWit
>>>>>>> hCo
>>>>>>>
>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run
>>>>>>> (Manage
>>>>>>> dCo
>>>>>>>
>>>>>>> ntextRunnable.java:46)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.ru
>>>>>>> n(Manag
>>>>>>> edC
>>>>>>>
>>>>>>> ontextTimerTask.java:27)
>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>        at 
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>> Method)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
>>>>>>> tructor
>>>>>>> Acc
>>>>>>>
>>>>>>> essorImpl.java:57)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
>>>>>>> tingCon
>>>>>>> str
>>>>>>>
>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>        at
>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStateme
>>>>>>> nt.java
>>>>>>> :21
>>>>>>>
>>>>>>> 55)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.
>>>>>>> java:23
>>>>>>> 18)
>>>>>>>
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>> (Delega
>>>>>>> tin
>>>>>>>
>>>>>>> gPreparedStatement.java:96)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery
>>>>>>> (Delega
>>>>>>> tin
>>>>>>>
>>>>>>> gPreparedStatement.java:96)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>>        ... 27 more
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Francois Gaudreault
>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>>>>> fgaudreault@cloudops.com
>>>>>>> 514-629-6775
>>>>>>> - - -
>>>>>>> CloudOps
>>>>>>> 420 rue Guy
>>>>>>> Montréal QC  H3J 1S6
>>>>>>> www.cloudops.com
>>>>>>> @CloudOps_
>>>>>>>
>>>>>>>
>>>>>>
>>>> --
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect 
>>>> fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>>>>
>>>
>>>
>>
>> --
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect 
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>
>


--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Francois Gaudreault <fg...@cloudops.com>.

I guess in my case, it's fine. It was a fresh install...

Francois

On 10/8/2013, 2:08 PM, Darren Shepherd wrote:
> Deploy db from maven will drop all the tables.  Not sure if this is
> fresh install or not.  For master, running mvn will be your best bet.
> Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
> manually if your adventurous.
>
> Darren
>
> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
> <fg...@cloudops.com> wrote:
>> Thanks Alena for the explaination.
>>
>> What is the better path to fix this on our setup? Should I wait for a fix in
>> master or should I manually run the deploydb with mvn? I guess the second
>> option won't work since I used RPM?
>>
>> Francois
>>
>>
>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>> Ok, this is what going on - the DB upgrade procedure is different on
>>> developer's setup and when deployed using cloudstack-setup-databases
>>>
>>> On developers setup:
>>>
>>> 1) you deploy the code
>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
>>> part of this step, the DataBaseUpgradeChecker:
>>>
>>> * first deploys the base DB version - 4.0.0
>>> * then checks the current version of the code, and performs the db upgrade
>>> if needed. So on master, version table looks like this after the db is
>>> deployed:
>>>
>>> mysql> select * from version;
>>> +----+---------+---------------------+----------+
>>> | id | version | updated             | step     |
>>> +----+---------+---------------------+----------+
>>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>>> +----+---------+---------------------+----------+
>>> 4 rows in set (0.00 sec)
>>>
>>>
>>> 3) Start management server.
>>>
>>>
>>> When deployed from rpm:
>>>
>>> 1) you deploy the code
>>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0 base
>>> version of the DB is deployed. Thats why you see only 4.0.0 record in the
>>> DB.
>>> 3) Start management server. DataBaseUpgradeChecker is being invoked as a
>>> part of it, and performs the db upgrade to the version of the current
>>> code. Only after that all the managers get invoked + system caller context
>>> get initialized.
>>>
>>> Looks like the load order for step 3) got broken recently, and system
>>> context gets initialized before the db upgrade is finished. So we either
>>> need to fix the order, or invoke DataBaseUpgradeChecker as a part of
>>> cloudstack-setup-databases so at the point when management server starts
>>> up, the DB already upgraded to the latest version.
>>>
>>> -Alena.
>>>
>>>
>>> On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>> wrote:
>>>
>>>> Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>>>> no?
>>>>
>>>> mysql> select * from version;
>>>> +----+---------+---------------------+----------+
>>>> | id | version | updated             | step     |
>>>> +----+---------+---------------------+----------+
>>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>>> +----+---------+---------------------+----------+
>>>> 1 row in set (0.00 sec)
>>>>
>>>> I installed cloudstack-management-4.3.0:
>>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>>> /usr/bin/cloudstack-setup-databases
>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>>
>>>> Francois
>>>>
>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>> It's a fresh master RPM install.
>>>>>
>>>>> Francois
>>>>>
>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>> It is not a small issue. is_default filed was added to the table as a
>>>>>> part
>>>>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>>>>> user
>>>>>> before the db upgrade is completed.
>>>>>>
>>>>>> DB upgrade is a major part of system integrity check; no queries to
>>>>>> the DB
>>>>>> should be made before its completed. Francois, did you start seeing
>>>>>> this
>>>>>> problem just recently?
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I compiled Master this morning, and there is a small DB issue. One
>>>>>>> field
>>>>>>> is missing in the account table (field default). CS will not start
>>>>>>> because of that.
>>>>>>>
>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>>>>> Exiting the system because we're unable to register the system call
>>>>>>> context.
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>>>>> account.account_name, account.type, account.domain_id, account.state,
>>>>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>>> orD
>>>>>>>
>>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>        at
>>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.java:166
>>>>>>> )
>>>>>>>
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallContextOnce
>>>>>>> Onl
>>>>>>>
>>>>>>> y(CallContext.java:141)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext(CallCo
>>>>>>> nte
>>>>>>>
>>>>>>> xtListener.java:36)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>>> thC
>>>>>>>
>>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>>> hCo
>>>>>>>
>>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>>> dCo
>>>>>>>
>>>>>>> ntextRunnable.java:46)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Manag
>>>>>>> edC
>>>>>>>
>>>>>>> ontextTimerTask.java:27)
>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>>        at java.util.TimerThread.run(Timer.java:484)
>>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>> Method)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
>>>>>>> Acc
>>>>>>>
>>>>>>> essorImpl.java:57)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
>>>>>>> str
>>>>>>>
>>>>>>> uctorAccessorImpl.java:45)
>>>>>>>        at
>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>>        at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>>        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>>        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>>        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
>>>>>>> :21
>>>>>>>
>>>>>>> 55)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:23
>>>>>>> 18)
>>>>>>>
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>>> tin
>>>>>>>
>>>>>>> gPreparedStatement.java:96)
>>>>>>>        at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>>> tin
>>>>>>>
>>>>>>> gPreparedStatement.java:96)
>>>>>>>        at
>>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>>        ... 27 more
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Francois Gaudreault
>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>>> fgaudreault@cloudops.com
>>>>>>> 514-629-6775
>>>>>>> - - -
>>>>>>> CloudOps
>>>>>>> 420 rue Guy
>>>>>>> Montréal QC  H3J 1S6
>>>>>>> www.cloudops.com
>>>>>>> @CloudOps_
>>>>>>>
>>>>>>>
>>>>>>
>>>> --
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>> fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>>>>
>>>
>>>
>>
>> --
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>
>


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Darren Shepherd <da...@gmail.com>.

Deploy db from maven will drop all the tables.  Not sure if this is
fresh install or not.  For master, running mvn will be your best bet.
Otherwise you can look at running com.cloud.upgrade.DatabaseCreator
manually if your adventurous.

Darren

On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault
<fg...@cloudops.com> wrote:
> Thanks Alena for the explaination.
>
> What is the better path to fix this on our setup? Should I wait for a fix in
> master or should I manually run the deploydb with mvn? I guess the second
> option won't work since I used RPM?
>
> Francois
>
>
> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>>
>> Ok, this is what going on - the DB upgrade procedure is different on
>> developer's setup and when deployed using cloudstack-setup-databases
>>
>> On developers setup:
>>
>> 1) you deploy the code
>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
>> part of this step, the DataBaseUpgradeChecker:
>>
>> * first deploys the base DB version - 4.0.0
>> * then checks the current version of the code, and performs the db upgrade
>> if needed. So on master, version table looks like this after the db is
>> deployed:
>>
>> mysql> select * from version;
>> +----+---------+---------------------+----------+
>> | id | version | updated             | step     |
>> +----+---------+---------------------+----------+
>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>> +----+---------+---------------------+----------+
>> 4 rows in set (0.00 sec)
>>
>>
>> 3) Start management server.
>>
>>
>> When deployed from rpm:
>>
>> 1) you deploy the code
>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0 base
>> version of the DB is deployed. Thats why you see only 4.0.0 record in the
>> DB.
>> 3) Start management server. DataBaseUpgradeChecker is being invoked as a
>> part of it, and performs the db upgrade to the version of the current
>> code. Only after that all the managers get invoked + system caller context
>> get initialized.
>>
>> Looks like the load order for step 3) got broken recently, and system
>> context gets initialized before the db upgrade is finished. So we either
>> need to fix the order, or invoke DataBaseUpgradeChecker as a part of
>> cloudstack-setup-databases so at the point when management server starts
>> up, the DB already upgraded to the latest version.
>>
>> -Alena.
>>
>>
>> On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
>> wrote:
>>
>>> Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>>> no?
>>>
>>> mysql> select * from version;
>>> +----+---------+---------------------+----------+
>>> | id | version | updated             | step     |
>>> +----+---------+---------------------+----------+
>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>> +----+---------+---------------------+----------+
>>> 1 row in set (0.00 sec)
>>>
>>> I installed cloudstack-management-4.3.0:
>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>> /usr/bin/cloudstack-setup-databases
>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>
>>> Francois
>>>
>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>>
>>>> It's a fresh master RPM install.
>>>>
>>>> Francois
>>>>
>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>>
>>>>> It is not a small issue. is_default filed was added to the table as a
>>>>> part
>>>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>>>> user
>>>>> before the db upgrade is completed.
>>>>>
>>>>> DB upgrade is a major part of system integrity check; no queries to
>>>>> the DB
>>>>> should be made before its completed. Francois, did you start seeing
>>>>> this
>>>>> problem just recently?
>>>>>
>>>>> -Alena.
>>>>>
>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I compiled Master this morning, and there is a small DB issue. One
>>>>>> field
>>>>>> is missing in the account table (field default). CS will not start
>>>>>> because of that.
>>>>>>
>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>>>> Exiting the system because we're unable to register the system call
>>>>>> context.
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>>>> account.account_name, account.type, account.domain_id, account.state,
>>>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.java:166
>>>>>> )
>>>>>>
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallContextOnce
>>>>>> Onl
>>>>>>
>>>>>> y(CallContext.java:141)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext(CallCo
>>>>>> nte
>>>>>>
>>>>>> xtListener.java:36)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>> thC
>>>>>>
>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>> hCo
>>>>>>
>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>> dCo
>>>>>>
>>>>>> ntextRunnable.java:46)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Manag
>>>>>> edC
>>>>>>
>>>>>> ontextTimerTask.java:27)
>>>>>>       at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>       at java.util.TimerThread.run(Timer.java:484)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
>>>>>> Acc
>>>>>>
>>>>>> essorImpl.java:57)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
>>>>>> str
>>>>>>
>>>>>> uctorAccessorImpl.java:45)
>>>>>>       at
>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>       at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>       at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>       at
>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>       at
>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
>>>>>> :21
>>>>>>
>>>>>> 55)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:23
>>>>>> 18)
>>>>>>
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>> tin
>>>>>>
>>>>>> gPreparedStatement.java:96)
>>>>>>       at
>>>>>>
>>>>>>
>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>>> tin
>>>>>>
>>>>>> gPreparedStatement.java:96)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>       ... 27 more
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Francois Gaudreault
>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>> fgaudreault@cloudops.com
>>>>>> 514-629-6775
>>>>>> - - -
>>>>>> CloudOps
>>>>>> 420 rue Guy
>>>>>> Montréal QC  H3J 1S6
>>>>>> www.cloudops.com
>>>>>> @CloudOps_
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>> fgaudreault@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>
>>>
>>
>>
>>
>
>
> --
> Francois Gaudreault
> Architecte de Solution Cloud | Cloud Solutions Architect
> fgaudreault@cloudops.com
> 514-629-6775
> - - -
> CloudOps
> 420 rue Guy
> Montréal QC  H3J 1S6
> www.cloudops.com
> @CloudOps_
>

Re: Latest Master DB issue

Posted by Alena Prokharchyk <Al...@citrix.com>.

I'm not familiar with RPM install, but I don't think that the workaround
would be valid for everyone - QA folks will not have mvn install on their
machines, so this issue is a blocker for them. You can try it on your
setup though to unblock yourself.

It does have to be fixed in master, either by spring-modularization branch
merge, or by applying some temporarily checkin till the merge is done if
too many people are blocked by that.

-Alena.

On 10/8/13 10:55 AM, "Francois Gaudreault" <fg...@cloudops.com>
wrote:

>Thanks Alena for the explaination.
>
>What is the better path to fix this on our setup? Should I wait for a
>fix in master or should I manually run the deploydb with mvn? I guess
>the second option won't work since I used RPM?
>
>Francois
>
>On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
>> Ok, this is what going on - the DB upgrade procedure is different on
>> developer's setup and when deployed using cloudstack-setup-databases
>>
>> On developers setup:
>>
>> 1) you deploy the code
>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
>> part of this step, the DataBaseUpgradeChecker:
>>
>> * first deploys the base DB version - 4.0.0
>> * then checks the current version of the code, and performs the db
>>upgrade
>> if needed. So on master, version table looks like this after the db is
>> deployed:
>>
>> mysql> select * from version;
>> +----+---------+---------------------+----------+
>> | id | version | updated             | step     |
>> +----+---------+---------------------+----------+
>> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
>> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
>> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
>> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
>> +----+---------+---------------------+----------+
>> 4 rows in set (0.00 sec)
>>
>>
>> 3) Start management server.
>>
>>
>> When deployed from rpm:
>>
>> 1) you deploy the code
>> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0
>>base
>> version of the DB is deployed. Thats why you see only 4.0.0 record in
>>the
>> DB.
>> 3) Start management server. DataBaseUpgradeChecker is being invoked as a
>> part of it, and performs the db upgrade to the version of the current
>> code. Only after that all the managers get invoked + system caller
>>context
>> get initialized.
>>
>> Looks like the load order for step 3) got broken recently, and system
>> context gets initialized before the db upgrade is finished. So we either
>> need to fix the order, or invoke DataBaseUpgradeChecker as a part of
>> cloudstack-setup-databases so at the point when management server starts
>> up, the DB already upgraded to the latest version.
>>
>> -Alena.
>>
>>
>> On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
>> wrote:
>>
>>> Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>>> no?
>>>
>>> mysql> select * from version;
>>> +----+---------+---------------------+----------+
>>> | id | version | updated             | step     |
>>> +----+---------+---------------------+----------+
>>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>>> +----+---------+---------------------+----------+
>>> 1 row in set (0.00 sec)
>>>
>>> I installed cloudstack-management-4.3.0:
>>> [root@eng-testing-cstack_master ~]# rpm -qf
>>> /usr/bin/cloudstack-setup-databases
>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>>
>>> Francois
>>>
>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>>> It's a fresh master RPM install.
>>>>
>>>> Francois
>>>>
>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>>> It is not a small issue. is_default filed was added to the table as a
>>>>> part
>>>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>>>> user
>>>>> before the db upgrade is completed.
>>>>>
>>>>> DB upgrade is a major part of system integrity check; no queries to
>>>>> the DB
>>>>> should be made before its completed. Francois, did you start seeing
>>>>> this
>>>>> problem just recently?
>>>>>
>>>>> -Alena.
>>>>>
>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I compiled Master this morning, and there is a small DB issue. One
>>>>>> field
>>>>>> is missing in the account table (field default). CS will not start
>>>>>> because of that.
>>>>>>
>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>>>> Exiting the system because we're unable to register the system call
>>>>>> context.
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>>>> account.account_name, account.type, account.domain_id,
>>>>>>account.state,
>>>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interce
>>>>>>pt
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interce
>>>>>>pt
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interce
>>>>>>pt
>>>>>> orD
>>>>>>
>>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>       at
>>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.context.CallContext.register(CallContext.java:1
>>>>>>66
>>>>>> )
>>>>>>
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.context.CallContext.registerSystemCallContextOn
>>>>>>ce
>>>>>> Onl
>>>>>>
>>>>>> y(CallContext.java:141)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.context.CallContextListener.onEnterContext(Call
>>>>>>Co
>>>>>> nte
>>>>>>
>>>>>> xtListener.java:36)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.call
>>>>>>Wi
>>>>>> thC
>>>>>>
>>>>>> ontext(DefaultManagedContext.java:83)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runW
>>>>>>it
>>>>>> hCo
>>>>>>
>>>>>> ntext(DefaultManagedContext.java:53)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Mana
>>>>>>ge
>>>>>> dCo
>>>>>>
>>>>>> ntextRunnable.java:46)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Man
>>>>>>ag
>>>>>> edC
>>>>>>
>>>>>> ontextTimerTask.java:27)
>>>>>>       at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>>       at java.util.TimerThread.run(Timer.java:484)
>>>>>> Caused by: 
>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>>> Unknown column 'account.default' in 'field list'
>>>>>>       at 
>>>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
>>>>>>or
>>>>>> Acc
>>>>>>
>>>>>> essorImpl.java:57)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
>>>>>>on
>>>>>> str
>>>>>>
>>>>>> uctorAccessorImpl.java:45)
>>>>>>       at
>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>       at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>       at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>       at 
>>>>>>com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>>       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>       at
>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.ja
>>>>>>va
>>>>>> :21
>>>>>>
>>>>>> 55)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:
>>>>>>23
>>>>>> 18)
>>>>>>
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Dele
>>>>>>ga
>>>>>> tin
>>>>>>
>>>>>> gPreparedStatement.java:96)
>>>>>>       at
>>>>>>
>>>>>> 
>>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Dele
>>>>>>ga
>>>>>> tin
>>>>>>
>>>>>> gPreparedStatement.java:96)
>>>>>>       at
>>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>>       ... 27 more
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Francois Gaudreault
>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>> fgaudreault@cloudops.com
>>>>>> 514-629-6775
>>>>>> - - -
>>>>>> CloudOps
>>>>>> 420 rue Guy
>>>>>> Montréal QC  H3J 1S6
>>>>>> www.cloudops.com
>>>>>> @CloudOps_
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> -- 
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>> fgaudreault@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>
>>>
>>
>>
>>
>
>
>-- 
>Francois Gaudreault
>Architecte de Solution Cloud | Cloud Solutions Architect
>fgaudreault@cloudops.com
>514-629-6775
>- - -
>CloudOps
>420 rue Guy
>Montréal QC  H3J 1S6
>www.cloudops.com
>@CloudOps_
>
>

Re: Latest Master DB issue

Posted by Francois Gaudreault <fg...@cloudops.com>.

Thanks Alena for the explaination.

What is the better path to fix this on our setup? Should I wait for a 
fix in master or should I manually run the deploydb with mvn? I guess 
the second option won't work since I used RPM?

Francois

On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote:
> Ok, this is what going on - the DB upgrade procedure is different on
> developer's setup and when deployed using cloudstack-setup-databases
>
> On developers setup:
>
> 1) you deploy the code
> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
> part of this step, the DataBaseUpgradeChecker:
>
> * first deploys the base DB version - 4.0.0
> * then checks the current version of the code, and performs the db upgrade
> if needed. So on master, version table looks like this after the db is
> deployed:
>
> mysql> select * from version;
> +----+---------+---------------------+----------+
> | id | version | updated             | step     |
> +----+---------+---------------------+----------+
> |  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
> |  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
> |  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
> |  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
> +----+---------+---------------------+----------+
> 4 rows in set (0.00 sec)
>
>
> 3) Start management server.
>
>
> When deployed from rpm:
>
> 1) you deploy the code
> 2) run cloudstack-setup-databases. As the result of this step, 4.0.0 base
> version of the DB is deployed. Thats why you see only 4.0.0 record in the
> DB.
> 3) Start management server. DataBaseUpgradeChecker is being invoked as a
> part of it, and performs the db upgrade to the version of the current
> code. Only after that all the managers get invoked + system caller context
> get initialized.
>
> Looks like the load order for step 3) got broken recently, and system
> context gets initialized before the db upgrade is finished. So we either
> need to fix the order, or invoke DataBaseUpgradeChecker as a part of
> cloudstack-setup-databases so at the point when management server starts
> up, the DB already upgraded to the latest version.
>
> -Alena.
>
>
> On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
> wrote:
>
>> Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>> no?
>>
>> mysql> select * from version;
>> +----+---------+---------------------+----------+
>> | id | version | updated             | step     |
>> +----+---------+---------------------+----------+
>> |  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>> +----+---------+---------------------+----------+
>> 1 row in set (0.00 sec)
>>
>> I installed cloudstack-management-4.3.0:
>> [root@eng-testing-cstack_master ~]# rpm -qf
>> /usr/bin/cloudstack-setup-databases
>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>>
>> Francois
>>
>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>>> It's a fresh master RPM install.
>>>
>>> Francois
>>>
>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>>> It is not a small issue. is_default filed was added to the table as a
>>>> part
>>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>>> user
>>>> before the db upgrade is completed.
>>>>
>>>> DB upgrade is a major part of system integrity check; no queries to
>>>> the DB
>>>> should be made before its completed. Francois, did you start seeing
>>>> this
>>>> problem just recently?
>>>>
>>>> -Alena.
>>>>
>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I compiled Master this morning, and there is a small DB issue. One
>>>>> field
>>>>> is missing in the account table (field default). CS will not start
>>>>> because of that.
>>>>>
>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>>> Exiting the system because we're unable to register the system call
>>>>> context.
>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>>> account.account_name, account.type, account.domain_id, account.state,
>>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>>       at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>>       at
>>>>>
>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>> orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>       at
>>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>>       at
>>>>>
>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>> orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>       at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>>       at
>>>>>
>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>> orD
>>>>>
>>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>       at
>>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.context.CallContext.register(CallContext.java:166
>>>>> )
>>>>>
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.context.CallContext.registerSystemCallContextOnce
>>>>> Onl
>>>>>
>>>>> y(CallContext.java:141)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.context.CallContextListener.onEnterContext(CallCo
>>>>> nte
>>>>>
>>>>> xtListener.java:36)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>> thC
>>>>>
>>>>> ontext(DefaultManagedContext.java:83)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>> hCo
>>>>>
>>>>> ntext(DefaultManagedContext.java:53)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>> dCo
>>>>>
>>>>> ntextRunnable.java:46)
>>>>>       at
>>>>>
>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Manag
>>>>> edC
>>>>>
>>>>> ontextTimerTask.java:27)
>>>>>       at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>       at java.util.TimerThread.run(Timer.java:484)
>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>>> Unknown column 'account.default' in 'field list'
>>>>>       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>       at
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
>>>>> Acc
>>>>>
>>>>> essorImpl.java:57)
>>>>>       at
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
>>>>> str
>>>>>
>>>>> uctorAccessorImpl.java:45)
>>>>>       at
>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>       at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>       at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>>       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>       at
>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>       at
>>>>>
>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
>>>>> :21
>>>>>
>>>>> 55)
>>>>>       at
>>>>>
>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:23
>>>>> 18)
>>>>>
>>>>>       at
>>>>>
>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>> tin
>>>>>
>>>>> gPreparedStatement.java:96)
>>>>>       at
>>>>>
>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>> tin
>>>>>
>>>>> gPreparedStatement.java:96)
>>>>>       at
>>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>>       ... 27 more
>>>>>
>>>>>
>>>>> -- 
>>>>> Francois Gaudreault
>>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>>> fgaudreault@cloudops.com
>>>>> 514-629-6775
>>>>> - - -
>>>>> CloudOps
>>>>> 420 rue Guy
>>>>> Montréal QC  H3J 1S6
>>>>> www.cloudops.com
>>>>> @CloudOps_
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>> -- 
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>
>>
>
>
>


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Alena Prokharchyk <Al...@citrix.com>.

Ok, this is what going on - the DB upgrade procedure is different on
developer's setup and when deployed using cloudstack-setup-databases

On developers setup:

1) you deploy the code
2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. As a
part of this step, the DataBaseUpgradeChecker:

* first deploys the base DB version - 4.0.0
* then checks the current version of the code, and performs the db upgrade
if needed. So on master, version table looks like this after the db is
deployed:

mysql> select * from version;
+----+---------+---------------------+----------+
| id | version | updated             | step     |
+----+---------+---------------------+----------+
|  1 | 4.0.0   | 2013-10-08 10:34:47 | Complete |
|  2 | 4.1.0   | 2013-10-08 17:35:22 | Complete |
|  3 | 4.2.0   | 2013-10-08 17:35:22 | Complete |
|  4 | 4.3.0   | 2013-10-08 17:35:22 | Complete |
+----+---------+---------------------+----------+
4 rows in set (0.00 sec)


3) Start management server.


When deployed from rpm:

1) you deploy the code
2) run cloudstack-setup-databases. As the result of this step, 4.0.0 base
version of the DB is deployed. Thats why you see only 4.0.0 record in the
DB.
3) Start management server. DataBaseUpgradeChecker is being invoked as a
part of it, and performs the db upgrade to the version of the current
code. Only after that all the managers get invoked + system caller context
get initialized.

Looks like the load order for step 3) got broken recently, and system
context gets initialized before the db upgrade is finished. So we either
need to fix the order, or invoke DataBaseUpgradeChecker as a part of
cloudstack-setup-databases so at the point when management server starts
up, the DB already upgraded to the latest version.

-Alena.


On 10/8/13 10:25 AM, "Francois Gaudreault" <fg...@cloudops.com>
wrote:

>Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0
>no?
>
>mysql> select * from version;
>+----+---------+---------------------+----------+
>| id | version | updated             | step     |
>+----+---------+---------------------+----------+
>|  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
>+----+---------+---------------------+----------+
>1 row in set (0.00 sec)
>
>I installed cloudstack-management-4.3.0:
>[root@eng-testing-cstack_master ~]# rpm -qf
>/usr/bin/cloudstack-setup-databases
>cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64
>
>Francois
>
>On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
>> It's a fresh master RPM install.
>>
>> Francois
>>
>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>>> It is not a small issue. is_default filed was added to the table as a
>>> part
>>> of the 41-42 db upgrade. Looks like the code tries to retrieve system
>>> user
>>> before the db upgrade is completed.
>>>
>>> DB upgrade is a major part of system integrity check; no queries to
>>> the DB
>>> should be made before its completed. Francois, did you start seeing
>>>this
>>> problem just recently?
>>>
>>> -Alena.
>>>
>>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I compiled Master this morning, and there is a small DB issue. One
>>>> field
>>>> is missing in the account table (field default). CS will not start
>>>> because of that.
>>>>
>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>>> Exiting the system because we're unable to register the system call
>>>> context.
>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>>> account.account_name, account.type, account.domain_id, account.state,
>>>> account.removed, account.cleanup_needed, account.network_domain,
>>>> account.uuid, account.default_zone_id, account.default FROM account
>>>> WHERE account.id = 1  AND account.removed IS NULL
>>>>      at
>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>>      at
>>>> 
>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>orD 
>>>>
>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>      at 
>>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>>      at
>>>> 
>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>orD 
>>>>
>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>      at
>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>>      at
>>>> 
>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercept
>>>>orD 
>>>>
>>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>      at
>>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.context.CallContext.register(CallContext.java:166
>>>>) 
>>>>
>>>>      at
>>>> 
>>>>org.apache.cloudstack.context.CallContext.registerSystemCallContextOnce
>>>>Onl 
>>>>
>>>> y(CallContext.java:141)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.context.CallContextListener.onEnterContext(CallCo
>>>>nte 
>>>>
>>>> xtListener.java:36)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>thC 
>>>>
>>>> ontext(DefaultManagedContext.java:83)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>hCo 
>>>>
>>>> ntext(DefaultManagedContext.java:53)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>dCo 
>>>>
>>>> ntextRunnable.java:46)
>>>>      at
>>>> 
>>>>org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(Manag
>>>>edC 
>>>>
>>>> ontextTimerTask.java:27)
>>>>      at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>      at java.util.TimerThread.run(Timer.java:484)
>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>>> Unknown column 'account.default' in 'field list'
>>>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>>      at
>>>> 
>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
>>>>Acc 
>>>>
>>>> essorImpl.java:57)
>>>>      at
>>>> 
>>>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
>>>>str 
>>>>
>>>> uctorAccessorImpl.java:45)
>>>>      at 
>>>>java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>      at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>      at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>>      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>      at 
>>>>com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>      at
>>>> 
>>>>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
>>>>:21 
>>>>
>>>> 55)
>>>>      at
>>>> 
>>>>com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:23
>>>>18) 
>>>>
>>>>      at
>>>> 
>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>tin 
>>>>
>>>> gPreparedStatement.java:96)
>>>>      at
>>>> 
>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delega
>>>>tin 
>>>>
>>>> gPreparedStatement.java:96)
>>>>      at
>>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>>      ... 27 more
>>>>
>>>>
>>>> -- 
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>> fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>-- 
>Francois Gaudreault
>Architecte de Solution Cloud | Cloud Solutions Architect
>fgaudreault@cloudops.com
>514-629-6775
>- - -
>CloudOps
>420 rue Guy
>Montréal QC  H3J 1S6
>www.cloudops.com
>@CloudOps_
>
>

Re: Latest Master DB issue

Posted by Francois Gaudreault <fg...@cloudops.com>.

Hmm... I just checked the DB version and it's 4.0??? It should be 4.3.0 no?

mysql> select * from version;
+----+---------+---------------------+----------+
| id | version | updated             | step     |
+----+---------+---------------------+----------+
|  1 | 4.0.0   | 2013-10-08 10:58:49 | Complete |
+----+---------+---------------------+----------+
1 row in set (0.00 sec)

I installed cloudstack-management-4.3.0:
[root@eng-testing-cstack_master ~]# rpm -qf 
/usr/bin/cloudstack-setup-databases
cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64

Francois

On 10/8/2013, 1:04 PM, Francois Gaudreault wrote:
> It's a fresh master RPM install.
>
> Francois
>
> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
>> It is not a small issue. is_default filed was added to the table as a 
>> part
>> of the 41-42 db upgrade. Looks like the code tries to retrieve system 
>> user
>> before the db upgrade is completed.
>>
>> DB upgrade is a major part of system integrity check; no queries to 
>> the DB
>> should be made before its completed. Francois, did you start seeing this
>> problem just recently?
>>
>> -Alena.
>>
>> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com> 
>> wrote:
>>
>>> Hi,
>>>
>>> I compiled Master this morning, and there is a small DB issue. One 
>>> field
>>> is missing in the account table (field default). CS will not start
>>> because of that.
>>>
>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>>> Exiting the system because we're unable to register the system call
>>> context.
>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>>> account.account_name, account.type, account.domain_id, account.state,
>>> account.removed, account.cleanup_needed, account.network_domain,
>>> account.uuid, account.default_zone_id, account.default FROM account
>>> WHERE account.id = 1  AND account.removed IS NULL
>>>      at
>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>>      at
>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD 
>>>
>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>      at 
>>> com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>>      at
>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD 
>>>
>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>      at
>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>>      at
>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD 
>>>
>>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>      at
>>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>>      at
>>> org.apache.cloudstack.context.CallContext.register(CallContext.java:166) 
>>>
>>>      at
>>> org.apache.cloudstack.context.CallContext.registerSystemCallContextOnceOnl 
>>>
>>> y(CallContext.java:141)
>>>      at
>>> org.apache.cloudstack.context.CallContextListener.onEnterContext(CallConte 
>>>
>>> xtListener.java:36)
>>>      at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC 
>>>
>>> ontext(DefaultManagedContext.java:83)
>>>      at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo 
>>>
>>> ntext(DefaultManagedContext.java:53)
>>>      at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCo 
>>>
>>> ntextRunnable.java:46)
>>>      at
>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedC 
>>>
>>> ontextTimerTask.java:27)
>>>      at java.util.TimerThread.mainLoop(Timer.java:534)
>>>      at java.util.TimerThread.run(Timer.java:484)
>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>>> Unknown column 'account.default' in 'field list'
>>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>      at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc 
>>>
>>> essorImpl.java:57)
>>>      at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr 
>>>
>>> uctorAccessorImpl.java:45)
>>>      at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>      at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>      at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>>      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>      at
>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:21 
>>>
>>> 55)
>>>      at
>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318) 
>>>
>>>      at
>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin 
>>>
>>> gPreparedStatement.java:96)
>>>      at
>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin 
>>>
>>> gPreparedStatement.java:96)
>>>      at
>>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>>      ... 27 more
>>>
>>>
>>> -- 
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>> fgaudreault@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>
>>>
>>
>>
>>
>
>


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Francois Gaudreault <fg...@cloudops.com>.

It's a fresh master RPM install.

Francois

On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote:
> It is not a small issue. is_default filed was added to the table as a part
> of the 41-42 db upgrade. Looks like the code tries to retrieve system user
> before the db upgrade is completed.
>
> DB upgrade is a major part of system integrity check; no queries to the DB
> should be made before its completed. Francois, did you start seeing this
> problem just recently?
>
> -Alena.
>
> On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com> wrote:
>
>> Hi,
>>
>> I compiled Master this morning, and there is a small DB issue. One field
>> is missing in the account table (field default). CS will not start
>> because of that.
>>
>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>> Exiting the system because we're unable to register the system call
>> context.
>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>> account.account_name, account.type, account.domain_id, account.state,
>> account.removed, account.cleanup_needed, account.network_domain,
>> account.uuid, account.default_zone_id, account.default FROM account
>> WHERE account.id = 1  AND account.removed IS NULL
>>      at
>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>>      at
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>      at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>>      at
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>      at
>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>>      at
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>> ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>      at
>> com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>>      at
>> org.apache.cloudstack.context.CallContext.register(CallContext.java:166)
>>      at
>> org.apache.cloudstack.context.CallContext.registerSystemCallContextOnceOnl
>> y(CallContext.java:141)
>>      at
>> org.apache.cloudstack.context.CallContextListener.onEnterContext(CallConte
>> xtListener.java:36)
>>      at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC
>> ontext(DefaultManagedContext.java:83)
>>      at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo
>> ntext(DefaultManagedContext.java:53)
>>      at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCo
>> ntextRunnable.java:46)
>>      at
>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedC
>> ontextTimerTask.java:27)
>>      at java.util.TimerThread.mainLoop(Timer.java:534)
>>      at java.util.TimerThread.run(Timer.java:484)
>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>> Unknown column 'account.default' in 'field list'
>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>      at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
>> essorImpl.java:57)
>>      at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
>> uctorAccessorImpl.java:45)
>>      at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>      at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>      at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>>      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>>      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>      at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>      at
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:21
>> 55)
>>      at
>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
>>      at
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin
>> gPreparedStatement.java:96)
>>      at
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin
>> gPreparedStatement.java:96)
>>      at
>> com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>>      ... 27 more
>>
>>
>> -- 
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>
>>
>
>
>


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Latest Master DB issue

Posted by Alena Prokharchyk <Al...@citrix.com>.

It is not a small issue. is_default filed was added to the table as a part
of the 41-42 db upgrade. Looks like the code tries to retrieve system user
before the db upgrade is completed.

DB upgrade is a major part of system integrity check; no queries to the DB
should be made before its completed. Francois, did you start seeing this
problem just recently?

-Alena.

On 10/8/13 8:04 AM, "Francois Gaudreault" <fg...@cloudops.com> wrote:

>Hi,
>
>I compiled Master this morning, and there is a small DB issue. One field
>is missing in the account table (field default). CS will not start
>because of that.
>
>2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] (Timer-2:null)
>Exiting the system because we're unable to register the system call
>context.
>com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
>com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT account.id,
>account.account_name, account.type, account.domain_id, account.state,
>account.removed, account.cleanup_needed, account.network_domain,
>account.uuid, account.default_zone_id, account.default FROM account
>WHERE account.id = 1  AND account.removed IS NULL
>     at 
>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:986)
>     at 
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>     at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:963)
>     at 
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>     at 
>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:926)
>     at 
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>     at 
>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:45)
>     at 
>org.apache.cloudstack.context.CallContext.register(CallContext.java:166)
>     at 
>org.apache.cloudstack.context.CallContext.registerSystemCallContextOnceOnl
>y(CallContext.java:141)
>     at 
>org.apache.cloudstack.context.CallContextListener.onEnterContext(CallConte
>xtListener.java:36)
>     at 
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC
>ontext(DefaultManagedContext.java:83)
>     at 
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo
>ntext(DefaultManagedContext.java:53)
>     at 
>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCo
>ntextRunnable.java:46)
>     at 
>org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedC
>ontextTimerTask.java:27)
>     at java.util.TimerThread.mainLoop(Timer.java:534)
>     at java.util.TimerThread.run(Timer.java:484)
>Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
>Unknown column 'account.default' in 'field list'
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>Method)
>     at 
>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
>essorImpl.java:57)
>     at 
>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
>uctorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>     at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>     at com.mysql.jdbc.Util.getInstance(Util.java:386)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>     at 
>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:21
>55)
>     at 
>com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
>     at 
>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin
>gPreparedStatement.java:96)
>     at 
>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Delegatin
>gPreparedStatement.java:96)
>     at 
>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:983)
>     ... 27 more
>
>
>-- 
>Francois Gaudreault
>Architecte de Solution Cloud | Cloud Solutions Architect
>fgaudreault@cloudops.com
>514-629-6775
>- - -
>CloudOps
>420 rue Guy
>Montréal QC  H3J 1S6
>www.cloudops.com
>@CloudOps_
>
>