You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Dave Cahill <dc...@midokura.jp> on 2012/11/02 09:26:07 UTC
EntityExistsException on management server startup (createDefaultNetworks)
Hi,
Over the past few days, I've run into EntityExistsException several times
when I shut down and restart my management server:
ERROR [cloud.servlet.CloudStartupServlet] (main:) Exception starting
management server
javax.persistence.EntityExistsException: Entity already exists:
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1318)
This generally requires a drop database, redeploy etc - very time consuming.
I dug into the error a little, and the code behavior seems a little strange.
See com.cloud.server.ConfigurationServerImpl.java, createDefaultNetworks(),
line 1085:
long id = 1;
And line 1141:
_networkDao.persist(network, false,
getServicesAndProvidersForNetwork(networkOfferingId));
id++;
The code is trying to insert rows into the networks table, with "id"s
starting at 1. However, the existing values in the networks table also
start at 1, giving a primary key exception and causing the management
server not to start successfully.
Is the networks table supposed to always be empty at startup so that these
inserts will always succeed even though they are setting id to 1 blindly?
Or is the code wrong to be starting IDs at 1 and failing on duplicate key
exceptions?
Thanks,
Dave.
*Info on my setup:*
Running CloudStack from master (4.1.0-SNAPSHOT).
Setup process for the management server was:
Grab latest from git repo, then:
mvn install
mvn -P developer -pl developer -Ddeploydb
export MAVEN_OPTS="-Xmx1024m -Xdebug
-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
mvn -pl :cloud-client-ui jetty:run
Re: EntityExistsException on management server startup
(createDefaultNetworks)
Posted by Alena Prokharchyk <Al...@citrix.com>.
Hi Dave,
See my comments inline.
-Alena.
On 11/4/12 9:49 PM, "Dave Cahill" <dc...@midokura.jp> wrote:
>Hi Alena,
>
>Thanks very much for your reply.
>
>> 1) ConfigurationServerImpl, persistDefaultValues method
>>
>> Wrap all the DB updates in "if (init == null || init.equals("false"))
>>{}"
>in the single transaction. So we don't up with partially updated DB.
>
>To do this, would we just add a line like this as the first line of the
>if() block:
> Transaction persistDefaultsTxn = Transaction.currentTxn();
>
>And then this as the last line of the if() block:
> persistDefaultsTxn.commit()
Yes, that should be enough. Just make sure that all of the methods called
inside the block, don't do any database callbacks outside the current
transaction. Otherwise they might not get the latest data, and will fail.
Basically call to each and every method should be unit tested before
committing the fix.
>
>There are also several lines which are outside the if() block, from
>updateResourceCount() to updateUuids(). I'm guessing these should stay
>outside the if() block, but I'd imagine we need to bring the
>_configDao.update("init", "Hidden", "true"); line back into the if()
>block?
These lines should stay outside the if() block.
>
>I'm happy to create a patch once I'm sure I understand the intention.
>
>> 2) We should remove call to createDefaultNetworks from
>persistDefaultValues().
>
>From your description, it sounds like removing this might cause issues for
>people who manually enter zone data. If you're sure this is not an issue,
>I'm happy to remove it and submit a patch.
I think we should leave it for now as people still might be using manual
mysql update prior to the management server start.
>
>Thanks,
>Dave.
>
>
>On Sat, Nov 3, 2012 at 2:29 AM, Alena Prokharchyk <
>Alena.Prokharchyk@citrix.com> wrote:
>
>> Dave,
>>
>> This code - default network creation - is initiated only when the Global
>> config value "init" is false, as a part of persisting the default
>> cloudStack values. Init is false at the very first management server
>> startup, and it's reset to true after the default values are persisted.
>>
>> Looks like in your setup init process was interrupted in the middle of
>> the execution, so init was never reset to true, but some default values
>> were persisted and we never rolled them back. There are 2 bugs to be
>>fixed
>> in this area:
>>
>> 1) ConfigurationServerImpl, persistDefaultValues method
>>
>> Wrap all the DB updates in "if (init == null || init.equals("false"))
>>{}"
>> in the single transaction. So we don't up with partially updated DB.
>>
>> 2) We should remove call to createDefaultNetworks from
>> persistDefaultValues(). This call was added long ago to support the
>> scenario when:
>>
>> * developer manually inserts zone entry to the cloud.data_center table
>> before the very first management server startup. We used to have the dev
>> helper script that read data center and other config info from
>> server-setup.xml file, and did DB update.
>> * so when the management server starts up initially, the code finds the
>> zone, and creates the default networks for it.
>>
>> As cloudStack customers and most of the cloudStack developers create the
>> zone using API after the management server startup (and not by using
>> manual db update prior to startup), the call to "createDefaultNetworks"
>> should become obsolete.
>>
>> To fix it on your setup, you can either re-deploy the DB - safer
>>solution
>> as we don't know which default values were persisted, and what was
>> dropped; or set init=true and restart management server. Also if you
>> re-deploy the DB, check if you see any exceptions in the log while
>> cloudStack persisting the default values (the log snippet should begin
>> with "ConfigurationServer is saving default values to the database")
>>
>> -Alena.
>>
>>
>> On 11/2/12 1:26 AM, "Dave Cahill" <dc...@midokura.jp> wrote:
>>
>> >Hi,
>> >
>> >Over the past few days, I've run into EntityExistsException several
>>times
>> >when I shut down and restart my management server:
>> >
>> >ERROR [cloud.servlet.CloudStartupServlet] (main:) Exception starting
>> >management server
>> >javax.persistence.EntityExistsException: Entity already exists:
>> >at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1318)
>> >
>> >This generally requires a drop database, redeploy etc - very time
>> >consuming.
>> >
>> >I dug into the error a little, and the code behavior seems a little
>> >strange.
>> >
>> >See com.cloud.server.ConfigurationServerImpl.java,
>> >createDefaultNetworks(),
>> >line 1085:
>> >
>> >long id = 1;
>> >
>> >And line 1141:
>> >_networkDao.persist(network, false,
>> >getServicesAndProvidersForNetwork(networkOfferingId));
>> >id++;
>> >
>> >The code is trying to insert rows into the networks table, with "id"s
>> >starting at 1. However, the existing values in the networks table also
>> >start at 1, giving a primary key exception and causing the management
>> >server not to start successfully.
>> >
>> >Is the networks table supposed to always be empty at startup so that
>>these
>> >inserts will always succeed even though they are setting id to 1
>>blindly?
>> >Or is the code wrong to be starting IDs at 1 and failing on duplicate
>>key
>> >exceptions?
>> >
>> >Thanks,
>> >Dave.
>> >
>> >
>> >
>> >*Info on my setup:*
>> >Running CloudStack from master (4.1.0-SNAPSHOT).
>> >
>> >Setup process for the management server was:
>> >
>> >Grab latest from git repo, then:
>> >
>> >mvn install
>> >mvn -P developer -pl developer -Ddeploydb
>> >export MAVEN_OPTS="-Xmx1024m -Xdebug
>> >-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
>> >mvn -pl :cloud-client-ui jetty:run
>> >
>>
>>
>>
>
>
>--
>Thanks,
>Dave.
>
Re: EntityExistsException on management server startup (createDefaultNetworks)
Posted by Dave Cahill <dc...@midokura.jp>.
Hi Alena,
Thanks very much for your reply.
> 1) ConfigurationServerImpl, persistDefaultValues method
>
> Wrap all the DB updates in "if (init == null || init.equals("false")) {}"
in the single transaction. So we don't up with partially updated DB.
To do this, would we just add a line like this as the first line of the
if() block:
Transaction persistDefaultsTxn = Transaction.currentTxn();
And then this as the last line of the if() block:
persistDefaultsTxn.commit()
There are also several lines which are outside the if() block, from
updateResourceCount() to updateUuids(). I'm guessing these should stay
outside the if() block, but I'd imagine we need to bring the
_configDao.update("init", "Hidden", "true"); line back into the if() block?
I'm happy to create a patch once I'm sure I understand the intention.
> 2) We should remove call to createDefaultNetworks from
persistDefaultValues().
>From your description, it sounds like removing this might cause issues for
people who manually enter zone data. If you're sure this is not an issue,
I'm happy to remove it and submit a patch.
Thanks,
Dave.
On Sat, Nov 3, 2012 at 2:29 AM, Alena Prokharchyk <
Alena.Prokharchyk@citrix.com> wrote:
> Dave,
>
> This code - default network creation - is initiated only when the Global
> config value "init" is false, as a part of persisting the default
> cloudStack values. Init is false at the very first management server
> startup, and it's reset to true after the default values are persisted.
>
> Looks like in your setup init process was interrupted in the middle of
> the execution, so init was never reset to true, but some default values
> were persisted and we never rolled them back. There are 2 bugs to be fixed
> in this area:
>
> 1) ConfigurationServerImpl, persistDefaultValues method
>
> Wrap all the DB updates in "if (init == null || init.equals("false")) {}"
> in the single transaction. So we don't up with partially updated DB.
>
> 2) We should remove call to createDefaultNetworks from
> persistDefaultValues(). This call was added long ago to support the
> scenario when:
>
> * developer manually inserts zone entry to the cloud.data_center table
> before the very first management server startup. We used to have the dev
> helper script that read data center and other config info from
> server-setup.xml file, and did DB update.
> * so when the management server starts up initially, the code finds the
> zone, and creates the default networks for it.
>
> As cloudStack customers and most of the cloudStack developers create the
> zone using API after the management server startup (and not by using
> manual db update prior to startup), the call to "createDefaultNetworks"
> should become obsolete.
>
> To fix it on your setup, you can either re-deploy the DB - safer solution
> as we don't know which default values were persisted, and what was
> dropped; or set init=true and restart management server. Also if you
> re-deploy the DB, check if you see any exceptions in the log while
> cloudStack persisting the default values (the log snippet should begin
> with "ConfigurationServer is saving default values to the database")
>
> -Alena.
>
>
> On 11/2/12 1:26 AM, "Dave Cahill" <dc...@midokura.jp> wrote:
>
> >Hi,
> >
> >Over the past few days, I've run into EntityExistsException several times
> >when I shut down and restart my management server:
> >
> >ERROR [cloud.servlet.CloudStartupServlet] (main:) Exception starting
> >management server
> >javax.persistence.EntityExistsException: Entity already exists:
> >at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1318)
> >
> >This generally requires a drop database, redeploy etc - very time
> >consuming.
> >
> >I dug into the error a little, and the code behavior seems a little
> >strange.
> >
> >See com.cloud.server.ConfigurationServerImpl.java,
> >createDefaultNetworks(),
> >line 1085:
> >
> >long id = 1;
> >
> >And line 1141:
> >_networkDao.persist(network, false,
> >getServicesAndProvidersForNetwork(networkOfferingId));
> >id++;
> >
> >The code is trying to insert rows into the networks table, with "id"s
> >starting at 1. However, the existing values in the networks table also
> >start at 1, giving a primary key exception and causing the management
> >server not to start successfully.
> >
> >Is the networks table supposed to always be empty at startup so that these
> >inserts will always succeed even though they are setting id to 1 blindly?
> >Or is the code wrong to be starting IDs at 1 and failing on duplicate key
> >exceptions?
> >
> >Thanks,
> >Dave.
> >
> >
> >
> >*Info on my setup:*
> >Running CloudStack from master (4.1.0-SNAPSHOT).
> >
> >Setup process for the management server was:
> >
> >Grab latest from git repo, then:
> >
> >mvn install
> >mvn -P developer -pl developer -Ddeploydb
> >export MAVEN_OPTS="-Xmx1024m -Xdebug
> >-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
> >mvn -pl :cloud-client-ui jetty:run
> >
>
>
>
--
Thanks,
Dave.
Re: EntityExistsException on management server startup
(createDefaultNetworks)
Posted by Alena Prokharchyk <Al...@citrix.com>.
Dave,
This code - default network creation - is initiated only when the Global
config value "init" is false, as a part of persisting the default
cloudStack values. Init is false at the very first management server
startup, and it's reset to true after the default values are persisted.
Looks like in your setup init process was interrupted in the middle of
the execution, so init was never reset to true, but some default values
were persisted and we never rolled them back. There are 2 bugs to be fixed
in this area:
1) ConfigurationServerImpl, persistDefaultValues method
Wrap all the DB updates in "if (init == null || init.equals("false")) {}"
in the single transaction. So we don't up with partially updated DB.
2) We should remove call to createDefaultNetworks from
persistDefaultValues(). This call was added long ago to support the
scenario when:
* developer manually inserts zone entry to the cloud.data_center table
before the very first management server startup. We used to have the dev
helper script that read data center and other config info from
server-setup.xml file, and did DB update.
* so when the management server starts up initially, the code finds the
zone, and creates the default networks for it.
As cloudStack customers and most of the cloudStack developers create the
zone using API after the management server startup (and not by using
manual db update prior to startup), the call to "createDefaultNetworks"
should become obsolete.
To fix it on your setup, you can either re-deploy the DB - safer solution
as we don't know which default values were persisted, and what was
dropped; or set init=true and restart management server. Also if you
re-deploy the DB, check if you see any exceptions in the log while
cloudStack persisting the default values (the log snippet should begin
with "ConfigurationServer is saving default values to the database")
-Alena.
On 11/2/12 1:26 AM, "Dave Cahill" <dc...@midokura.jp> wrote:
>Hi,
>
>Over the past few days, I've run into EntityExistsException several times
>when I shut down and restart my management server:
>
>ERROR [cloud.servlet.CloudStartupServlet] (main:) Exception starting
>management server
>javax.persistence.EntityExistsException: Entity already exists:
>at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1318)
>
>This generally requires a drop database, redeploy etc - very time
>consuming.
>
>I dug into the error a little, and the code behavior seems a little
>strange.
>
>See com.cloud.server.ConfigurationServerImpl.java,
>createDefaultNetworks(),
>line 1085:
>
>long id = 1;
>
>And line 1141:
>_networkDao.persist(network, false,
>getServicesAndProvidersForNetwork(networkOfferingId));
>id++;
>
>The code is trying to insert rows into the networks table, with "id"s
>starting at 1. However, the existing values in the networks table also
>start at 1, giving a primary key exception and causing the management
>server not to start successfully.
>
>Is the networks table supposed to always be empty at startup so that these
>inserts will always succeed even though they are setting id to 1 blindly?
>Or is the code wrong to be starting IDs at 1 and failing on duplicate key
>exceptions?
>
>Thanks,
>Dave.
>
>
>
>*Info on my setup:*
>Running CloudStack from master (4.1.0-SNAPSHOT).
>
>Setup process for the management server was:
>
>Grab latest from git repo, then:
>
>mvn install
>mvn -P developer -pl developer -Ddeploydb
>export MAVEN_OPTS="-Xmx1024m -Xdebug
>-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
>mvn -pl :cloud-client-ui jetty:run
>