You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Miguel Ferreira <MF...@schubergphilis.com> on 2014/03/10 16:05:58 UTC
[DISCUSS] Enabling databse upgrades on master branch
Hi all,
At Schuberg Philis we are interested in upgrading our running installation of ACS more frequently than the current release cycle.
To that end, we are working on tooling to detect potentially conflict introducing changes to the ACS database and upgrade software.
By conflict introducing change I mean a change to ACS that requires a clean database to start with. Thus, rendering a database of a running installation useless.
Once we can detect the changes that introduce conflicts, we will start to monitor them to better understand how to mitigate and possibly work around the conflicts.
One thing we can already foresee as problematic is the way the upgrade software (SQL scripts and Java classes) are being maintained. It is part of the current way-of-working to make all kinds of changes to the upgrade software in the master branch. Say, if a create statement was introduced in a SQL script in commit C1, then the same create statement might be changed in commit C2, or even further down the line. If we want to continuously upgrade our running installation, we would not be able to upgrade past C1 without at least losing some data. One possible way out of this problem is to add an alter statement in C2 instead of changing the create statement introduced in C1.
We are in favor of all changes to the upgrade software being made in such a way that they can be applied independently and incrementally. We do realize that this entails more effort from developers, but we also see the benefit of enabling continuous delivery: we basically get a shorter feedback loop on the quality of ACS in a real-world scenario.
We are making all tools related to this open source, so anyone that shares the same interest is welcome to join the effort.
Lastly, we would like to hear your opinions about the issue I described, as well as, the proposed solution and any other solutions you might come up with.
Kind regards,
Miguel Ferreira
Mission Critical Engineer
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com
+31 207506617
+31 610907106
_____________________
Re: [DISCUSS] Enabling databse upgrades on master branch
Posted by Rajani Karuturi <Ra...@citrix.com>.
both are under Open Source: Apache 2.0 License
I would also vote for liquibase as it xml driven(and database independent format) and it can output SQL to file(easier for db reviews).
But, as Miguel already said, our current setup is more close flywaydb and it may be easier to migrate to it.
~Rajani
On 12-Mar-2014, at 3:28 pm, Daan Hoogland <da...@gmail.com>> wrote:
H Miguel,
Are these tools free for each dev. This might not be a restriction but
would have my preference. what about redistributable parts, like
classes that are part of the engine they implement?
On Wed, Mar 12, 2014 at 10:41 AM, Miguel Ferreira
<MF...@schubergphilis.com>> wrote:
I did a quick scan of the two tools proposed by Rajani and here's what I found:
- Liquibase:
o With Liquibase we would need to specify the changes (in change sets) to the database (using either XML, YAML, JSON or SQL).
o Each change set would then be applied by the tool to a running DB , and that would entail schema changes as well as data migration.
o The data migration part seems especially simple, yet powerful.
o The tool seems to offer the rollback capability for free as long as the change sets are written using its own dialect. But if the change set is specified directly in SQL, there is the option to define (also in SQL) the rollback procedure.
- Flyaway
o Flyaway seems to be very much similar to the DB upgrade tool ACS currently has, although more elaborate and feature rich.
o It provides a java API, a command line tool, as well as maven, gradle, ant and SBT support.
o Compared with Liquibase it seems to me a more free-form-like approach (again, similar to what exists now in ACS)
o Incremental changes to the DB are programmed in Java and then applied by the tool.
o It supports upgrades to specific versions (say from 4.2 to 3.4 when the latest is 4.0 for example), as proposed by Alex Huang.
Both tools seem capable of improving the current DB upgrade procedure, but they both require the change I already proposed in the way ACS developers maintain DB related code. They both require that changes are made incrementally and independently of each other.
All-in-all I think Liquibase is a better tool for the job.
Cheers,
Miguel
From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
Sent: dinsdag 11 maart 2014 6:52
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Miguel Ferreira
Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
Right...I'll just quickly chime in on this, but Daan described the issue well.
We simply do not have a way to upgrade the DB (in an automated way) in a fine-grained enough fashion during development. This is a common problem on many projects...not just CloudStack.
Sometimes it's easy enough to perform a manual upgrade. In these cases, I would encourage developers to send out an e-mail with the [DB-CHANGE] tag and let others now how to perform such an upgrade. This will decrease the number of times people have to destroy and re-create their environments.
At some point, hopefully we can have Git send out an e-mail to notify people of schema changes. Then they will at least know not to fetch the latest until they are ready to deal with the consequences.
On Tue, Mar 11, 2014 at 4:18 AM, Daan Hoogland <da...@gmail.com>> wrote:
Alex,
That would only work in a dev environment, would it? so running maven
you could choose to upgrade your db. At the moment when your db is at
master, how would you decide you need to upgrade beyond your prior
commit up to the present one? Also, having pulled in new code, not
upgrading the db is not an option with our present way of working as
the dao object would throw all kinds of errors.
What Mike is seeing is due to the granularity of the upgrade being not
fine grained enough for his needs, not due to the absence of it, i
agree. Any one, developer or not that dares to upgrade more often the
by following the release schedule, thus helping us with valuable test
data, will run into this problem.
On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang <Al...@citrix.com>> wrote:
I'm fine with us moving toward other tools but I do want to point this out. What Mike and others are seeing is NOT due to CloudStack's schema upgrade procedure is not incremental but the fact that incremental upgrade is automatically performed when CloudStack management server is rebooted.
As I've said in other threads, we should just replace the automatic db upgrade in CloudStack with a check for the correct version and put in a pom step to upgrade the db so a developer can upgrade the db schema when they so chooses.
--Alex
-----Original Message-----
From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com<ma...@schubergphilis.com>]
Sent: Tuesday, March 11, 2014 2:58 AM
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Alex Huang
Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
Hi Rajani,
Indeed I see the overlap with the previous discussion that is taking place. I
actually tried to chip-in that discussion.
I do agree with you on introducing tools that can support the database
upgrade process, but meanwhile a small change in the way developers
maintain the database related code could already bring us very far.
As of now I'm working on a separate project with the following roadmap:
- detect potential conflicts (and collect data about them)
- learn from the data collected which conflicts can we resolve automatically,
which we cannot
- for the conflicts that we cannot resolve automatically, provide some
guidance to the operator on how proceed with the upgrade
I'm working on a repository that is not accessible from the outside of
Schuberg Philis, but the intention is to move it to github.com<http://github.com><http://github.com> ASAP.
I'm very interested in aligning efforts with the rest of the community, and I'm
available to help out in whatever is decided. Meanwhile, I will continue to
develop the ideas we have, and anyone interested in helping out with that is
very welcome.
If you have any ideas on how to collaborate, please let me know.
Cheers,
Miguel
-----Original Message-----
From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com<ma...@citrix.com>]
Sent: dinsdag 11 maart 2014 7:19
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Alex Huang
Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
Hi Miguel,
This is in-line with discussions related to db changes we are having at [1] and
[2]
I think it would be better to use existing tools like [liquibase] or [flyway]
instead of writing a new one. A good comparison of the both is available at
[3].
Also, how do we join the efforts? Is there any design doc? Are you working
on a separate branch or as separate project?
[1] http://markmail.org/message/r7wv36o356nolq7f
[2] http://markmail.org/message/wlua3l4o5shayidf
[liquibase] http://www.liquibase.org/
[flyway] http://flywaydb.org/
[3] http://stackoverflow.com/a/8489144/201514
~Rajani
On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
<MF...@schubergphilis.com>>>
wrote:
Hi all,
At Schuberg Philis we are interested in upgrading our running installation of
ACS more frequently than the current release cycle.
To that end, we are working on tooling to detect potentially conflict
introducing changes to the ACS database and upgrade software.
By conflict introducing change I mean a change to ACS that requires a clean
database to start with. Thus, rendering a database of a running installation
useless.
Once we can detect the changes that introduce conflicts, we will start to
monitor them to better understand how to mitigate and possibly work
around the conflicts.
One thing we can already foresee as problematic is the way the upgrade
software (SQL scripts and Java classes) are being maintained. It is part of the
current way-of-working to make all kinds of changes to the upgrade software
in the master branch. Say, if a create statement was introduced in a SQL
script in commit C1, then the same create statement might be changed in
commit C2, or even further down the line. If we want to continuously
upgrade our running installation, we would not be able to upgrade past C1
without at least losing some data. One possible way out of this problem is to
add an alter statement in C2 instead of changing the create statement
introduced in C1.
We are in favor of all changes to the upgrade software being made in such a
way that they can be applied independently and incrementally. We do realize
that this entails more effort from developers, but we also see the benefit of
enabling continuous delivery: we basically get a shorter feedback loop on the
quality of ACS in a real-world scenario.
We are making all tools related to this open source, so anyone that shares
the same interest is welcome to join the effort.
Lastly, we would like to hear your opinions about the issue I described, as
well as, the proposed solution and any other solutions you might come up
with.
Kind regards,
Miguel Ferreira
Mission Critical Engineer
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com<http://schubergphilis.com><http://schubergphilis.com><http://schubergphilis.com>
+31 207506617<tel:%2B31%20207506617>
+31 610907106<tel:%2B31%20610907106>
_____________________
--
Daan
--
Mike Tutkowski
Senior CloudStack Developer, SolidFire Inc.
e: mike.tutkowski@solidfire.com<ma...@solidfire.com>
o: 303.746.7302
Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>(tm)
--
Daan
Re: [DISCUSS] Enabling databse upgrades on master branch
Posted by Daan Hoogland <da...@gmail.com>.
H Miguel,
Are these tools free for each dev. This might not be a restriction but
would have my preference. what about redistributable parts, like
classes that are part of the engine they implement?
On Wed, Mar 12, 2014 at 10:41 AM, Miguel Ferreira
<MF...@schubergphilis.com> wrote:
> I did a quick scan of the two tools proposed by Rajani and here's what I found:
>
>
> - Liquibase:
>
> o With Liquibase we would need to specify the changes (in change sets) to the database (using either XML, YAML, JSON or SQL).
>
> o Each change set would then be applied by the tool to a running DB , and that would entail schema changes as well as data migration.
>
> o The data migration part seems especially simple, yet powerful.
>
> o The tool seems to offer the rollback capability for free as long as the change sets are written using its own dialect. But if the change set is specified directly in SQL, there is the option to define (also in SQL) the rollback procedure.
>
> - Flyaway
>
> o Flyaway seems to be very much similar to the DB upgrade tool ACS currently has, although more elaborate and feature rich.
>
> o It provides a java API, a command line tool, as well as maven, gradle, ant and SBT support.
>
> o Compared with Liquibase it seems to me a more free-form-like approach (again, similar to what exists now in ACS)
>
> o Incremental changes to the DB are programmed in Java and then applied by the tool.
>
> o It supports upgrades to specific versions (say from 4.2 to 3.4 when the latest is 4.0 for example), as proposed by Alex Huang.
>
> Both tools seem capable of improving the current DB upgrade procedure, but they both require the change I already proposed in the way ACS developers maintain DB related code. They both require that changes are made incrementally and independently of each other.
>
> All-in-all I think Liquibase is a better tool for the job.
>
> Cheers,
> Miguel
>
> From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> Sent: dinsdag 11 maart 2014 6:52
> To: dev@cloudstack.apache.org
> Cc: Miguel Ferreira
> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
>
> Right...I'll just quickly chime in on this, but Daan described the issue well.
>
> We simply do not have a way to upgrade the DB (in an automated way) in a fine-grained enough fashion during development. This is a common problem on many projects...not just CloudStack.
>
> Sometimes it's easy enough to perform a manual upgrade. In these cases, I would encourage developers to send out an e-mail with the [DB-CHANGE] tag and let others now how to perform such an upgrade. This will decrease the number of times people have to destroy and re-create their environments.
>
>
> At some point, hopefully we can have Git send out an e-mail to notify people of schema changes. Then they will at least know not to fetch the latest until they are ready to deal with the consequences.
>
> On Tue, Mar 11, 2014 at 4:18 AM, Daan Hoogland <da...@gmail.com>> wrote:
> Alex,
>
> That would only work in a dev environment, would it? so running maven
> you could choose to upgrade your db. At the moment when your db is at
> master, how would you decide you need to upgrade beyond your prior
> commit up to the present one? Also, having pulled in new code, not
> upgrading the db is not an option with our present way of working as
> the dao object would throw all kinds of errors.
>
> What Mike is seeing is due to the granularity of the upgrade being not
> fine grained enough for his needs, not due to the absence of it, i
> agree. Any one, developer or not that dares to upgrade more often the
> by following the release schedule, thus helping us with valuable test
> data, will run into this problem.
>
>
> On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang <Al...@citrix.com>> wrote:
>> I'm fine with us moving toward other tools but I do want to point this out. What Mike and others are seeing is NOT due to CloudStack's schema upgrade procedure is not incremental but the fact that incremental upgrade is automatically performed when CloudStack management server is rebooted.
>>
>> As I've said in other threads, we should just replace the automatic db upgrade in CloudStack with a check for the correct version and put in a pom step to upgrade the db so a developer can upgrade the db schema when they so chooses.
>>
>> --Alex
>>
>>> -----Original Message-----
>>> From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com<ma...@schubergphilis.com>]
>>> Sent: Tuesday, March 11, 2014 2:58 AM
>>> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>> Cc: Alex Huang
>>> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
>>>
>>> Hi Rajani,
>>>
>>> Indeed I see the overlap with the previous discussion that is taking place. I
>>> actually tried to chip-in that discussion.
>>> I do agree with you on introducing tools that can support the database
>>> upgrade process, but meanwhile a small change in the way developers
>>> maintain the database related code could already bring us very far.
>>>
>>> As of now I'm working on a separate project with the following roadmap:
>>> - detect potential conflicts (and collect data about them)
>>> - learn from the data collected which conflicts can we resolve automatically,
>>> which we cannot
>>> - for the conflicts that we cannot resolve automatically, provide some
>>> guidance to the operator on how proceed with the upgrade
>>>
>>> I'm working on a repository that is not accessible from the outside of
>>> Schuberg Philis, but the intention is to move it to github.com<http://github.com> ASAP.
>>>
>>> I'm very interested in aligning efforts with the rest of the community, and I'm
>>> available to help out in whatever is decided. Meanwhile, I will continue to
>>> develop the ideas we have, and anyone interested in helping out with that is
>>> very welcome.
>>>
>>> If you have any ideas on how to collaborate, please let me know.
>>>
>>> Cheers,
>>> Miguel
>>>
>>> -----Original Message-----
>>> From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com<ma...@citrix.com>]
>>> Sent: dinsdag 11 maart 2014 7:19
>>> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>> Cc: Alex Huang
>>> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
>>>
>>> Hi Miguel,
>>>
>>> This is in-line with discussions related to db changes we are having at [1] and
>>> [2]
>>>
>>> I think it would be better to use existing tools like [liquibase] or [flyway]
>>> instead of writing a new one. A good comparison of the both is available at
>>> [3].
>>>
>>> Also, how do we join the efforts? Is there any design doc? Are you working
>>> on a separate branch or as separate project?
>>>
>>>
>>> [1] http://markmail.org/message/r7wv36o356nolq7f
>>> [2] http://markmail.org/message/wlua3l4o5shayidf
>>> [liquibase] http://www.liquibase.org/
>>> [flyway] http://flywaydb.org/
>>> [3] http://stackoverflow.com/a/8489144/201514
>>>
>>>
>>> ~Rajani
>>>
>>>
>>>
>>> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
>>> <MF...@schubergphilis.com>>>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> At Schuberg Philis we are interested in upgrading our running installation of
>>> ACS more frequently than the current release cycle.
>>> To that end, we are working on tooling to detect potentially conflict
>>> introducing changes to the ACS database and upgrade software.
>>> By conflict introducing change I mean a change to ACS that requires a clean
>>> database to start with. Thus, rendering a database of a running installation
>>> useless.
>>> Once we can detect the changes that introduce conflicts, we will start to
>>> monitor them to better understand how to mitigate and possibly work
>>> around the conflicts.
>>>
>>> One thing we can already foresee as problematic is the way the upgrade
>>> software (SQL scripts and Java classes) are being maintained. It is part of the
>>> current way-of-working to make all kinds of changes to the upgrade software
>>> in the master branch. Say, if a create statement was introduced in a SQL
>>> script in commit C1, then the same create statement might be changed in
>>> commit C2, or even further down the line. If we want to continuously
>>> upgrade our running installation, we would not be able to upgrade past C1
>>> without at least losing some data. One possible way out of this problem is to
>>> add an alter statement in C2 instead of changing the create statement
>>> introduced in C1.
>>>
>>> We are in favor of all changes to the upgrade software being made in such a
>>> way that they can be applied independently and incrementally. We do realize
>>> that this entails more effort from developers, but we also see the benefit of
>>> enabling continuous delivery: we basically get a shorter feedback loop on the
>>> quality of ACS in a real-world scenario.
>>>
>>> We are making all tools related to this open source, so anyone that shares
>>> the same interest is welcome to join the effort.
>>> Lastly, we would like to hear your opinions about the issue I described, as
>>> well as, the proposed solution and any other solutions you might come up
>>> with.
>>>
>>>
>>> Kind regards,
>>> Miguel Ferreira
>>>
>>> Mission Critical Engineer
>>> Schuberg Philis
>>> Boeingavenue 271
>>> 1119 PD Schiphol-Rijk
>>> schubergphilis.com<http://schubergphilis.com><http://schubergphilis.com>
>>>
>>> +31 207506617<tel:%2B31%20207506617>
>>> +31 610907106<tel:%2B31%20610907106>
>>> _____________________
>>>
>>
>
>
> --
> Daan
>
>
>
> --
> Mike Tutkowski
> Senior CloudStack Developer, SolidFire Inc.
> e: mike.tutkowski@solidfire.com<ma...@solidfire.com>
> o: 303.746.7302
> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>(tm)
--
Daan
RE: [DISCUSS] Enabling databse upgrades on master branch
Posted by Miguel Ferreira <MF...@schubergphilis.com>.
I did a quick scan of the two tools proposed by Rajani and here's what I found:
- Liquibase:
o With Liquibase we would need to specify the changes (in change sets) to the database (using either XML, YAML, JSON or SQL).
o Each change set would then be applied by the tool to a running DB , and that would entail schema changes as well as data migration.
o The data migration part seems especially simple, yet powerful.
o The tool seems to offer the rollback capability for free as long as the change sets are written using its own dialect. But if the change set is specified directly in SQL, there is the option to define (also in SQL) the rollback procedure.
- Flyaway
o Flyaway seems to be very much similar to the DB upgrade tool ACS currently has, although more elaborate and feature rich.
o It provides a java API, a command line tool, as well as maven, gradle, ant and SBT support.
o Compared with Liquibase it seems to me a more free-form-like approach (again, similar to what exists now in ACS)
o Incremental changes to the DB are programmed in Java and then applied by the tool.
o It supports upgrades to specific versions (say from 4.2 to 3.4 when the latest is 4.0 for example), as proposed by Alex Huang.
Both tools seem capable of improving the current DB upgrade procedure, but they both require the change I already proposed in the way ACS developers maintain DB related code. They both require that changes are made incrementally and independently of each other.
All-in-all I think Liquibase is a better tool for the job.
Cheers,
Miguel
From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
Sent: dinsdag 11 maart 2014 6:52
To: dev@cloudstack.apache.org
Cc: Miguel Ferreira
Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
Right...I'll just quickly chime in on this, but Daan described the issue well.
We simply do not have a way to upgrade the DB (in an automated way) in a fine-grained enough fashion during development. This is a common problem on many projects...not just CloudStack.
Sometimes it's easy enough to perform a manual upgrade. In these cases, I would encourage developers to send out an e-mail with the [DB-CHANGE] tag and let others now how to perform such an upgrade. This will decrease the number of times people have to destroy and re-create their environments.
At some point, hopefully we can have Git send out an e-mail to notify people of schema changes. Then they will at least know not to fetch the latest until they are ready to deal with the consequences.
On Tue, Mar 11, 2014 at 4:18 AM, Daan Hoogland <da...@gmail.com>> wrote:
Alex,
That would only work in a dev environment, would it? so running maven
you could choose to upgrade your db. At the moment when your db is at
master, how would you decide you need to upgrade beyond your prior
commit up to the present one? Also, having pulled in new code, not
upgrading the db is not an option with our present way of working as
the dao object would throw all kinds of errors.
What Mike is seeing is due to the granularity of the upgrade being not
fine grained enough for his needs, not due to the absence of it, i
agree. Any one, developer or not that dares to upgrade more often the
by following the release schedule, thus helping us with valuable test
data, will run into this problem.
On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang <Al...@citrix.com>> wrote:
> I'm fine with us moving toward other tools but I do want to point this out. What Mike and others are seeing is NOT due to CloudStack's schema upgrade procedure is not incremental but the fact that incremental upgrade is automatically performed when CloudStack management server is rebooted.
>
> As I've said in other threads, we should just replace the automatic db upgrade in CloudStack with a check for the correct version and put in a pom step to upgrade the db so a developer can upgrade the db schema when they so chooses.
>
> --Alex
>
>> -----Original Message-----
>> From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com<ma...@schubergphilis.com>]
>> Sent: Tuesday, March 11, 2014 2:58 AM
>> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Cc: Alex Huang
>> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
>>
>> Hi Rajani,
>>
>> Indeed I see the overlap with the previous discussion that is taking place. I
>> actually tried to chip-in that discussion.
>> I do agree with you on introducing tools that can support the database
>> upgrade process, but meanwhile a small change in the way developers
>> maintain the database related code could already bring us very far.
>>
>> As of now I'm working on a separate project with the following roadmap:
>> - detect potential conflicts (and collect data about them)
>> - learn from the data collected which conflicts can we resolve automatically,
>> which we cannot
>> - for the conflicts that we cannot resolve automatically, provide some
>> guidance to the operator on how proceed with the upgrade
>>
>> I'm working on a repository that is not accessible from the outside of
>> Schuberg Philis, but the intention is to move it to github.com<http://github.com> ASAP.
>>
>> I'm very interested in aligning efforts with the rest of the community, and I'm
>> available to help out in whatever is decided. Meanwhile, I will continue to
>> develop the ideas we have, and anyone interested in helping out with that is
>> very welcome.
>>
>> If you have any ideas on how to collaborate, please let me know.
>>
>> Cheers,
>> Miguel
>>
>> -----Original Message-----
>> From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com<ma...@citrix.com>]
>> Sent: dinsdag 11 maart 2014 7:19
>> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Cc: Alex Huang
>> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
>>
>> Hi Miguel,
>>
>> This is in-line with discussions related to db changes we are having at [1] and
>> [2]
>>
>> I think it would be better to use existing tools like [liquibase] or [flyway]
>> instead of writing a new one. A good comparison of the both is available at
>> [3].
>>
>> Also, how do we join the efforts? Is there any design doc? Are you working
>> on a separate branch or as separate project?
>>
>>
>> [1] http://markmail.org/message/r7wv36o356nolq7f
>> [2] http://markmail.org/message/wlua3l4o5shayidf
>> [liquibase] http://www.liquibase.org/
>> [flyway] http://flywaydb.org/
>> [3] http://stackoverflow.com/a/8489144/201514
>>
>>
>> ~Rajani
>>
>>
>>
>> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
>> <MF...@schubergphilis.com>>>
>> wrote:
>>
>> Hi all,
>>
>> At Schuberg Philis we are interested in upgrading our running installation of
>> ACS more frequently than the current release cycle.
>> To that end, we are working on tooling to detect potentially conflict
>> introducing changes to the ACS database and upgrade software.
>> By conflict introducing change I mean a change to ACS that requires a clean
>> database to start with. Thus, rendering a database of a running installation
>> useless.
>> Once we can detect the changes that introduce conflicts, we will start to
>> monitor them to better understand how to mitigate and possibly work
>> around the conflicts.
>>
>> One thing we can already foresee as problematic is the way the upgrade
>> software (SQL scripts and Java classes) are being maintained. It is part of the
>> current way-of-working to make all kinds of changes to the upgrade software
>> in the master branch. Say, if a create statement was introduced in a SQL
>> script in commit C1, then the same create statement might be changed in
>> commit C2, or even further down the line. If we want to continuously
>> upgrade our running installation, we would not be able to upgrade past C1
>> without at least losing some data. One possible way out of this problem is to
>> add an alter statement in C2 instead of changing the create statement
>> introduced in C1.
>>
>> We are in favor of all changes to the upgrade software being made in such a
>> way that they can be applied independently and incrementally. We do realize
>> that this entails more effort from developers, but we also see the benefit of
>> enabling continuous delivery: we basically get a shorter feedback loop on the
>> quality of ACS in a real-world scenario.
>>
>> We are making all tools related to this open source, so anyone that shares
>> the same interest is welcome to join the effort.
>> Lastly, we would like to hear your opinions about the issue I described, as
>> well as, the proposed solution and any other solutions you might come up
>> with.
>>
>>
>> Kind regards,
>> Miguel Ferreira
>>
>> Mission Critical Engineer
>> Schuberg Philis
>> Boeingavenue 271
>> 1119 PD Schiphol-Rijk
>> schubergphilis.com<http://schubergphilis.com><http://schubergphilis.com>
>>
>> +31 207506617<tel:%2B31%20207506617>
>> +31 610907106<tel:%2B31%20610907106>
>> _____________________
>>
>
--
Daan
--
Mike Tutkowski
Senior CloudStack Developer, SolidFire Inc.
e: mike.tutkowski@solidfire.com<ma...@solidfire.com>
o: 303.746.7302
Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>(tm)
Re: [DISCUSS] Enabling databse upgrades on master branch
Posted by Mike Tutkowski <mi...@solidfire.com>.
Right...I'll just quickly chime in on this, but Daan described the issue
well.
We simply do not have a way to upgrade the DB (in an automated way) in a
fine-grained enough fashion during development. This is a common problem on
many projects...not just CloudStack.
Sometimes it's easy enough to perform a manual upgrade. In these cases, I
would encourage developers to send out an e-mail with the [DB-CHANGE] tag
and let others now how to perform such an upgrade. This will decrease the
number of times people have to destroy and re-create their environments.
At some point, hopefully we can have Git send out an e-mail to notify
people of schema changes. Then they will at least know not to fetch the
latest until they are ready to deal with the consequences.
On Tue, Mar 11, 2014 at 4:18 AM, Daan Hoogland <da...@gmail.com>wrote:
> Alex,
>
> That would only work in a dev environment, would it? so running maven
> you could choose to upgrade your db. At the moment when your db is at
> master, how would you decide you need to upgrade beyond your prior
> commit up to the present one? Also, having pulled in new code, not
> upgrading the db is not an option with our present way of working as
> the dao object would throw all kinds of errors.
>
> What Mike is seeing is due to the granularity of the upgrade being not
> fine grained enough for his needs, not due to the absence of it, i
> agree. Any one, developer or not that dares to upgrade more often the
> by following the release schedule, thus helping us with valuable test
> data, will run into this problem.
>
>
> On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang <Al...@citrix.com>
> wrote:
> > I'm fine with us moving toward other tools but I do want to point this
> out. What Mike and others are seeing is NOT due to CloudStack's schema
> upgrade procedure is not incremental but the fact that incremental upgrade
> is automatically performed when CloudStack management server is rebooted.
> >
> > As I've said in other threads, we should just replace the automatic db
> upgrade in CloudStack with a check for the correct version and put in a pom
> step to upgrade the db so a developer can upgrade the db schema when they
> so chooses.
> >
> > --Alex
> >
> >> -----Original Message-----
> >> From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com]
> >> Sent: Tuesday, March 11, 2014 2:58 AM
> >> To: dev@cloudstack.apache.org
> >> Cc: Alex Huang
> >> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
> >>
> >> Hi Rajani,
> >>
> >> Indeed I see the overlap with the previous discussion that is taking
> place. I
> >> actually tried to chip-in that discussion.
> >> I do agree with you on introducing tools that can support the database
> >> upgrade process, but meanwhile a small change in the way developers
> >> maintain the database related code could already bring us very far.
> >>
> >> As of now I'm working on a separate project with the following roadmap:
> >> - detect potential conflicts (and collect data about them)
> >> - learn from the data collected which conflicts can we resolve
> automatically,
> >> which we cannot
> >> - for the conflicts that we cannot resolve automatically, provide some
> >> guidance to the operator on how proceed with the upgrade
> >>
> >> I'm working on a repository that is not accessible from the outside of
> >> Schuberg Philis, but the intention is to move it to github.com ASAP.
> >>
> >> I'm very interested in aligning efforts with the rest of the community,
> and I'm
> >> available to help out in whatever is decided. Meanwhile, I will
> continue to
> >> develop the ideas we have, and anyone interested in helping out with
> that is
> >> very welcome.
> >>
> >> If you have any ideas on how to collaborate, please let me know.
> >>
> >> Cheers,
> >> Miguel
> >>
> >> -----Original Message-----
> >> From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com]
> >> Sent: dinsdag 11 maart 2014 7:19
> >> To: dev@cloudstack.apache.org
> >> Cc: Alex Huang
> >> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
> >>
> >> Hi Miguel,
> >>
> >> This is in-line with discussions related to db changes we are having at
> [1] and
> >> [2]
> >>
> >> I think it would be better to use existing tools like [liquibase] or
> [flyway]
> >> instead of writing a new one. A good comparison of the both is
> available at
> >> [3].
> >>
> >> Also, how do we join the efforts? Is there any design doc? Are you
> working
> >> on a separate branch or as separate project?
> >>
> >>
> >> [1] http://markmail.org/message/r7wv36o356nolq7f
> >> [2] http://markmail.org/message/wlua3l4o5shayidf
> >> [liquibase] http://www.liquibase.org/
> >> [flyway] http://flywaydb.org/
> >> [3] http://stackoverflow.com/a/8489144/201514
> >>
> >>
> >> ~Rajani
> >>
> >>
> >>
> >> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
> >> <MF...@schubergphilis.com>>
> >> wrote:
> >>
> >> Hi all,
> >>
> >> At Schuberg Philis we are interested in upgrading our running
> installation of
> >> ACS more frequently than the current release cycle.
> >> To that end, we are working on tooling to detect potentially conflict
> >> introducing changes to the ACS database and upgrade software.
> >> By conflict introducing change I mean a change to ACS that requires a
> clean
> >> database to start with. Thus, rendering a database of a running
> installation
> >> useless.
> >> Once we can detect the changes that introduce conflicts, we will start
> to
> >> monitor them to better understand how to mitigate and possibly work
> >> around the conflicts.
> >>
> >> One thing we can already foresee as problematic is the way the upgrade
> >> software (SQL scripts and Java classes) are being maintained. It is
> part of the
> >> current way-of-working to make all kinds of changes to the upgrade
> software
> >> in the master branch. Say, if a create statement was introduced in a SQL
> >> script in commit C1, then the same create statement might be changed in
> >> commit C2, or even further down the line. If we want to continuously
> >> upgrade our running installation, we would not be able to upgrade past
> C1
> >> without at least losing some data. One possible way out of this problem
> is to
> >> add an alter statement in C2 instead of changing the create statement
> >> introduced in C1.
> >>
> >> We are in favor of all changes to the upgrade software being made in
> such a
> >> way that they can be applied independently and incrementally. We do
> realize
> >> that this entails more effort from developers, but we also see the
> benefit of
> >> enabling continuous delivery: we basically get a shorter feedback loop
> on the
> >> quality of ACS in a real-world scenario.
> >>
> >> We are making all tools related to this open source, so anyone that
> shares
> >> the same interest is welcome to join the effort.
> >> Lastly, we would like to hear your opinions about the issue I
> described, as
> >> well as, the proposed solution and any other solutions you might come up
> >> with.
> >>
> >>
> >> Kind regards,
> >> Miguel Ferreira
> >>
> >> Mission Critical Engineer
> >> Schuberg Philis
> >> Boeingavenue 271
> >> 1119 PD Schiphol-Rijk
> >> schubergphilis.com<http://schubergphilis.com>
> >>
> >> +31 207506617
> >> +31 610907106
> >> _____________________
> >>
> >
>
>
>
> --
> Daan
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*(tm)*
Re: [DISCUSS] Enabling databse upgrades on master branch
Posted by Daan Hoogland <da...@gmail.com>.
Alex,
That would only work in a dev environment, would it? so running maven
you could choose to upgrade your db. At the moment when your db is at
master, how would you decide you need to upgrade beyond your prior
commit up to the present one? Also, having pulled in new code, not
upgrading the db is not an option with our present way of working as
the dao object would throw all kinds of errors.
What Mike is seeing is due to the granularity of the upgrade being not
fine grained enough for his needs, not due to the absence of it, i
agree. Any one, developer or not that dares to upgrade more often the
by following the release schedule, thus helping us with valuable test
data, will run into this problem.
On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang <Al...@citrix.com> wrote:
> I'm fine with us moving toward other tools but I do want to point this out. What Mike and others are seeing is NOT due to CloudStack's schema upgrade procedure is not incremental but the fact that incremental upgrade is automatically performed when CloudStack management server is rebooted.
>
> As I've said in other threads, we should just replace the automatic db upgrade in CloudStack with a check for the correct version and put in a pom step to upgrade the db so a developer can upgrade the db schema when they so chooses.
>
> --Alex
>
>> -----Original Message-----
>> From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com]
>> Sent: Tuesday, March 11, 2014 2:58 AM
>> To: dev@cloudstack.apache.org
>> Cc: Alex Huang
>> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
>>
>> Hi Rajani,
>>
>> Indeed I see the overlap with the previous discussion that is taking place. I
>> actually tried to chip-in that discussion.
>> I do agree with you on introducing tools that can support the database
>> upgrade process, but meanwhile a small change in the way developers
>> maintain the database related code could already bring us very far.
>>
>> As of now I'm working on a separate project with the following roadmap:
>> - detect potential conflicts (and collect data about them)
>> - learn from the data collected which conflicts can we resolve automatically,
>> which we cannot
>> - for the conflicts that we cannot resolve automatically, provide some
>> guidance to the operator on how proceed with the upgrade
>>
>> I'm working on a repository that is not accessible from the outside of
>> Schuberg Philis, but the intention is to move it to github.com ASAP.
>>
>> I'm very interested in aligning efforts with the rest of the community, and I'm
>> available to help out in whatever is decided. Meanwhile, I will continue to
>> develop the ideas we have, and anyone interested in helping out with that is
>> very welcome.
>>
>> If you have any ideas on how to collaborate, please let me know.
>>
>> Cheers,
>> Miguel
>>
>> -----Original Message-----
>> From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com]
>> Sent: dinsdag 11 maart 2014 7:19
>> To: dev@cloudstack.apache.org
>> Cc: Alex Huang
>> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
>>
>> Hi Miguel,
>>
>> This is in-line with discussions related to db changes we are having at [1] and
>> [2]
>>
>> I think it would be better to use existing tools like [liquibase] or [flyway]
>> instead of writing a new one. A good comparison of the both is available at
>> [3].
>>
>> Also, how do we join the efforts? Is there any design doc? Are you working
>> on a separate branch or as separate project?
>>
>>
>> [1] http://markmail.org/message/r7wv36o356nolq7f
>> [2] http://markmail.org/message/wlua3l4o5shayidf
>> [liquibase] http://www.liquibase.org/
>> [flyway] http://flywaydb.org/
>> [3] http://stackoverflow.com/a/8489144/201514
>>
>>
>> ~Rajani
>>
>>
>>
>> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
>> <MF...@schubergphilis.com>>
>> wrote:
>>
>> Hi all,
>>
>> At Schuberg Philis we are interested in upgrading our running installation of
>> ACS more frequently than the current release cycle.
>> To that end, we are working on tooling to detect potentially conflict
>> introducing changes to the ACS database and upgrade software.
>> By conflict introducing change I mean a change to ACS that requires a clean
>> database to start with. Thus, rendering a database of a running installation
>> useless.
>> Once we can detect the changes that introduce conflicts, we will start to
>> monitor them to better understand how to mitigate and possibly work
>> around the conflicts.
>>
>> One thing we can already foresee as problematic is the way the upgrade
>> software (SQL scripts and Java classes) are being maintained. It is part of the
>> current way-of-working to make all kinds of changes to the upgrade software
>> in the master branch. Say, if a create statement was introduced in a SQL
>> script in commit C1, then the same create statement might be changed in
>> commit C2, or even further down the line. If we want to continuously
>> upgrade our running installation, we would not be able to upgrade past C1
>> without at least losing some data. One possible way out of this problem is to
>> add an alter statement in C2 instead of changing the create statement
>> introduced in C1.
>>
>> We are in favor of all changes to the upgrade software being made in such a
>> way that they can be applied independently and incrementally. We do realize
>> that this entails more effort from developers, but we also see the benefit of
>> enabling continuous delivery: we basically get a shorter feedback loop on the
>> quality of ACS in a real-world scenario.
>>
>> We are making all tools related to this open source, so anyone that shares
>> the same interest is welcome to join the effort.
>> Lastly, we would like to hear your opinions about the issue I described, as
>> well as, the proposed solution and any other solutions you might come up
>> with.
>>
>>
>> Kind regards,
>> Miguel Ferreira
>>
>> Mission Critical Engineer
>> Schuberg Philis
>> Boeingavenue 271
>> 1119 PD Schiphol-Rijk
>> schubergphilis.com<http://schubergphilis.com>
>>
>> +31 207506617
>> +31 610907106
>> _____________________
>>
>
--
Daan
RE: [DISCUSS] Enabling databse upgrades on master branch
Posted by Alex Huang <Al...@citrix.com>.
I'm fine with us moving toward other tools but I do want to point this out. What Mike and others are seeing is NOT due to CloudStack's schema upgrade procedure is not incremental but the fact that incremental upgrade is automatically performed when CloudStack management server is rebooted.
As I've said in other threads, we should just replace the automatic db upgrade in CloudStack with a check for the correct version and put in a pom step to upgrade the db so a developer can upgrade the db schema when they so chooses.
--Alex
> -----Original Message-----
> From: Miguel Ferreira [mailto:MFerreira@schubergphilis.com]
> Sent: Tuesday, March 11, 2014 2:58 AM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
>
> Hi Rajani,
>
> Indeed I see the overlap with the previous discussion that is taking place. I
> actually tried to chip-in that discussion.
> I do agree with you on introducing tools that can support the database
> upgrade process, but meanwhile a small change in the way developers
> maintain the database related code could already bring us very far.
>
> As of now I'm working on a separate project with the following roadmap:
> - detect potential conflicts (and collect data about them)
> - learn from the data collected which conflicts can we resolve automatically,
> which we cannot
> - for the conflicts that we cannot resolve automatically, provide some
> guidance to the operator on how proceed with the upgrade
>
> I'm working on a repository that is not accessible from the outside of
> Schuberg Philis, but the intention is to move it to github.com ASAP.
>
> I'm very interested in aligning efforts with the rest of the community, and I'm
> available to help out in whatever is decided. Meanwhile, I will continue to
> develop the ideas we have, and anyone interested in helping out with that is
> very welcome.
>
> If you have any ideas on how to collaborate, please let me know.
>
> Cheers,
> Miguel
>
> -----Original Message-----
> From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com]
> Sent: dinsdag 11 maart 2014 7:19
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
>
> Hi Miguel,
>
> This is in-line with discussions related to db changes we are having at [1] and
> [2]
>
> I think it would be better to use existing tools like [liquibase] or [flyway]
> instead of writing a new one. A good comparison of the both is available at
> [3].
>
> Also, how do we join the efforts? Is there any design doc? Are you working
> on a separate branch or as separate project?
>
>
> [1] http://markmail.org/message/r7wv36o356nolq7f
> [2] http://markmail.org/message/wlua3l4o5shayidf
> [liquibase] http://www.liquibase.org/
> [flyway] http://flywaydb.org/
> [3] http://stackoverflow.com/a/8489144/201514
>
>
> ~Rajani
>
>
>
> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira
> <MF...@schubergphilis.com>>
> wrote:
>
> Hi all,
>
> At Schuberg Philis we are interested in upgrading our running installation of
> ACS more frequently than the current release cycle.
> To that end, we are working on tooling to detect potentially conflict
> introducing changes to the ACS database and upgrade software.
> By conflict introducing change I mean a change to ACS that requires a clean
> database to start with. Thus, rendering a database of a running installation
> useless.
> Once we can detect the changes that introduce conflicts, we will start to
> monitor them to better understand how to mitigate and possibly work
> around the conflicts.
>
> One thing we can already foresee as problematic is the way the upgrade
> software (SQL scripts and Java classes) are being maintained. It is part of the
> current way-of-working to make all kinds of changes to the upgrade software
> in the master branch. Say, if a create statement was introduced in a SQL
> script in commit C1, then the same create statement might be changed in
> commit C2, or even further down the line. If we want to continuously
> upgrade our running installation, we would not be able to upgrade past C1
> without at least losing some data. One possible way out of this problem is to
> add an alter statement in C2 instead of changing the create statement
> introduced in C1.
>
> We are in favor of all changes to the upgrade software being made in such a
> way that they can be applied independently and incrementally. We do realize
> that this entails more effort from developers, but we also see the benefit of
> enabling continuous delivery: we basically get a shorter feedback loop on the
> quality of ACS in a real-world scenario.
>
> We are making all tools related to this open source, so anyone that shares
> the same interest is welcome to join the effort.
> Lastly, we would like to hear your opinions about the issue I described, as
> well as, the proposed solution and any other solutions you might come up
> with.
>
>
> Kind regards,
> Miguel Ferreira
>
> Mission Critical Engineer
> Schuberg Philis
> Boeingavenue 271
> 1119 PD Schiphol-Rijk
> schubergphilis.com<http://schubergphilis.com>
>
> +31 207506617
> +31 610907106
> _____________________
>
RE: [DISCUSS] Enabling databse upgrades on master branch
Posted by Miguel Ferreira <MF...@schubergphilis.com>.
Hi Rajani,
Indeed I see the overlap with the previous discussion that is taking place. I actually tried to chip-in that discussion.
I do agree with you on introducing tools that can support the database upgrade process, but meanwhile a small change in the way developers maintain the database related code could already bring us very far.
As of now I'm working on a separate project with the following roadmap:
- detect potential conflicts (and collect data about them)
- learn from the data collected which conflicts can we resolve automatically, which we cannot
- for the conflicts that we cannot resolve automatically, provide some guidance to the operator on how proceed with the upgrade
I'm working on a repository that is not accessible from the outside of Schuberg Philis, but the intention is to move it to github.com ASAP.
I'm very interested in aligning efforts with the rest of the community, and I'm available to help out in whatever is decided. Meanwhile, I will continue to develop the ideas we have, and anyone interested in helping out with that is very welcome.
If you have any ideas on how to collaborate, please let me know.
Cheers,
Miguel
-----Original Message-----
From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com]
Sent: dinsdag 11 maart 2014 7:19
To: dev@cloudstack.apache.org
Cc: Alex Huang
Subject: Re: [DISCUSS] Enabling databse upgrades on master branch
Hi Miguel,
This is in-line with discussions related to db changes we are having at [1] and [2]
I think it would be better to use existing tools like [liquibase] or [flyway] instead of writing a new one. A good comparison of the both is available at [3].
Also, how do we join the efforts? Is there any design doc? Are you working on a separate branch or as separate project?
[1] http://markmail.org/message/r7wv36o356nolq7f
[2] http://markmail.org/message/wlua3l4o5shayidf
[liquibase] http://www.liquibase.org/
[flyway] http://flywaydb.org/
[3] http://stackoverflow.com/a/8489144/201514
~Rajani
On 10-Mar-2014, at 8:35 pm, Miguel Ferreira <MF...@schubergphilis.com>> wrote:
Hi all,
At Schuberg Philis we are interested in upgrading our running installation of ACS more frequently than the current release cycle.
To that end, we are working on tooling to detect potentially conflict introducing changes to the ACS database and upgrade software.
By conflict introducing change I mean a change to ACS that requires a clean database to start with. Thus, rendering a database of a running installation useless.
Once we can detect the changes that introduce conflicts, we will start to monitor them to better understand how to mitigate and possibly work around the conflicts.
One thing we can already foresee as problematic is the way the upgrade software (SQL scripts and Java classes) are being maintained. It is part of the current way-of-working to make all kinds of changes to the upgrade software in the master branch. Say, if a create statement was introduced in a SQL script in commit C1, then the same create statement might be changed in commit C2, or even further down the line. If we want to continuously upgrade our running installation, we would not be able to upgrade past C1 without at least losing some data. One possible way out of this problem is to add an alter statement in C2 instead of changing the create statement introduced in C1.
We are in favor of all changes to the upgrade software being made in such a way that they can be applied independently and incrementally. We do realize that this entails more effort from developers, but we also see the benefit of enabling continuous delivery: we basically get a shorter feedback loop on the quality of ACS in a real-world scenario.
We are making all tools related to this open source, so anyone that shares the same interest is welcome to join the effort.
Lastly, we would like to hear your opinions about the issue I described, as well as, the proposed solution and any other solutions you might come up with.
Kind regards,
Miguel Ferreira
Mission Critical Engineer
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com<http://schubergphilis.com>
+31 207506617
+31 610907106
_____________________
Re: [DISCUSS] Enabling databse upgrades on master branch
Posted by Rajani Karuturi <Ra...@citrix.com>.
Hi Miguel,
This is in-line with discussions related to db changes we are having at [1] and [2]
I think it would be better to use existing tools like [liquibase] or [flyway] instead of writing a new one. A good comparison of the both is available at [3].
Also, how do we join the efforts? Is there any design doc? Are you working on a separate branch or as separate project?
[1] http://markmail.org/message/r7wv36o356nolq7f
[2] http://markmail.org/message/wlua3l4o5shayidf
[liquibase] http://www.liquibase.org/
[flyway] http://flywaydb.org/
[3] http://stackoverflow.com/a/8489144/201514
~Rajani
On 10-Mar-2014, at 8:35 pm, Miguel Ferreira <MF...@schubergphilis.com>> wrote:
Hi all,
At Schuberg Philis we are interested in upgrading our running installation of ACS more frequently than the current release cycle.
To that end, we are working on tooling to detect potentially conflict introducing changes to the ACS database and upgrade software.
By conflict introducing change I mean a change to ACS that requires a clean database to start with. Thus, rendering a database of a running installation useless.
Once we can detect the changes that introduce conflicts, we will start to monitor them to better understand how to mitigate and possibly work around the conflicts.
One thing we can already foresee as problematic is the way the upgrade software (SQL scripts and Java classes) are being maintained. It is part of the current way-of-working to make all kinds of changes to the upgrade software in the master branch. Say, if a create statement was introduced in a SQL script in commit C1, then the same create statement might be changed in commit C2, or even further down the line. If we want to continuously upgrade our running installation, we would not be able to upgrade past C1 without at least losing some data. One possible way out of this problem is to add an alter statement in C2 instead of changing the create statement introduced in C1.
We are in favor of all changes to the upgrade software being made in such a way that they can be applied independently and incrementally. We do realize that this entails more effort from developers, but we also see the benefit of enabling continuous delivery: we basically get a shorter feedback loop on the quality of ACS in a real-world scenario.
We are making all tools related to this open source, so anyone that shares the same interest is welcome to join the effort.
Lastly, we would like to hear your opinions about the issue I described, as well as, the proposed solution and any other solutions you might come up with.
Kind regards,
Miguel Ferreira
Mission Critical Engineer
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com<http://schubergphilis.com>
+31 207506617
+31 610907106
_____________________