You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Rohit Yadav <bh...@apache.org> on 2013/02/11 11:15:13 UTC

[DISCUSS] Fixing cloud-setup-databases

DatabaseCreator was introduce for 4.1 so both maven deploydb target
and cloud-setup-databases would use that.
Following does not run on maven's deploydb:
cloudbridge_db.sql
schema-level.sql
server-setup.sql

Following runs only from maven's deploydb:
com.cloud.upgrade.DatabaseUpgradeChecker

Following runs only from cloud-setup-databases:
com.cloud.test.DatabaseConfig

Pl. advise if we need to run those as part of maven's deploydb, or
cloud-setup-database or both?

Regards.

Re: [DISCUSS] Fixing cloud-setup-databases

Posted by Rohit Yadav <ro...@citrix.com>.

I'm online, in another meeting but we can discuss. It's 10:43AM here,
must be 9:13PST.
Regards.

On Thu, Feb 14, 2013 at 6:46 AM, Alex Huang <Al...@citrix.com> wrote:
> Rohit,
>
> I've actually been trying to reach you on irc regarding this.  Let's see if we can hook up tonight to get this done.  I should be on around 9p.m. pst.
>
> --Alex
>
>> -----Original Message-----
>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On Behalf
>> Of Rohit Yadav
>> Sent: Wednesday, February 13, 2013 1:25 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>>
>> Unassigned CLOUDSTACK-1019 to be picked up by someone else.
>> I'm not sure how to do it correctly, picked up other things to work on.
>>
>> TODO includes from the discussion on this thread:
>> - Fix cloudstack-setup-databases to use DatabaseCreator and to do that
>> we require configuring packaging so a classpath is expanded and used
>> to call DatabaseCreator
>> - Refactor and move out logic that does upgrades from mgmt server to
>> databasecreator.
>>
>> Regards.
>>
>> On Tue, Feb 12, 2013 at 5:02 PM, Abhinandan Prateek
>> <Ab...@citrix.com> wrote:
>> > We can discuss this one ?
>> >
>> > On 12/02/13 3:46 PM, "Rohit Yadav" <bh...@apache.org> wrote:
>> >
>> >>On Mon, Feb 11, 2013 at 11:45 PM, Alex Huang <Al...@citrix.com>
>> >>wrote:
>> >>> Rohit,
>> >>>
>> >>> The two should be one.  Having the two separate is a bad idea because
>> >>>it means developers is running something very different from the actual
>> >>>deployment version which means actual deployment version is not
>> tested
>> >>>until QA hits it.  The setup database script should give options and the
>> >>>target in maven should just have all of the options standardized for
>> >>>developer.
>> >>>
>> >>> The other thing is you should announce this to the list with big
>> >>>headlines.  Starting 4.1, we no longer modify the create-schema.sql.
>> >>>Only modify the upgrade schemas.  The process should look like this and
>> >>>should be documented.
>> >>
>> >>+1
>> >>
>> >>> - For every major release (first dot release), we need to roll-up all
>> >>>of the changes in the schema and produce a create-schema.sql.  The
>> >>>version recorded inside the create-schema.sql is the version of the
>> >>>major release.  The steps to do this should be recorded in a wiki for
>> >>>the release manager to use.
>> >>
>> >>Okay will start a wiki on this. I just want to say that I don't know
>> >>how to fix it, I don't understand the requirements clearly but will
>> >>try my best to fix it.
>> >>
>> >>> - Every release from this major release, we always create the database
>> >>>from create-schema.sql and then run the upgrades.  The upgrades are
>> >>>exactly the same as what the management server would do to upgrade
>> from
>> >>>existing version to current.
>> >>>
>> >>> We should discuss the following things:
>> >>> - Management Server no longer auto-upgrades.  It brings more
>> problems
>> >>>than it solves.
>> >>
>> >>Alright, so there are a lot of Upgrade versionx-versiony classes which
>> >>are used by DatabaseUpgradeChecker. I did not write DatabaseCreator in
>> >>python because I wanted to skip rewriting logic for these upgrade
>> >>paths again. Do we want to keep it that way or should I start porting
>> >>them to Python?
>> >>
>> >>>  It should just detect that the version in the database is different
>> >>>than its software version and exit.  We can provide a flag to override
>> >>>that function.  I just didn't think this through when I designed it but
>> >>>the good thing is it's all self-contained in a plugin so it's not
>> >>>difficult to change it.
>> >>
>> >>Instead of management server calling DatabaseUpgradeChecker,
>> >>DatabaseCreator should call it. The upgrade path does not work in
>> >>DatabaseCreator because I'll have to clean the spring stuff and get
>> >>upgrading logic out of mgmt server if we don't want mgmt server to do
>> >>it.
>> >>
>> >>> - DatabaseCreator should take care of both upgrade situations and
>> >>>initial schema creation (which I believe you did already) and upgrade
>> >>>cleanup (not sure if you did this part) after all management servers
>> >>>have been upgraded.
>> >>>
>> >>
>> >>Yes, the initial schema creation work and no the upgrade and cleanup
>> >>does not work because they need some refactoring and mgmt server in
>> >>that case does not need to call DatabaseUpgradeChecker.
>> >>
>> >>> This means
>> >>> - Developers don't write schemas in two places.  It's always written in
>> >>>the upgrade files.
>> >>
>> >>+1
>> >>
>> >>> - Upgrade process for the admin is multi-steps.
>> >>
>> >>So, do we require sysadmin to run DatabaseCreator from version to
>> >>version or should n't it just automatically upgrade to latest version?
>> >>
>> >>>         - Run DatabaseCreator (may be in a script) to do pre-upgrade
>> >>>and as part of this step should be the script to dump the database in
>> >>>case problems happen.
>> >>>         - Upgrade all management server
>> >>>         - Run DatabaseCreator (again maybe in a script) to do
>> >>>post-upgrade cleanup.
>> >>> - Release manager produces the create-schema.sql for every major
>> >>>release.
>> >>>
>> >>> For 4.1, we need to get rid of the 4.1-new-schema.  I have no idea why
>> >>>that's created as it obviously doesn't fit into our upgrade strategy.
>> >>>Everything in that file should be moved into the DB upgrade scripts and
>> >>>that file removed in git and from the developer/pom.xml.  Also, the
>> >>>version in create-schema.sql needs to stay the same and not changed
>> with
>> >>>the release version any more.
>> >>
>> >>Alright.
>> >>
>> >>Regards.
>> >>
>> >>>
>> >>> --Alex
>> >>>
>> >>>> -----Original Message-----
>> >>>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On
>> Behalf
>> >>>> Of Rohit Yadav
>> >>>> Sent: Monday, February 11, 2013 9:13 AM
>> >>>> To: Hugo Trippaers
>> >>>> Cc: cloudstack-dev@incubator.apache.org
>> >>>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>> >>>>
>> >>>> Forgot to mention, one more reason why it was written in Java, as we
>> >>>> pass a database upgrade class which is suppose to do the upgrade,
>> >>>> tests etc. If we take the python route, these things can be python
>> >>>> modules instead of java classes to do the magic.
>> >>>>
>> >>>> Regards.
>> >>>>
>> >>>> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <bh...@apache.org>
>> >>>> wrote:
>> >>>> > Initially I thought to write it in python, which I can do it now as
>> >>>> > well. I just wanted to re-use some of the stuff that Transaction and
>> >>>> > ScriptRunner provides. Just let me know if it's a pain to package, we
>> >>>> > don't need the java one, besides it's really small and I can write a
>> >>>> > replacement in python (because it's available on most linux distros
>> >>>> > and osx, can write in any other language as well) in no time.
>> >>>> >
>> >>>> > Regards.
>> >>>> >
>> >>>> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers
>> >>>> > <HT...@schubergphilis.com> wrote:
>> >>>> >>
>> >>>> >>
>> >>>> >>> -----Original Message-----
>> >>>> >>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com]
>> On
>> >>>> Behalf
>> >>>> >>> Of Rohit Yadav
>> >>>> >>> Sent: Monday, February 11, 2013 5:49 PM
>> >>>> >>> To: Hugo Trippaers
>> >>>> >>> Cc: cloudstack-dev@incubator.apache.org
>> >>>> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>> >>>> >>>
>> >>>> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers
>> >>>> >>> <HT...@schubergphilis.com> wrote:
>> >>>> >>> > Hey Rohit,
>> >>>> >>> >
>> >>>> >>> > As far as I know these are two disctict use cases.
>> >>>> >>> >
>> >>>> >>> I agree, but only in few sets of sql files. The reasons why
>> >>>> DatabaseCreator
>> >>>> >>> was introduced:
>> >>>> >>>
>> >>>> >>> 0. Make life of sysadmins and developers easier 1. If passed
>> >>>>--database,
>> >>>> it
>> >>>> >>> would drop and recreate databases 2. Utility to run sql files,
>> >>>>based on
>> >>>> passwd
>> >>>> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader
>> >>>>class,
>> >>>> so
>> >>>> >>> we won't have to run custom upgrade path scripts
>> >>>> >>>
>> >>>> >>> > The maven deploy db is just for developers or at least people
>> >>>>who are
>> >>>> just
>> >>>> >>> working from source directly, and should provide a convenient way
>> >>>>for
>> >>>> >>> developers to refresh their test database or deploy a new schema.
>> >>>> >>>
>> >>>> >>> So, maven's deploydb and cloud-setup-databases (or
>> >>>>cloudstack-setup-
>> >>>> >>> databases now) take different routes. Channeling both of them
>> >>>>through
>> >>>> >>> DatabaseCreator would ensure same kind of database
>> >>>> deploying/creating
>> >>>> >>> environment for both developers and sysadmins, less bugs and
>> >>>>issues.
>> >>>> >>>
>> >>>> >>> It's not required that you've to pass the same set of sql files or
>> >>>> configuration
>> >>>> >>> in both cases, for example the developer-prefill or devcloud.sql
>> >>>>should
>> >>>> not
>> >>>> >>> be part of cloudstack-setup-databases.
>> >>>> >>>
>> >>>> >>> I saw three files which were not part of one of the use cases, so
>> >>>>I asked
>> >>>> why
>> >>>> >>> they are there and who's using them. I don't know so I ask
>> >>>> >>> :)
>> >>>> >>>
>> >>>> >>> > The cloud-setup-databases is a tool that is used the sysadmins
>> >>>>to setup
>> >>>> a
>> >>>> >>> database after they have deployed the packages. It has several
>> >>>> purposes,
>> >>>> >>> setup db.properties, configure encryption and load a database (if
>> >>>>the
>> >>>> >>> supplied database is empty).
>> >>>> >>>
>> >>>> >>> Yes, my point is as a developer I want to test/play around them as
>> >>>>well.
>> >>>> >>>
>> >>>> >>> >
>> >>>> >>> > In my view both should remain for these purposes, but they
>> >>>>should be
>> >>>> in-
>> >>>> >>> sync and also the upgrade sql script should be in sync with these
>> >>>>changes.
>> >>>> >>>
>> >>>> >>> You got it! Keeping them in sync is an issue, channeling them
>> >>>>through
>> >>>> >>> DatabaseCreator would make things little more maintainable. I also
>> >>>>see
>> >>>> >>> DatabaseCreator as a utility that can be perhaps used by plugins
>> >>>>and
>> >>>> external
>> >>>> >>> tools as well (distant future) to do magical stuff, reusable and
>> >>>> maintainable.
>> >>>> >>
>> >>>> >> Sounds good to me :-)  Anything that makes my life as an admin
>> >>>>easier
>> >>>> gets a +1. Downside it that it is another java tool that has
>> >>>>requirements and
>> >>>> dependencies and needs to be packaged to be use full. We are already
>> >>>> sneaking in some jasypt jars into /usr/share/java to not break the
>> >>>>existing
>> >>>> scripts. Having a separate java bases tool would probably require its
>> >>>>own set
>> >>>> of dependencies.
>> >>>> >>
>> >>>> >>>
>> >>>> >>> Thoughts, comments?
>> >>>> >>>
>> >>>> >>> Regards.
>> >>>> >>>
>> >>>> >>> >
>> >>>> >>> > Cheers,
>> >>>> >>> >
>> >>>> >>> > Hugo
>> >>>> >>> >
>> >>>> >>> >> -----Original Message-----
>> >>>> >>> >> From: rohityadav89@gmail.com
>> [mailto:rohityadav89@gmail.com] On
>> >>>> >>> >> Behalf Of Rohit Yadav
>> >>>> >>> >> Sent: Monday, February 11, 2013 11:15 AM
>> >>>> >>> >> To: cloudstack-dev@incubator.apache.org
>> >>>> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases
>> >>>> >>> >>
>> >>>> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb
>> >>>> target
>> >>>> >>> >> and cloud-setup-databases would use that.
>> >>>> >>> >> Following does not run on maven's deploydb:
>> >>>> >>> >> cloudbridge_db.sql
>> >>>> >>> >> schema-level.sql
>> >>>> >>> >> server-setup.sql
>> >>>> >>> >>
>> >>>> >>> >> Following runs only from maven's deploydb:
>> >>>> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker
>> >>>> >>> >>
>> >>>> >>> >> Following runs only from cloud-setup-databases:
>> >>>> >>> >> com.cloud.test.DatabaseConfig
>> >>>> >>> >>
>> >>>> >>> >> Pl. advise if we need to run those as part of maven's deploydb,
>> >>>>or
>> >>>> >>> >> cloud- setup-database or both?
>> >>>> >>> >>
>> >>>> >>> >> Regards.
>> >