You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Hugo Trippaers <HT...@schubergphilis.com> on 2012/10/02 02:00:16 UTC
Updates to maven build
Hey guys,
I'm working very hard to get the maven stuff working in the master branch. I'll try not to disrupt the existing ant builds of master, but sooner or later we need to go either left or right. For now I've updated the page https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven with instructions on how to use maven. If you feel there is information missing or not clear, let me know and I'll try to update it.
The short term goal is to have maven compile the code and build the following artifacts:
* Client.war (the management website including dependencies)
* Usage.jar (executable jar for the usage server including deps)
* Awsapi.war (awsapi website including deps)
* Systemvm.iso
* Agent.zip
Next to this:
* Developer support, deploydb, refreshdb, start debug server
If anyone wants to help feel free! Just testing, working with the maven build and sending feedback will help enormously.
I know this work is happening parallel to the 4.0 release, but I already planned to spend this week in the states with Edison to work on this. None of the work we do will affect the 4.0 branch anyway.
Cheers,
Hugo
Re: Updates to maven build
Posted by David Nalley <da...@gnsa.us>.
On Thu, Oct 4, 2012 at 12:55 PM, Chiradeep Vittal
<Ch...@citrix.com> wrote:
> This was discussed here: http://markmail.org/thread/l456jfyeypotz6kd
> TLDR; it was always the intention to use migrations but wasn't completed.
Is that something we should start enforcing now that we have branched 4.0?
--David
Re: Updates to maven build
Posted by Chiradeep Vittal <Ch...@citrix.com>.
This was discussed here: http://markmail.org/thread/l456jfyeypotz6kd
TLDR; it was always the intention to use migrations but wasn't completed.
On 10/3/12 10:19 PM, "Rohit Yadav" <ro...@citrix.com> wrote:
>Hi John,
>
>For database upgrade/migration, in the release version there are
>migration paths defined in the code; for example in setup/db/db/,
>schema-302to40* etc.
>Maven (deploydb etc.) is only used for development where I think we
>generally don't deal with migration, we usually take/do fresh
>setup/installation.
>
>Flyway sounds interesting, added on the TODOs for Maven; database
>migration during development, just in case someone needs such a feature:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven
>#BuildingwithMaven-TODOs
>
>Regards.
>
>On 03-Oct-2012, at 5:25 PM, John Burwell <jb...@basho.com> wrote:
>
>> Rohit,
>>
>> Has their been any consideration given to shifting a migrations model
>>for database schema management? In addition to providing deploydb
>>capability in Maven, it would also provide DRY schema management.
>>Migrations inherently support both clean database creation and arbitrary
>>schema upgrade from a single set of scripts. Currently, schema changes
>>must be put in both the create_schema.sql and appropriate release
>>upgrade scripts. I have used Flyway (http://code.google.com/p/flyway/)
>>in the past with great success. In addition providing Maven
>>integration, it also provides an API that could used by
>>DatabaseUpgradeChecker.
>>
>> Thanks,
>> -John
>>
>> On Oct 3, 2012, at 1:55 AM, Rohit Yadav <ro...@citrix.com> wrote:
>>
>>> Thanks Hugo!
>>>
>>> On 02-Oct-2012, at 5:30 AM, Hugo Trippaers
>>><ht...@schubergphilis.com> wrote:
>>>
>>>> Hey guys,
>>>>
>>>> I'm working very hard to get the maven stuff working in the master
>>>>branch. I'll try not to disrupt the existing ant builds of master, but
>>>>sooner or later we need to go either left or right. For now I've
>>>>updated the page
>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Ma
>>>>ven with instructions on how to use maven. If you feel there is
>>>>information missing or not clear, let me know and I'll try to update
>>>>it.
>>>>
>>>> The short term goal is to have maven compile the code and build the
>>>>following artifacts:
>>>>
>>>> * Client.war (the management website including dependencies)
>>>>
>>>> * Usage.jar (executable jar for the usage server including
>>>>deps)
>>>>
>>>> * Awsapi.war (awsapi website including deps)
>>>>
>>>> * Systemvm.iso
>>>>
>>>> * Agent.zip
>>>
>>> I'd added a TODOs sections for Maven:
>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Mav
>>>en#BuildingwithMaven-TODOs
>>> Let's update on that, to keep us updated asynchronously. And there are
>>>couple of bugs on JIRA as well.
>>>
>>>>
>>>> Next to this:
>>>>
>>>> * Developer support, deploydb, refreshdb, start debug server
>>>
>>> For the developer workflow, why not move the plugins to top level pom
>>>under a profile? Right now I see they are in a profile already?
>>>Although, if we were to move db properties, assemblies etc, moving
>>>stuff into the developer/ makes sense.
>>>
>>> I've some working in progress to use exec plugin to do some of the
>>>deploy db/server commands. After I deploy with ant, I was able to run
>>>the server using Jetty:
>>>
>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Mav
>>>en#BuildingwithMaven-Runningaserverfortest%2Fdebugpurposes.
>>>
>>> I'm still not able to configure/use the tomcat plugin correctly.
>>>
>>> Also if you could you fix your editor to insert spaces instead of tabs
>>>as it distorts for me on vim. 2 spaces/1tab for .xmls :D
>>>
>>>> If anyone wants to help feel free! Just testing, working with the
>>>>maven build and sending feedback will help enormously.
>>>
>>> +1
>>>
>>> Alright, can we now discuss the packaging strategy as well? Will we
>>>continue using waf+maven+ant hack or use maven plugins for packaging,
>>>or specs and control files for the purists? Deb packaging is working
>>>fine, just some resolvable issues. RPM was WIP, but I pre-empted that
>>>work after the discussion on IRC last week.
>>>
>>> Okay, here are the pros and cons I can think of if we use packaging
>>>using maven plugins; (in the following package means deb or rpms)
>>>
>>> Pros:
>>> - Each submodule/artifacts gets its own package
>>> - With this design, it's possible to develop, test and package each
>>>submodule individually
>>> - With this design, it's possible for a plugin developer and
>>>developers in general to just focus on their work and not be bothered
>>>by learning the build system, they can just drop in their code in
>>>plugins/ and not touch any other part of the source tree.
>>> - It's very fast, right now it takes 30-50 mins to test any
>>>build/source changes.
>>> - Variables in pom.xml can be imported dynamically in control/conf
>>>files using [[ ]] .
>>>
>>> Cons:
>>> - I was told on IRC that build system folks and purists prefer their
>>>specs, control files.
>>> - I'm still to figure out how to expand variables like @SYSDIR@ in
>>>several .in scripts which are expanded by waf now. Exec plugin is one
>>>option.
>>> (Purists will still require waf or something to expand vars in these
>>>.in scripts before they can call rpm build or debuild)
>>>
>>> Pl. add pros/cons you may think and let's decide based on build system
>>>maintainability, ease for developers.
>>>
>>>> I know this work is happening parallel to the 4.0 release, but I
>>>>already planned to spend this week in the states with Edison to work
>>>>on this. None of the work we do will affect the 4.0 branch anyway.
>>>
>>> +1 let's do this in parallel, without removing any ant build system
>>>parts for now, and do housekeeping only after we've a stable build
>>>system with maven. How about making rule that all build patches go
>>>though reviews, as build bugs become blockers.
>>>
>>> Regards.
>>>
>>>>
>>>> Cheers,
>>>>
>>>> Hugo
>>>
>>
>
Re: Updates to maven build
Posted by Rohit Yadav <ro...@citrix.com>.
Hi John,
For database upgrade/migration, in the release version there are migration paths defined in the code; for example in setup/db/db/, schema-302to40* etc.
Maven (deploydb etc.) is only used for development where I think we generally don't deal with migration, we usually take/do fresh setup/installation.
Flyway sounds interesting, added on the TODOs for Maven; database migration during development, just in case someone needs such a feature: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-TODOs
Regards.
On 03-Oct-2012, at 5:25 PM, John Burwell <jb...@basho.com> wrote:
> Rohit,
>
> Has their been any consideration given to shifting a migrations model for database schema management? In addition to providing deploydb capability in Maven, it would also provide DRY schema management. Migrations inherently support both clean database creation and arbitrary schema upgrade from a single set of scripts. Currently, schema changes must be put in both the create_schema.sql and appropriate release upgrade scripts. I have used Flyway (http://code.google.com/p/flyway/) in the past with great success. In addition providing Maven integration, it also provides an API that could used by DatabaseUpgradeChecker.
>
> Thanks,
> -John
>
> On Oct 3, 2012, at 1:55 AM, Rohit Yadav <ro...@citrix.com> wrote:
>
>> Thanks Hugo!
>>
>> On 02-Oct-2012, at 5:30 AM, Hugo Trippaers <ht...@schubergphilis.com> wrote:
>>
>>> Hey guys,
>>>
>>> I'm working very hard to get the maven stuff working in the master branch. I'll try not to disrupt the existing ant builds of master, but sooner or later we need to go either left or right. For now I've updated the page https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven with instructions on how to use maven. If you feel there is information missing or not clear, let me know and I'll try to update it.
>>>
>>> The short term goal is to have maven compile the code and build the following artifacts:
>>>
>>> * Client.war (the management website including dependencies)
>>>
>>> * Usage.jar (executable jar for the usage server including deps)
>>>
>>> * Awsapi.war (awsapi website including deps)
>>>
>>> * Systemvm.iso
>>>
>>> * Agent.zip
>>
>> I'd added a TODOs sections for Maven: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-TODOs
>> Let's update on that, to keep us updated asynchronously. And there are couple of bugs on JIRA as well.
>>
>>>
>>> Next to this:
>>>
>>> * Developer support, deploydb, refreshdb, start debug server
>>
>> For the developer workflow, why not move the plugins to top level pom under a profile? Right now I see they are in a profile already? Although, if we were to move db properties, assemblies etc, moving stuff into the developer/ makes sense.
>>
>> I've some working in progress to use exec plugin to do some of the deploy db/server commands. After I deploy with ant, I was able to run the server using Jetty:
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-Runningaserverfortest%2Fdebugpurposes.
>>
>> I'm still not able to configure/use the tomcat plugin correctly.
>>
>> Also if you could you fix your editor to insert spaces instead of tabs as it distorts for me on vim. 2 spaces/1tab for .xmls :D
>>
>>> If anyone wants to help feel free! Just testing, working with the maven build and sending feedback will help enormously.
>>
>> +1
>>
>> Alright, can we now discuss the packaging strategy as well? Will we continue using waf+maven+ant hack or use maven plugins for packaging, or specs and control files for the purists? Deb packaging is working fine, just some resolvable issues. RPM was WIP, but I pre-empted that work after the discussion on IRC last week.
>>
>> Okay, here are the pros and cons I can think of if we use packaging using maven plugins; (in the following package means deb or rpms)
>>
>> Pros:
>> - Each submodule/artifacts gets its own package
>> - With this design, it's possible to develop, test and package each submodule individually
>> - With this design, it's possible for a plugin developer and developers in general to just focus on their work and not be bothered by learning the build system, they can just drop in their code in plugins/ and not touch any other part of the source tree.
>> - It's very fast, right now it takes 30-50 mins to test any build/source changes.
>> - Variables in pom.xml can be imported dynamically in control/conf files using [[ ]] .
>>
>> Cons:
>> - I was told on IRC that build system folks and purists prefer their specs, control files.
>> - I'm still to figure out how to expand variables like @SYSDIR@ in several .in scripts which are expanded by waf now. Exec plugin is one option.
>> (Purists will still require waf or something to expand vars in these .in scripts before they can call rpm build or debuild)
>>
>> Pl. add pros/cons you may think and let's decide based on build system maintainability, ease for developers.
>>
>>> I know this work is happening parallel to the 4.0 release, but I already planned to spend this week in the states with Edison to work on this. None of the work we do will affect the 4.0 branch anyway.
>>
>> +1 let's do this in parallel, without removing any ant build system parts for now, and do housekeeping only after we've a stable build system with maven. How about making rule that all build patches go though reviews, as build bugs become blockers.
>>
>> Regards.
>>
>>>
>>> Cheers,
>>>
>>> Hugo
>>
>
Re: Updates to maven build
Posted by John Burwell <jb...@basho.com>.
Rohit,
Has their been any consideration given to shifting a migrations model for database schema management? In addition to providing deploydb capability in Maven, it would also provide DRY schema management. Migrations inherently support both clean database creation and arbitrary schema upgrade from a single set of scripts. Currently, schema changes must be put in both the create_schema.sql and appropriate release upgrade scripts. I have used Flyway (http://code.google.com/p/flyway/) in the past with great success. In addition providing Maven integration, it also provides an API that could used by DatabaseUpgradeChecker.
Thanks,
-John
On Oct 3, 2012, at 1:55 AM, Rohit Yadav <ro...@citrix.com> wrote:
> Thanks Hugo!
>
> On 02-Oct-2012, at 5:30 AM, Hugo Trippaers <ht...@schubergphilis.com> wrote:
>
>> Hey guys,
>>
>> I'm working very hard to get the maven stuff working in the master branch. I'll try not to disrupt the existing ant builds of master, but sooner or later we need to go either left or right. For now I've updated the page https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven with instructions on how to use maven. If you feel there is information missing or not clear, let me know and I'll try to update it.
>>
>> The short term goal is to have maven compile the code and build the following artifacts:
>>
>> * Client.war (the management website including dependencies)
>>
>> * Usage.jar (executable jar for the usage server including deps)
>>
>> * Awsapi.war (awsapi website including deps)
>>
>> * Systemvm.iso
>>
>> * Agent.zip
>
> I'd added a TODOs sections for Maven: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-TODOs
> Let's update on that, to keep us updated asynchronously. And there are couple of bugs on JIRA as well.
>
>>
>> Next to this:
>>
>> * Developer support, deploydb, refreshdb, start debug server
>
> For the developer workflow, why not move the plugins to top level pom under a profile? Right now I see they are in a profile already? Although, if we were to move db properties, assemblies etc, moving stuff into the developer/ makes sense.
>
> I've some working in progress to use exec plugin to do some of the deploy db/server commands. After I deploy with ant, I was able to run the server using Jetty:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-Runningaserverfortest%2Fdebugpurposes.
>
> I'm still not able to configure/use the tomcat plugin correctly.
>
> Also if you could you fix your editor to insert spaces instead of tabs as it distorts for me on vim. 2 spaces/1tab for .xmls :D
>
>> If anyone wants to help feel free! Just testing, working with the maven build and sending feedback will help enormously.
>
> +1
>
> Alright, can we now discuss the packaging strategy as well? Will we continue using waf+maven+ant hack or use maven plugins for packaging, or specs and control files for the purists? Deb packaging is working fine, just some resolvable issues. RPM was WIP, but I pre-empted that work after the discussion on IRC last week.
>
> Okay, here are the pros and cons I can think of if we use packaging using maven plugins; (in the following package means deb or rpms)
>
> Pros:
> - Each submodule/artifacts gets its own package
> - With this design, it's possible to develop, test and package each submodule individually
> - With this design, it's possible for a plugin developer and developers in general to just focus on their work and not be bothered by learning the build system, they can just drop in their code in plugins/ and not touch any other part of the source tree.
> - It's very fast, right now it takes 30-50 mins to test any build/source changes.
> - Variables in pom.xml can be imported dynamically in control/conf files using [[ ]] .
>
> Cons:
> - I was told on IRC that build system folks and purists prefer their specs, control files.
> - I'm still to figure out how to expand variables like @SYSDIR@ in several .in scripts which are expanded by waf now. Exec plugin is one option.
> (Purists will still require waf or something to expand vars in these .in scripts before they can call rpm build or debuild)
>
> Pl. add pros/cons you may think and let's decide based on build system maintainability, ease for developers.
>
>> I know this work is happening parallel to the 4.0 release, but I already planned to spend this week in the states with Edison to work on this. None of the work we do will affect the 4.0 branch anyway.
>
> +1 let's do this in parallel, without removing any ant build system parts for now, and do housekeeping only after we've a stable build system with maven. How about making rule that all build patches go though reviews, as build bugs become blockers.
>
> Regards.
>
>>
>> Cheers,
>>
>> Hugo
>
Re: Updates to maven build
Posted by Rohit Yadav <ro...@citrix.com>.
Thanks Hugo!
On 02-Oct-2012, at 5:30 AM, Hugo Trippaers <ht...@schubergphilis.com> wrote:
> Hey guys,
>
> I'm working very hard to get the maven stuff working in the master branch. I'll try not to disrupt the existing ant builds of master, but sooner or later we need to go either left or right. For now I've updated the page https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven with instructions on how to use maven. If you feel there is information missing or not clear, let me know and I'll try to update it.
>
> The short term goal is to have maven compile the code and build the following artifacts:
>
> * Client.war (the management website including dependencies)
>
> * Usage.jar (executable jar for the usage server including deps)
>
> * Awsapi.war (awsapi website including deps)
>
> * Systemvm.iso
>
> * Agent.zip
I'd added a TODOs sections for Maven: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-TODOs
Let's update on that, to keep us updated asynchronously. And there are couple of bugs on JIRA as well.
>
> Next to this:
>
> * Developer support, deploydb, refreshdb, start debug server
For the developer workflow, why not move the plugins to top level pom under a profile? Right now I see they are in a profile already? Although, if we were to move db properties, assemblies etc, moving stuff into the developer/ makes sense.
I've some working in progress to use exec plugin to do some of the deploy db/server commands. After I deploy with ant, I was able to run the server using Jetty:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven#BuildingwithMaven-Runningaserverfortest%2Fdebugpurposes.
I'm still not able to configure/use the tomcat plugin correctly.
Also if you could you fix your editor to insert spaces instead of tabs as it distorts for me on vim. 2 spaces/1tab for .xmls :D
> If anyone wants to help feel free! Just testing, working with the maven build and sending feedback will help enormously.
+1
Alright, can we now discuss the packaging strategy as well? Will we continue using waf+maven+ant hack or use maven plugins for packaging, or specs and control files for the purists? Deb packaging is working fine, just some resolvable issues. RPM was WIP, but I pre-empted that work after the discussion on IRC last week.
Okay, here are the pros and cons I can think of if we use packaging using maven plugins; (in the following package means deb or rpms)
Pros:
- Each submodule/artifacts gets its own package
- With this design, it's possible to develop, test and package each submodule individually
- With this design, it's possible for a plugin developer and developers in general to just focus on their work and not be bothered by learning the build system, they can just drop in their code in plugins/ and not touch any other part of the source tree.
- It's very fast, right now it takes 30-50 mins to test any build/source changes.
- Variables in pom.xml can be imported dynamically in control/conf files using [[ ]] .
Cons:
- I was told on IRC that build system folks and purists prefer their specs, control files.
- I'm still to figure out how to expand variables like @SYSDIR@ in several .in scripts which are expanded by waf now. Exec plugin is one option.
(Purists will still require waf or something to expand vars in these .in scripts before they can call rpm build or debuild)
Pl. add pros/cons you may think and let's decide based on build system maintainability, ease for developers.
> I know this work is happening parallel to the 4.0 release, but I already planned to spend this week in the states with Edison to work on this. None of the work we do will affect the 4.0 branch anyway.
+1 let's do this in parallel, without removing any ant build system parts for now, and do housekeeping only after we've a stable build system with maven. How about making rule that all build patches go though reviews, as build bugs become blockers.
Regards.
>
> Cheers,
>
> Hugo