Maven status (from my point of view)

[Hugo Trippaers <HT...@schubergphilis.com>]

Hey all,

The switch to maven is coming along nicely I think, however we are not there yet. With the dependencies gone from the tree, the maven build is now the easiest way to build CloudStack IMHO. All classes are properly compiled and stored in jar files and dependencies are automagically downloaded to make sure everything compiles. Over the last few days I've added packaging of the systemvm.iso to the maven files and I did some work on the awsapi. The awsapi is proving to be difficult as it has multiple results (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the other hand, maven has plugins to deal with axis2 and generate everything, so with a  bit of work we should be able to simplify this. Test code is not compiled in some cases as there are some interesting circular references (test code for server depending on vmware plugin comes to mind), that needs to be sorted if we want maven to be able to do its job.

I've spend the last two days to get packaging back into something that looks workable. Even though we want to move away from waf, it seemed like the best way to fix the packaging for now. With the exception of awsapi all RPM packages are being built on my dev environment so that is looking good. I don't have a debian environment yet, so I'm not able to test a debian build. I haven't done any testing on the rpms yet, just verified that they are build, hoping to do that tomorrow.

Couple of questions that I would like some feedback on:
* Maven includes the dependencies with their original filename, I believe we used to ship our deps with cloud- prepended to the name. Now we need to either check if our dependencies aren't already supplied by the OS or put them in a different directory, like /usr/share/java/cloud/*jar. What is the best route to take?
* Awsapi should be split in three sub projects (directories)  I think, the autogenerated ec2 part, the autogenerated e3 part and cloudstack own code which might be distributed as a webapp. 
* I've split the server rpm in two, one called server and contains all oss stuff and one called server-nonoss containing the modules that depend on non-oss libs, is that ok?

Darren, is this more or less in line with the stuff you are doing?

Feedback, ideas, complaints anyone?

Will send an update when there is something to report.

Cheers,

Hugo

RE: Maven status (from my point of view)

[Edison Su <Ed...@citrix.com>]

> -----Original Message-----
> From: Wido den Hollander [mailto:wido@widodh.nl]
> Sent: Tuesday, September 04, 2012 2:23 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Maven status (from my point of view)
> 
> 
> 
> On 09/04/2012 04:12 PM, Hugo Trippaers wrote:
> > Hey all,
> >
> > The switch to maven is coming along nicely I think, however we are
> not there yet. With the dependencies gone from the tree, the maven
> build is now the easiest way to build CloudStack IMHO. All classes are
> properly compiled and stored in jar files and dependencies are
> automagically downloaded to make sure everything compiles. Over the
> last few days I've added packaging of the systemvm.iso to the maven
> files and I did some work on the awsapi. The awsapi is proving to be
> difficult as it has multiple results (.mar, .aar, .jar files and a
> webapp). Maven doesn't really like this. On the other hand, maven has
> plugins to deal with axis2 and generate everything, so with a  bit of
> work we should be able to simplify this. Test code is not compiled in
> some cases as there are some interesting circular references (test code
> for server depending on vmware plugin comes to mind), that needs to be
> sorted if we want maven to be able to do its job.
> >
> > I've spend the last two days to get packaging back into something
> that looks workable. Even though we want to move away from waf, it
> seemed like the best way to fix the packaging for now. With the
> exception of awsapi all RPM packages are being built on my dev
> environment so that is looking good. I don't have a debian environment
> yet, so I'm not able to test a debian build. I haven't done any testing
> on the rpms yet, just verified that they are build, hoping to do that
> tomorrow.
> >
> > Couple of questions that I would like some feedback on:
> > * Maven includes the dependencies with their original filename, I
> believe we used to ship our deps with cloud- prepended to the name. Now
> we need to either check if our dependencies aren't already supplied by
> the OS or put them in a different directory, like
> /usr/share/java/cloud/*jar. What is the best route to take?
> 
> I'd not prefix cloud-, to me that hints towards a homebrew version of
> that file.

+1, 
> 
> Whenever a library/JAR is available through the package manager of that
> distribution I'm voting for depending on that with the RPM/DEB packages,
> that is one dependency resolved.
> 
> For example the MySQL connector is available under Ubuntu and Debian, I
> already made a commit for that [0].
> 
> Placing the rest in /usr/share/java/cloud is fine with me, that way we
> don't pollute the system.


It's hard to hack waf, I am trying to move the dep jar files to /usr/share/java/cloud, no clue how to do that.

> 
> Wido
> 
> [0]:
> https://git-wip-us.apache.org/repos/asf?p=incubator-
> cloudstack.git;a=commit;h=9064236879dc9f2f11538d891271545e1ff10e9c
> 
> > * Awsapi should be split in three sub projects (directories)  I think,
> the autogenerated ec2 part, the autogenerated e3 part and cloudstack
> own code which might be distributed as a webapp.
> > * I've split the server rpm in two, one called server and contains
> all oss stuff and one called server-nonoss containing the modules that
> depend on non-oss libs, is that ok?
> >
> > Darren, is this more or less in line with the stuff you are doing?
> >
> > Feedback, ideas, complaints anyone?
> >
> > Will send an update when there is something to report.
> >
> > Cheers,
> >
> > Hugo
> >

Re: Maven status (from my point of view)

[Wido den Hollander <wi...@widodh.nl>]

On 09/04/2012 04:12 PM, Hugo Trippaers wrote:
> Hey all,
>
> The switch to maven is coming along nicely I think, however we are not there yet. With the dependencies gone from the tree, the maven build is now the easiest way to build CloudStack IMHO. All classes are properly compiled and stored in jar files and dependencies are automagically downloaded to make sure everything compiles. Over the last few days I've added packaging of the systemvm.iso to the maven files and I did some work on the awsapi. The awsapi is proving to be difficult as it has multiple results (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the other hand, maven has plugins to deal with axis2 and generate everything, so with a  bit of work we should be able to simplify this. Test code is not compiled in some cases as there are some interesting circular references (test code for server depending on vmware plugin comes to mind), that needs to be sorted if we want maven to be able to do its job.
>
> I've spend the last two days to get packaging back into something that looks workable. Even though we want to move away from waf, it seemed like the best way to fix the packaging for now. With the exception of awsapi all RPM packages are being built on my dev environment so that is looking good. I don't have a debian environment yet, so I'm not able to test a debian build. I haven't done any testing on the rpms yet, just verified that they are build, hoping to do that tomorrow.
>
> Couple of questions that I would like some feedback on:
> * Maven includes the dependencies with their original filename, I believe we used to ship our deps with cloud- prepended to the name. Now we need to either check if our dependencies aren't already supplied by the OS or put them in a different directory, like /usr/share/java/cloud/*jar. What is the best route to take?

I'd not prefix cloud-, to me that hints towards a homebrew version of 
that file.

Whenever a library/JAR is available through the package manager of that 
distribution I'm voting for depending on that with the RPM/DEB packages, 
that is one dependency resolved.

For example the MySQL connector is available under Ubuntu and Debian, I 
already made a commit for that [0].

Placing the rest in /usr/share/java/cloud is fine with me, that way we 
don't pollute the system.

Wido

[0]: 
https://git-wip-us.apache.org/repos/asf?p=incubator-cloudstack.git;a=commit;h=9064236879dc9f2f11538d891271545e1ff10e9c

> * Awsapi should be split in three sub projects (directories)  I think, the autogenerated ec2 part, the autogenerated e3 part and cloudstack own code which might be distributed as a webapp.
> * I've split the server rpm in two, one called server and contains all oss stuff and one called server-nonoss containing the modules that depend on non-oss libs, is that ok?
>
> Darren, is this more or less in line with the stuff you are doing?
>
> Feedback, ideas, complaints anyone?
>
> Will send an update when there is something to report.
>
> Cheers,
>
> Hugo
>

Re: Maven status (from my point of view)

[Wido den Hollander <wi...@widodh.nl>]

On 09/04/2012 04:33 PM, Rohit Yadav wrote:
>
> On 04-Sep-2012, at 7:42 PM, Hugo Trippaers <HT...@schubergphilis.com> wrote:
>
>> Hey all,
>>
>> The switch to maven is coming along nicely I think, however we are not there yet. With the dependencies gone from the tree, the maven build is now the easiest way to build CloudStack IMHO. All classes are properly compiled and stored in jar files and dependencies are automagically downloaded to make sure everything compiles. Over the last few days I've added packaging of the systemvm.iso to the maven files and I did some work on the awsapi. The awsapi is proving to be difficult as it has multiple results (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the other hand, maven has plugins to deal with axis2 and generate everything, so with a  bit of work we should be able to simplify this. Test code is not compiled in some cases as there are some interesting circular references (test code for server depending on vmware plugin comes to mind), that needs to be sorted if we want maven to be able to do its job.
>>
>> I've spend the last two days to get packaging back into something that looks workable. Even though we want to move away from waf, it seemed like the best way to fix the packaging for now. With the exception of awsapi all RPM packages are being built on my dev environment so that is looking good. I don't have a debian environment yet, so I'm not able to test a debian build. I haven't done any testing on the rpms yet, just verified that they are build, hoping to do that tomorrow.
>
> Let's keep waf for sometime. But since ant is already broken :(
>
>> Couple of questions that I would like some feedback on:
>> * Maven includes the dependencies with their original filename, I believe we used to ship our deps with cloud- prepended to the name. Now we need to either check if our dependencies aren't already supplied by the OS or put them in a different directory, like /usr/share/java/cloud/*jar. What is the best route to take?
>
> Fixable with a custom maven exec? I think it's better to ship these jars with a cloud- prefix (if there is no licensing issue) or for distro specific releases (debs, rpms) have the dependencies added in the rules.
>
>> Feedback, ideas,
>
> I'm new to maven and I think many of us are too. Could you share with us simple commands on how to use the ol' ant commands deploydb, deploy-server and debug with maven. Do we have those goals running now, or are the wip? For me ant commands don't work anymore.
>
>> complaints anyone?
>
> For the past one week I'm unable to get my server running.
> Not towards any specific person, but can we please not commit partial changes into master at least?
>

I'm agreeing on this one. I know this work had to be done, but my 
experience with Maven is little and right now I'm stuck with trying to 
get building working again.

For me Maven is still a big unknown which I'm trying to get a hold off.

A working build system would be very welcome.

Wido

> Regards,
> Rohit
>
>>
>> Will send an update when there is something to report.
>>
>> Cheers,
>>
>> Hugo
>>
>

RE: Maven status (from my point of view)

[Hugo Trippaers <HT...@schubergphilis.com>]

> -----Original Message-----
> From: Rohit Yadav [mailto:rohit.yadav@citrix.com]
> Sent: Tuesday, September 04, 2012 4:34 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Maven status (from my point of view)
> 
> 
> On 04-Sep-2012, at 7:42 PM, Hugo Trippaers
> <HT...@schubergphilis.com> wrote:
> 
> > Hey all,
> >
> > The switch to maven is coming along nicely I think, however we are not
> there yet. With the dependencies gone from the tree, the maven build is
> now the easiest way to build CloudStack IMHO. All classes are properly
> compiled and stored in jar files and dependencies are automagically
> downloaded to make sure everything compiles. Over the last few days I've
> added packaging of the systemvm.iso to the maven files and I did some work
> on the awsapi. The awsapi is proving to be difficult as it has multiple results
> (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the
> other hand, maven has plugins to deal with axis2 and generate everything, so
> with a  bit of work we should be able to simplify this. Test code is not
> compiled in some cases as there are some interesting circular references
> (test code for server depending on vmware plugin comes to mind), that
> needs to be sorted if we want maven to be able to do its job.
> >
> > I've spend the last two days to get packaging back into something that
> looks workable. Even though we want to move away from waf, it seemed
> like the best way to fix the packaging for now. With the exception of awsapi
> all RPM packages are being built on my dev environment so that is looking
> good. I don't have a debian environment yet, so I'm not able to test a debian
> build. I haven't done any testing on the rpms yet, just verified that they are
> build, hoping to do that tomorrow.
> 
> Let's keep waf for sometime. But since ant is already broken :(

There are a couple of issues with ant now. The main issue is that the dependencies have been removed from the source tree, which is a good thing. However it means that you will have to get them yourself. See my mail on the mailing list on how to do that using maven. The debug server is broken now because the mysql libraries are not copied to the lib directory anymore because that lib is also gone. Put it in the webapp lib folder and everything should work, but I did not test this.

> 
> > Couple of questions that I would like some feedback on:
> > * Maven includes the dependencies with their original filename, I believe
> we used to ship our deps with cloud- prepended to the name. Now we need
> to either check if our dependencies aren't already supplied by the OS or put
> them in a different directory, like /usr/share/java/cloud/*jar. What is the
> best route to take?
> 
> Fixable with a custom maven exec? I think it's better to ship these jars with a
> cloud- prefix (if there is no licensing issue) or for distro specific releases
> (debs, rpms) have the dependencies added in the rules.
> 
> > Feedback, ideas,
> 
> I'm new to maven and I think many of us are too. Could you share with us
> simple commands on how to use the ol' ant commands deploydb, deploy-
> server and debug with maven. Do we have those goals running now, or are
> the wip? For me ant commands don't work anymore.

Good point, I've consolidated some info that was posted to the dev list and updated the wiki page here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Building+with+Maven

> 
> > complaints anyone?
> 
> For the past one week I'm unable to get my server running.
> Not towards any specific person, but can we please not commit partial
> changes into master at least?

See the thread "[DISCUSS] Build process/binaries in our tree/candidate to begin testing with" for the how and why of this change. I agree it's a pita now, but hopefully it means we can ship version 4.0.0 when we fix this.

> 
> Regards,
> Rohit
> 
> >
> > Will send an update when there is something to report.
> >
> > Cheers,
> >
> > Hugo
> >

Re: Maven status (from my point of view)

[Rohit Yadav <ro...@citrix.com>]

On 04-Sep-2012, at 7:42 PM, Hugo Trippaers <HT...@schubergphilis.com> wrote:

> Hey all,
> 
> The switch to maven is coming along nicely I think, however we are not there yet. With the dependencies gone from the tree, the maven build is now the easiest way to build CloudStack IMHO. All classes are properly compiled and stored in jar files and dependencies are automagically downloaded to make sure everything compiles. Over the last few days I've added packaging of the systemvm.iso to the maven files and I did some work on the awsapi. The awsapi is proving to be difficult as it has multiple results (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the other hand, maven has plugins to deal with axis2 and generate everything, so with a  bit of work we should be able to simplify this. Test code is not compiled in some cases as there are some interesting circular references (test code for server depending on vmware plugin comes to mind), that needs to be sorted if we want maven to be able to do its job.
> 
> I've spend the last two days to get packaging back into something that looks workable. Even though we want to move away from waf, it seemed like the best way to fix the packaging for now. With the exception of awsapi all RPM packages are being built on my dev environment so that is looking good. I don't have a debian environment yet, so I'm not able to test a debian build. I haven't done any testing on the rpms yet, just verified that they are build, hoping to do that tomorrow.

Let's keep waf for sometime. But since ant is already broken :(

> Couple of questions that I would like some feedback on:
> * Maven includes the dependencies with their original filename, I believe we used to ship our deps with cloud- prepended to the name. Now we need to either check if our dependencies aren't already supplied by the OS or put them in a different directory, like /usr/share/java/cloud/*jar. What is the best route to take?

Fixable with a custom maven exec? I think it's better to ship these jars with a cloud- prefix (if there is no licensing issue) or for distro specific releases (debs, rpms) have the dependencies added in the rules.

> Feedback, ideas,

I'm new to maven and I think many of us are too. Could you share with us simple commands on how to use the ol' ant commands deploydb, deploy-server and debug with maven. Do we have those goals running now, or are the wip? For me ant commands don't work anymore.

> complaints anyone?

For the past one week I'm unable to get my server running.
Not towards any specific person, but can we please not commit partial changes into master at least?

Regards,
Rohit

> 
> Will send an update when there is something to report.
> 
> Cheers,
> 
> Hugo
> 

Re: Maven status (from my point of view)

[Marcus Sorensen <sh...@gmail.com>]

Should the current master have all of these changes now? Following
your procedure and then executing a './waf rpm' leaves me with no RPMs
and the messages:

[386/386] copy: usage/conf/log4j-cloud_usage.xml.in ->
artifacts/default/usage/conf/log4j-cloud_usage.xml
Error: Could not find or load main class com.cloud.api.doc.ApiXmlDocWriter
[INFO] Scanning for projects...

Build failed:  -> task failed (err #1):
	{task: xmlapi  -> commands.xml}


I do have ./artifacts/rpmbuild/BUILD/cloud-4.0/server/src/com/cloud/api/doc/ApiXmlDocWriter.java,
so I'm not sure if there's a path missing or if the order in which
things are being done is wrong.

On Tue, Sep 4, 2012 at 8:12 AM, Hugo Trippaers
<HT...@schubergphilis.com> wrote:
> Hey all,
>
> The switch to maven is coming along nicely I think, however we are not there yet. With the dependencies gone from the tree, the maven build is now the easiest way to build CloudStack IMHO. All classes are properly compiled and stored in jar files and dependencies are automagically downloaded to make sure everything compiles. Over the last few days I've added packaging of the systemvm.iso to the maven files and I did some work on the awsapi. The awsapi is proving to be difficult as it has multiple results (.mar, .aar, .jar files and a webapp). Maven doesn't really like this. On the other hand, maven has plugins to deal with axis2 and generate everything, so with a  bit of work we should be able to simplify this. Test code is not compiled in some cases as there are some interesting circular references (test code for server depending on vmware plugin comes to mind), that needs to be sorted if we want maven to be able to do its job.
>
> I've spend the last two days to get packaging back into something that looks workable. Even though we want to move away from waf, it seemed like the best way to fix the packaging for now. With the exception of awsapi all RPM packages are being built on my dev environment so that is looking good. I don't have a debian environment yet, so I'm not able to test a debian build. I haven't done any testing on the rpms yet, just verified that they are build, hoping to do that tomorrow.
>
> Couple of questions that I would like some feedback on:
> * Maven includes the dependencies with their original filename, I believe we used to ship our deps with cloud- prepended to the name. Now we need to either check if our dependencies aren't already supplied by the OS or put them in a different directory, like /usr/share/java/cloud/*jar. What is the best route to take?
> * Awsapi should be split in three sub projects (directories)  I think, the autogenerated ec2 part, the autogenerated e3 part and cloudstack own code which might be distributed as a webapp.
> * I've split the server rpm in two, one called server and contains all oss stuff and one called server-nonoss containing the modules that depend on non-oss libs, is that ok?
>
> Darren, is this more or less in line with the stuff you are doing?
>
> Feedback, ideas, complaints anyone?
>
> Will send an update when there is something to report.
>
> Cheers,
>
> Hugo
>