Composing a system out of projects

[ru...@us.ibm.com]

Disclaimer:

   I'm intentionally publishing this as work in progress.  Don't get hung
   up on the fact that the current implementation is using shell scripts or
   on the name of the URLs.  I'm looking to get feedback to influence the
   evolution of this work.  Also note that the initial focus is Apache,
   Java based projects.  I expect this to expand over time.

   Also forgive the cross-posting, please direct any discussion to ant-dev.

Vision:

   I would like to see us get to the point where a user can design their
   own custom system. Simple example: building a system consisting of
   JetSpeed plus Cocoon.  Users should be able to specify this using either
   the command line or a system.xml file.  This system.xml would reference
   projects, and the build system (presumably based on Ant) would order the
   building of projects much in the same way that the current Ant orders
   the execution of targets.

   Project.xml files would be checked into the same cvs tree as the source,
   and would essentially be declarative.  At a minimum, they would describe
   what projects they depend on, and what outputs (typically jar files) are
   produced.  Optional overrides for such things as the default target and
   name of the destination directory should be supported.  These
   project.xml files could also either contain the build instructions
   directly, or delegate to an existing build.xml via an <ant> task.

   Build.xml files would essentially remain as they are today.  Some
   enhancements are discussed below.

Progress:

   You can see the current state of evolution of this at the following
   URLs.  These are set up to update nightly at 1:00 a.m. EDT, and may be
   updated at other times if I run the script manually.  Per the disclaimer
   above, I realize that everything should be converted to Java and XML
   (both the inputs and outputs), but I chose to do it the "hard way" first
   so that I could understand the issues.

   Without further ado, here are the URLs:

     Script:
     http://oss.software.ibm.com/developerworks/opensource/jakarta/buildall.html
     Log:
     http://oss.software.ibm.com/developerworks/opensource/jakarta/buildlog.html
     Output:   ftp://anonjakarta@oss.software.ibm.com/

Todos:

   In addition to the parsing of the new XML elements and ordering of
   projects, the highest priority items as I see it deal with the
   processing of jar files.  We need a standard way to modify the class
   path when executing a nested build.xml - this internally could either be
   based on java fork=yes, or on a custom class loader.  Additionally I
   would like to see a new task which scanned the classpath and copied the
   actual jar files used in the build process to the destination directory;
   today most projects will happily compile with your choice of
   dependencies, then copy the developer's (possibly incompatible) choice
   of jar files to the destination directory.

   I'd also like to see a global approach to the choice of compiler, debug
   options, and the like.  And other user level choices like whether the
   javadocs or source are to be part of the custom distribution.  Many
   projects also can conditionally compile in support for other projects,
   so following the lead of many install programs, I would like a
   project.xml be able to define "minimal", "typical", and "full"
   configurations.

   Longer term, a GUI front end, and back ends which produce installable
   images.

Ok, that should be enough to get the discussion started...

- Sam Ruby

Re: Composing a system out of projects

[Peter Donald <do...@mad.scientist.com>]

At 09:43  16/6/00 -0400, you wrote:
>   Project.xml files would be checked into the same cvs tree as the source,
>   and would essentially be declarative.  At a minimum, they would describe
>   what projects they depend on, and what outputs (typically jar files) are
>   produced.  Optional overrides for such things as the default target and
>   name of the destination directory should be supported.  These
>   project.xml files could also either contain the build instructions
>   directly, or delegate to an existing build.xml via an <ant> task.

I like it so far ...

>   In addition to the parsing of the new XML elements and ordering of
>   projects, the highest priority items as I see it deal with the
>   processing of jar files.  We need a standard way to modify the class
>   path when executing a nested build.xml - this internally could either be
>   based on java fork=yes, or on a custom class loader. 

again I would like to put in my vote for the following syntax.

<sometask
ant:classpath="http://myBiz.com/some.jar|file://other.jar|./libs/myLib.zip"
... />

where the attribute is in the namespace ant and can accept an arbitary list
of URLs to classpath (With URLs resolved to new URL( "file:." ) for
relative URLS). Of course the whole issue of seperator between URLS would
have to be solved aswell :P. The task would be executed with that as the
classpath. The classpath could either be passed in to task via a new
ClassLoader (preferable method) or via a new VM. This choice could be
determined by another attribute in ant namespace such as 

<sometask ant:fork="true" ... />

>   Additionally I
>   would like to see a new task which scanned the classpath and copied the
>   actual jar files used in the build process to the destination directory;
>   today most projects will happily compile with your choice of
>   dependencies, then copy the developer's (possibly incompatible) choice
>   of jar files to the destination directory.

I like this but it could run into some problems with primordial classloader
during bootstrapping. Perhaps we could immediately instantiate another
classloader on loading ant so that the bootstrap classes (like
parser/tasks) can be excluded from classpath when scanning for the created
jars ... thoughts ???

>   I'd also like to see a global approach to the choice of compiler, debug
>   options, and the like.  And other user level choices like whether the
>   javadocs or source are to be part of the custom distribution.  Many
>   projects also can conditionally compile in support for other projects,
>   so following the lead of many install programs, I would like a
>   project.xml be able to define "minimal", "typical", and "full"
>   configurations.

good idea. I guess if we were going to go this path it looks very similar
to be a cross between apt/red-hat packages and installshield. Is this what
you are going for ? If so it may even be worthwhile to allow downloading of
binary images instead of source ... you wouldn't even have to change much.

You never mentioned detection of system features (like what
configure/autoconf does) but I was assuming you wanted something like that ?

It would also be nice to have a framework for merging config files and
generally configuring system (as in post compile configure that includes
things like adding entries to web.xml/cocoon.properties files). This would
allow a 1 click download-install-configure step that would gain the apache
projects the love and adoration of lazy system admins everywhere :P. 

>   Longer term, a GUI front end, and back ends which produce installable
>   images.

very long term could also create a client-side database of downloaded
projects + their locations + src etc etc

>Ok, that should be enough to get the discussion started...

:P

It looks very similar to a download-installshield-configure-autoupdate
system and there are many people out their in userland that have already
done partial implementations of this in various forms. There is even a few
instalshield apps for linux that use XML as a description database ... so
if I haven't read you wrong (which is quite possible) it may be an idea to
"send the word out" to see how much interest it could generate. There are a
few vocal individuals who are trying to unify linux packages (ie RPM vs the
other format - which I forget at the moment). So given enough interest it
could possibly become a vendor neutral thingie that used beyond the scope
of just java projects....

Hmmm getting way beyond what you intended thou I think. But theres my
thoughts ):P


Cheers,

Pete

*------------------------------------------------------*
| "Nearly all men can stand adversity, but if you want |
| to test a man's character, give him power."          |
|       -Abraham Lincoln                               |
*------------------------------------------------------*

Re: Composing a system out of projects

[James Duncan Davidson <ja...@eng.sun.com>]

on 6/16/00 6:43 AM, rubys@us.ibm.com at rubys@us.ibm.com wrote:

> I would like to see us get to the point where a user can design their
> own custom system. Simple example: building a system consisting of
> JetSpeed plus Cocoon.  Users should be able to specify this using either
> the command line or a system.xml file.  This system.xml would reference
> projects, and the build system (presumably based on Ant) would order the
> building of projects much in the same way that the current Ant orders
> the execution of targets.

Better yet -- a web page where a servlet handled the request, went off and
built everything up, then streamed the result back to the user.

.duncan