Maven and Ant collaboration on task codebase

[Brett Porter <br...@gmail.com>]

Hi,

I'm basically looking for some opportunities for sibling Apache
communities to work together in an area we have some overlap, and
wanted to see what thoughts the Ant community had on this. I've
invited folks on the Maven lists to follow this thread here too, so if
you see some strange faces - that might be it.

This has been discussed before here briefly by myself and Vincent
Massol, and is also inspired by the original Ant2 proposal by Peter
Donald which he has reviewed recently:
http://blogs.codehaus.org/people/vmassol/archives/001036_id_love_reusable_ant_tasks.html
http://codemyth.blogspot.com/2005/09/no-shared-vision-no-ant2.html

I'm not asking to revisit something that has already failed, BTW, but
am looking for a best path forward.

Maven is finding ourselves in the position now of wanting to reuse a
lot of the Ant task code. Some that particularly spring to mind:
- Exec
- compiler tasks
- archiving (jar, zip, tar)

We're not the first to have encountered this: other projects have
forked this code before. Eventually, the archiving tasks found their
way to jakarta commons-compress, and I've recently started
commons-exec with others to try and build reusable code from the Exec
tasks. commons-jci exists which came from Cocoon and could potentially
be merged with the Ant code for javac. There are others like
commons-io that share much code with Ant. At this point, I'm hoping to
start fleshing them out and using them instead of our own forked
versions (that mostly came via Avalon and have since permuted).

On the flip side, Maven has a number new tasks we are building that
could be used in Ant. We've already done so for dependency management,
but we have libraries for SCM (support for CVS, SVN, Clearcase,
Perforce and Starteam and IIUC it goes beyond the support provided by
Ant), Wagon (scp/sftp/http/ftp/file/scm transport through a common
API) and we are also planning others, like a generic issue tracking
interface library.
Probably most notable would be the various reports like the SCM change
logs, release notes, and so on.

Then there are others we have started together (and not progressed
much yet): commons-openpgp.

I think there is a definite need for these libraries to be separate
from Ant so that they can be used in other places where Ant is not
available. I know you can use Ant via the Java API, but Vincent
illustrates that it can be a bit verbose, and you end up with a rather
large dependency full of unneeded tasks and bindings to the Ant
project. These are still
options under Maven though, BTW - there are several ways to use Ant
"as is" if that is the best solution for the person in question.

My main concern is that with all these libraries, that the refactoring
may not be used in Ant, which means that should Ant expand or fix bugs
in its code, that is not propagated to the code others are using.

So, my question is - what is the road map for Ant 1.8 (possibly 1.7)
and beyond? It seems antlibs is becoming the way to extend Ant - is it
possible that more of the task code may be moved into that? If so, is
it possible that task code may be decoupled from Ant, or a smaller set
of types might be established? For example, Maven2 plugins can run
without
dependencies (like in ant, by reflecting execute), but also has a 5K
library with an implementable interface and logging provision. I
understand some tasks may require the basic fileset/path/etc support -
could they be made into a smaller API to allow better separation of
the antlibs?

Does the practice of creating commons libs like we have been doing
work for Ant, and would you accept patches to integrate them back into
Ant? I think this is the best first step, but see a few issues:
- backwards compatibility: Probably fine for the task usage, but I'm
not sure of the public API of the tasks may already be in use in java
code that could be affected by this
- JDK requirement: we've typically moved to support a minimum of 1.3
and sometimes 1.4
- JAR hell: harder to bootstrap Ant given dependencies that probably
need Ant to build (we encounter the same issue with Maven), and don't
want to get into issues of someone using different versions of the
libraries in their own tasks if it conflicts with Ant. Should be
easily
overcome by repackaging/recompiling the sources of the commons
libraries with a new package for the main Ant distributable (which
brings it back to the current solution at runtime, but allows more
flexibility for those that use the code aside of Ant).

Of course, if it is not seen as beneficial to Ant in any way, then I'm
happy to continue building up the common libraries as we've started,
and run the risk of having to fix similar bugs. The stability of the
Ant code base suggests this risk is quite low.

What benefits are there for Ant? A warm fuzzy feeling?  :)  As I
mentioned, I hope that Maven will produce more plugins that are useful
from Ant. We also have a growing and active community that code at
Apache (and those that build plugins outside of Apache due to
licensing issues). We even have a few Ant-only users that are getting
more involved in Maven due to the availability of the Maven dependency
tasks for Ant that could hopefully grow into contributors not only to
those but other antlibs.

Again, not looking to make a call to action, but just to make sure
we're all facing in the right direction and can naturally move to a
way to better collaborate.

I look forward to your thoughts, opinions and flames  :) 

Thanks,
Brett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Steve Loughran <st...@apache.org>]

Brett Porter wrote:
> On 9/21/05, Steve Loughran <st...@apache.org> wrote:
> 
>>yes, we can discuss it :)
>>
>>I do actually agree that execution and JAR files ought to be structured
>>so that they can be reused. The code in there is some of the best
>>debugged stuff out there in Java for doing both.
> 
> 
> Ok - good that we agree on the fundamental point :)
> 
> 
>>But I dont want to break ant or make Ant1.7 slip in the process; and we
>>cannot change the public interfaces for all the existing classes.
> 
> 
> Right. That was the first point I listed about backwards compat, and
> your facade plan would allow that. This in a beyond-Ant-1.7 timeframe
> sounds fine by me too - I would expect the time taken to get the
> refactoring happening, tests done, library documented would exceed
> that anyway. And hopefully there will be people outside of Ant willing
> to contribute to the effort.
> 
> 
>>which leaves a design like
>>
>>-move execution, Jar support self contained packages in the ant codebase
>>-provide self contained API with unit tests for that functionality.
>>-retain the existing stuff as facades
> 
> 
> yep.
> 
> 
>>-release a separate ant-execution library, at exactly the same frequency
>>as ant point releases.
> 
> 
> I'm not sure that this is required. Surely an Ant release can just use
> the last release of that library, whenever and whatever that was. If
> there is a real need to get a certain fix or feature in, then driving
> the release of that library before Ant needs to happen. This isn't
> cumbersome if the development of the library is set up to allow
> frequent releases (ie, a stable branch all the reliable fixes and
> improvements go onto).
> 
> 
>>The latter is going to be problematic, and here's why. If there is one
>>thing that always causes grief in Ant, its JAR files sticking other Jars
>>inside. Groovy, jython, maven-artifact-ant-2.0-beta-1-dep; all
>>incorporate other things like regexp, jsch, whatever into their JAR,
>>creating lots of classpath conflict. If we integrate execution into ant,
>>either within ant.jar or as an associated JAR, we cannot make it
>>possible for someone to stick an inconsistent version in, -or at least
>>we cannot make it possible for someone like me to ship a 3rd party ant
>>task that also includes an updated version of the libraries.
> 
> 
> Ok, I did fix your main gripes in the artifact ant tasks for b1 :) No
> libraries other than Maven ones and plexus (these need to change too
> eventually, but they don't have many 3rd party users right now, and
> none that are writing ant tasks).

thank you!

> 
> Anyway, this was my 3rd point in the original email. It is an issue,
> but it can be resolved by incorporating that code in a different
> package structure. This should be a relatively simple task:
> - get release being used from svn (externals?)
> - copy sources into ant source tree/temp source directory, mapping
> paths from o/a/commons/exec -> o/a/tools/ant/exec
> - replace package statement in source files
> 
> It is a bit gruesome, but it does work.

i see, a kind of automated boostrap process, requiring SVN, but only at 
the time you want to download and create the snapshot. then patching all 
the stuff to the new packages and inserting "dont edit this here" 
messages. yes, it might work. I've done worse.

> 
> What are the alternatives to this? You can keep the source in the Ant
> tree, but you still need to do the package munging in reverse:
> - keep the above layout, but build "ant-exec" library from the source
> tree after copying and adjusting the packages to something different
> from what Ant uses
> - build the "ant-exec" library separately, after ant is built, based
> on those sources. This would tie the exec library release to the Ant
> release.
> 
> To me, moving the code to commons (or similar subspace of the ant SVN
> tree), with its own release cycle and incorporating it back into the
> Ant bootstrap rather than the other seems the more natural option.
> We're faced with a similar problem in Maven 2 and that is the approach
> we plan to take. In fact, since this will be a standalone Java app, we
> could probably share it.

I suppose with your approach, it opens up execution management to a 
broader committer base, which can only be a good thing. I'd probably 
steal it for smartfrog BTW; the execution stuff there is not as 
sophisticated, though it does fork the running JVM better (though not as 
well as if Java had a fork() command)

what do you suppose the base exec and package libraries would look like? 
That is, how well can we decouple them from Ant's datatypes and other 
features.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Stefan Bodewig <bo...@apache.org>]

On Wed, 21 Sep 2005, Brett Porter <br...@gmail.com> wrote:
> On 9/21/05, Steve Loughran <st...@apache.org> wrote:

>> -release a separate ant-execution library, at exactly the same
>> frequency as ant point releases.
> 
> I'm not sure that this is required. Surely an Ant release can just
> use the last release of that library, whenever and whatever that
> was.

The "whenever that was" part scares me, honestly.  I've seen struts
releases slip because they needed a new release of some commons
libraries.

If we go the route of commons then we must accept that our release
schedule suddenly can be held up by a commons committer -1ing a
release of a component that Ant depends on - when Ant depends on
something not available in the latest release.

We must also accept that Ant committers have to monitor commons-dev
and ensure a component we depend on doesn't get changed in a way that
negatively affects Ant.  I know I've been overcommitting in the past
(see commons-openpgp) and doubt that I'd have the bandwidth to do
that.  We must understand that splitting out stiff we depend on means
more work for us, even after all the refactorings have been done.

And of course the new commons components must be buildable by Gump as
reliably as bootatrap-ant does today.  Gump is the major integration
test for Ant.

> - get release being used from svn (externals?)

Yes, we need to have them in Ant's build tree at bootstrap time.

> - copy sources into ant source tree/temp source directory, mapping
> paths from o/a/commons/exec -> o/a/tools/ant/exec
> - replace package statement in source files
> 
> It is a bit gruesome, but it does work.

gruesome, I agree.  I don't see any other approach either.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Brett Porter <br...@gmail.com>]

On 9/21/05, Steve Loughran <st...@apache.org> wrote:
> yes, we can discuss it :)
> 
> I do actually agree that execution and JAR files ought to be structured
> so that they can be reused. The code in there is some of the best
> debugged stuff out there in Java for doing both.

Ok - good that we agree on the fundamental point :)

> But I dont want to break ant or make Ant1.7 slip in the process; and we
> cannot change the public interfaces for all the existing classes.

Right. That was the first point I listed about backwards compat, and
your facade plan would allow that. This in a beyond-Ant-1.7 timeframe
sounds fine by me too - I would expect the time taken to get the
refactoring happening, tests done, library documented would exceed
that anyway. And hopefully there will be people outside of Ant willing
to contribute to the effort.

> which leaves a design like
> 
> -move execution, Jar support self contained packages in the ant codebase
> -provide self contained API with unit tests for that functionality.
> -retain the existing stuff as facades

yep.

> -release a separate ant-execution library, at exactly the same frequency
> as ant point releases.

I'm not sure that this is required. Surely an Ant release can just use
the last release of that library, whenever and whatever that was. If
there is a real need to get a certain fix or feature in, then driving
the release of that library before Ant needs to happen. This isn't
cumbersome if the development of the library is set up to allow
frequent releases (ie, a stable branch all the reliable fixes and
improvements go onto).

> The latter is going to be problematic, and here's why. If there is one
> thing that always causes grief in Ant, its JAR files sticking other Jars
> inside. Groovy, jython, maven-artifact-ant-2.0-beta-1-dep; all
> incorporate other things like regexp, jsch, whatever into their JAR,
> creating lots of classpath conflict. If we integrate execution into ant,
> either within ant.jar or as an associated JAR, we cannot make it
> possible for someone to stick an inconsistent version in, -or at least
> we cannot make it possible for someone like me to ship a 3rd party ant
> task that also includes an updated version of the libraries.

Ok, I did fix your main gripes in the artifact ant tasks for b1 :) No
libraries other than Maven ones and plexus (these need to change too
eventually, but they don't have many 3rd party users right now, and
none that are writing ant tasks).

Anyway, this was my 3rd point in the original email. It is an issue,
but it can be resolved by incorporating that code in a different
package structure. This should be a relatively simple task:
- get release being used from svn (externals?)
- copy sources into ant source tree/temp source directory, mapping
paths from o/a/commons/exec -> o/a/tools/ant/exec
- replace package statement in source files

It is a bit gruesome, but it does work.

What are the alternatives to this? You can keep the source in the Ant
tree, but you still need to do the package munging in reverse:
- keep the above layout, but build "ant-exec" library from the source
tree after copying and adjusting the packages to something different
from what Ant uses
- build the "ant-exec" library separately, after ant is built, based
on those sources. This would tie the exec library release to the Ant
release.

To me, moving the code to commons (or similar subspace of the ant SVN
tree), with its own release cycle and incorporating it back into the
Ant bootstrap rather than the other seems the more natural option.
We're faced with a similar problem in Maven 2 and that is the approach
we plan to take. In fact, since this will be a standalone Java app, we
could probably share it.

Is this feasible, or am I missing something?

Cheers,
Brett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Steve Loughran <st...@apache.org>]

Kev Jackson wrote:
> Here's my non-committer 2pennies worth...
> 
> I think it would be beneficial to refactor Ant for some of the 
> commons-libs (commons-exec for example), I also think it'd be useful for 
> Ant 1.7 to splitoff as many tasks as possible into antlibs.  The Antlib 
> idea seems to me to be the best way of refactoring Ant into a component 
> based system where individual tasks can be released without having to 
> release a new version of Ant (the main purpose of Antlibs was this as 
> far as I'm aware).

I think the main goal of antlibs was to
(a) permit stuff to be completely decoupled (eg dotnet)
(b) make it easier for 3rd party stuff to integrate withant without all 
the typedefs

As someone who does do third party tasks to work with both ant1.6 and 
1.7, I am also well aware of the price: versioning against Ant JARS. 
Even If we fix something in Ant SVN HEAD, anything released on its own 
schedule still has to work with the older stuff. I have real problems 
with things like java and the spawn options, and something in file utils 
warns me it is deprecated whenever I build, just because I want to 
retain compatibility with Ant1.6

> 
> With the Antlibs in place (assume all SCM related tasks and some others 
> converted to antlibs instead of core/optional), the process of 
> refactoring out commons-XXX libraries I would think would be simpler, 
> not least in tracking and fixing bugs.
> 
> You're right when you mention the Jar hell problem, I can imagine that 
> would be a major pain to deal with if ant required commons-xyz and 
> commons-xyz required ant to compile.

yes, ant has the bootstrap problem. Core ant cannot depend upon anything 
under than the JDK and an XML parser. And that includes execution.

> 
> Anyway, from me, I think it'd be worthwhile to at least discuss 
> refactoring out common components.

yes, we can discuss it :)

I do actually agree that execution and JAR files ought to be structured 
so that they can be reused. The code in there is some of the best 
debugged stuff out there in Java for doing both.

But I dont want to break ant or make Ant1.7 slip in the process; and we 
cannot change the public interfaces for all the existing classes.

which leaves a design like

-move execution, Jar support self contained packages in the ant codebase
-provide self contained API with unit tests for that functionality.
-retain the existing stuff as facades
-release a separate ant-execution library, at exactly the same frequency 
as ant point releases.

The latter is going to be problematic, and here's why. If there is one 
thing that always causes grief in Ant, its JAR files sticking other Jars 
inside. Groovy, jython, maven-artifact-ant-2.0-beta-1-dep; all 
incorporate other things like regexp, jsch, whatever into their JAR, 
creating lots of classpath conflict. If we integrate execution into ant, 
either within ant.jar or as an associated JAR, we cannot make it 
possible for someone to stick an inconsistent version in, -or at least 
we cannot make it possible for someone like me to ship a 3rd party ant 
task that also includes an updated version of the libraries.

-steve


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Stefan Bodewig <bo...@apache.org>]

On Wed, 21 Sep 2005, Kev Jackson <ke...@it.fts-vn.com> wrote:

> The Antlib idea seems to me to be the best way of refactoring Ant
> into a component based system where individual tasks can be released
> without having to release a new version of Ant (the main purpose of
> Antlibs was this as far as I'm aware).

Not directly.  We'd still need to bundle those split out antlibs with
core releases, but it would allow us to release them seperately, yes.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Maven and Ant collaboration on task codebase

[Kev Jackson <ke...@it.fts-vn.com>]

Here's my non-committer 2pennies worth...

I think it would be beneficial to refactor Ant for some of the 
commons-libs (commons-exec for example), I also think it'd be useful for 
Ant 1.7 to splitoff as many tasks as possible into antlibs.  The Antlib 
idea seems to me to be the best way of refactoring Ant into a component 
based system where individual tasks can be released without having to 
release a new version of Ant (the main purpose of Antlibs was this as 
far as I'm aware).

With the Antlibs in place (assume all SCM related tasks and some others 
converted to antlibs instead of core/optional), the process of 
refactoring out commons-XXX libraries I would think would be simpler, 
not least in tracking and fixing bugs.

You're right when you mention the Jar hell problem, I can imagine that 
would be a major pain to deal with if ant required commons-xyz and 
commons-xyz required ant to compile.

Anyway, from me, I think it'd be worthwhile to at least discuss 
refactoring out common components.

Kev

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org