A rethink of the Ivy migration

[Ross Gardler <rg...@apache.org>]

I've been having a rethink of our approach to the IVY migration.

Pretty much all of our java dependencies are Cocoon dependencies. So why 
should we bother to manage them ourselves?

We should be able to just have a dependency on Cocoon and the blocks 
that we require and that is it. Why can't we do that?

We can't just have a dependency on Cocoon because we use a version built 
from SVN. This version is not SVN head, but is a past version of Cocoon. 
If it were SVN head we could use the snapshot repository to pull down 
the latest snapshot build of Cocoon and go from there.

So why don't we use SVN head?

Because SVN head has no crawler.

So, we have two choices for progression.

Continue with the path I have outlined for the IVY migration whereby we 
use our own local repo to hold the specific version of Cocoon that we 
know will work for us, or, we can use the move to ivy to force us to 
upgrade to Cocoon head (something we should probably do for the 0.8 
release anyway).

The latter route will require us to build a crawler, but Thorsten has 
already started on that with the Droids lab.

I've tried setting it up with just a cocoon-core 2.2-snapshot 
dependency, there are a couple of unresolved dependencies, but I am sure 
these can be easily addressed.

The big advantage in going the latter route is that it will make future 
upgrades to Cocoon a simple case of running an ant task. The downside is 
that we will have to write a crawler (using the Droids lab code - which 
is largely based on the crawler from Nutch if I understand correctly).

The big advantage of the former route is that we will be able to make 
things work as they do now, in trunk, without any surprises from the 
Cocoon upgrade. The downside is that we have to manage all our own 
dependencies and will have to continue to build Cocoon independently.

So, should we continue with managing our own dependencies or should we 
jump the short term hurdle and get the ivy branch working with Cocoon 
2.2 snapshots?

Ross

Re: A rethink of the Ivy migration

[Ross Gardler <rg...@apache.org>]

I've put this move to Ivy on hold until after the 0.8 release. I'm 
finding little enough time to address this as it is. However, I have had 
some input from the Ivy team about the best approach to the repository 
issues.

More info after we complete 0.8

Ross

Re: A rethink of the Ivy migration

[Ross Gardler <rg...@apache.org>]

Ross Gardler wrote:
> David Crossley wrote:
>> Ross Gardler wrote:
>>> So, should we continue with managing our own dependencies or should 
>>> we jump the short term hurdle and get the ivy branch working with 
>>> Cocoon 2.2 snapshots?
>>
>> So far i like your plan.
>>
>> It would need some Forrest people with serious Java skills
>> to do it. There is more than just the CLI to deal with.
>> Carsten recently gave us some tips on this dev list about
>> how to proceed.
> 
> I think we have that (and I wouldn't say "serious" java skills, just 
> some java skill).
> 
>>
>> There are some issues.
>>
>> See the comments on [1] http://people.apache.org/builds.html
>> about the "Snapshot Repositories".
>> Particlarly: "...but should not be referenced in released
>> versions of your Apache project."
> 
> We already include a snapshot in our releases. As for referencing the 
> snapshot repo in the ivy configuration we need to do this in order to do 
> integration testing.
> 
>> One thing is that Cocoon-2.2 will probably be properly
>> released soon. We can help to move that forward.
> 
> Yes, I noticed that on their list (trying to catch up with what has 
> happened to date).
> 
>> Another thing is that the Forrest project has decided
>> in the past that it is okay to use snapshot versions.
>> Not sure if that is a wise or well-informed decision.
> 
> In the early days we were forced to use a snapshot as we were patching 
> trunk to achieve things in Cocoon that we needed. It's been a very long 
> time since we needed to do this. Perhaps it is time to revisit this issue.
> 
>> On a related matter, i was liking our new approach to
>> managing our own dependencies. Having an Ivy descriptor
>> for each product enabled us to clearly describe each,
>> say where its home, what its license is, and link to
>> a jar in a maven repo. Nice.
> 
> This is a very good point, if we take it along with Thorstens 
> observation that an output of managing our own descriptors would be a 
> valuable contribution to the Ivy project I begin to think the extra work 
> would be worth it.
> 
> However, for this to be the case we should remove the maven repos from 
> our ivyconf.xml file an only use the ivy repos. This increases our work 
> in the short term even more.
> 
> The advantages would be:
> 
> - significant contribution to an ivy repo
> - full control over our own descriptors and dependencies (i.e. control 
> over license information, second tier dependencies etc.)
> - no need to update Cocoon until 0.9-dev cycle (i.e. time to address the 
> CLI issue)
> 
> I'm starting to think my very first plan was the right one.

OK, in the absence of further comment I've decided to return to the 
original plan with a slight modification:

We will not be using the Maven repos at all. If we are going to put the 
effort into switching to Ivy lets do it properly and use only Ivy repos.

This means we need to remove the maven repo's from the ivyconf.xml file 
and reinstate the jars I deleted from our shared ivy repo.

Rather than kill the ivyconf.xml with respect to maven file I think I'll 
make an ivyconf-maven.xml which includes the maven repos and an 
ivyconf.xml file that does not. That way I can still cheat on my private 
projects ;-)

Ross

Re: A rethink of the Ivy migration

[Ross Gardler <rg...@apache.org>]

David Crossley wrote:
> Ross Gardler wrote:
>> So, should we continue with managing our own dependencies or should we 
>> jump the short term hurdle and get the ivy branch working with Cocoon 
>> 2.2 snapshots?
> 
> So far i like your plan.
> 
> It would need some Forrest people with serious Java skills
> to do it. There is more than just the CLI to deal with.
> Carsten recently gave us some tips on this dev list about
> how to proceed.

I think we have that (and I wouldn't say "serious" java skills, just 
some java skill).

> 
> There are some issues.
> 
> See the comments on [1] http://people.apache.org/builds.html
> about the "Snapshot Repositories".
> Particlarly: "...but should not be referenced in released
> versions of your Apache project."

We already include a snapshot in our releases. As for referencing the 
snapshot repo in the ivy configuration we need to do this in order to do 
integration testing.

> One thing is that Cocoon-2.2 will probably be properly
> released soon. We can help to move that forward.

Yes, I noticed that on their list (trying to catch up with what has 
happened to date).

> Another thing is that the Forrest project has decided
> in the past that it is okay to use snapshot versions.
> Not sure if that is a wise or well-informed decision.

In the early days we were forced to use a snapshot as we were patching 
trunk to achieve things in Cocoon that we needed. It's been a very long 
time since we needed to do this. Perhaps it is time to revisit this issue.

> On a related matter, i was liking our new approach to
> managing our own dependencies. Having an Ivy descriptor
> for each product enabled us to clearly describe each,
> say where its home, what its license is, and link to
> a jar in a maven repo. Nice.

This is a very good point, if we take it along with Thorstens 
observation that an output of managing our own descriptors would be a 
valuable contribution to the Ivy project I begin to think the extra work 
would be worth it.

However, for this to be the case we should remove the maven repos from 
our ivyconf.xml file an only use the ivy repos. This increases our work 
in the short term even more.

The advantages would be:

- significant contribution to an ivy repo
- full control over our own descriptors and dependencies (i.e. control 
over license information, second tier dependencies etc.)
- no need to update Cocoon until 0.9-dev cycle (i.e. time to address the 
CLI issue)

I'm starting to think my very first plan was the right one.

Ross

Re: A rethink of the Ivy migration

[David Crossley <cr...@apache.org>]

Ross Gardler wrote:
> 
> So, should we continue with managing our own dependencies or should we 
> jump the short term hurdle and get the ivy branch working with Cocoon 
> 2.2 snapshots?

So far i like your plan.

It would need some Forrest people with serious Java skills
to do it. There is more than just the CLI to deal with.
Carsten recently gave us some tips on this dev list about
how to proceed.

There are some issues.

See the comments on [1] http://people.apache.org/builds.html
about the "Snapshot Repositories".
Particlarly: "...but should not be referenced in released
versions of your Apache project."

One thing is that Cocoon-2.2 will probably be properly
released soon. We can help to move that forward.

Another thing is that the Forrest project has decided
in the past that it is okay to use snapshot versions.
Not sure if that is a wise or well-informed decision.
It seemed not too bad because Forrest committers were
making deliberate decisions about that and checking the
license issues for every product in our lib/* etc.
Now with handing off dependency management, things would
be different.

Also not sure what are ASF principles (beyond the new
document listed [1]) about using pre-released versions
of supporting software.

On a related matter, i was liking our new approach to
managing our own dependencies. Having an Ivy descriptor
for each product enabled us to clearly describe each,
say where its home, what its license is, and link to
a jar in a maven repo. Nice.

But it is more work.

-David

Re: A rethink of the Ivy migration

[Ross Gardler <rg...@apache.org>]

Thorsten Scherler wrote:
> On Wed, 2007-02-28 at 12:30 +0000, Ross Gardler wrote:
>> I've been having a rethink of our approach to the IVY migration.
>>
>> Pretty much all of our java dependencies are Cocoon dependencies. So why 
>> should we bother to manage them ourselves?
> 
> Totally agree. All dependencies should come from cocoon but cocoon would
> need to have an ivy.xml for it, right?

No, Ivy can parse maven dependencies. At least I assume that's the case. 
When I did a test with just cocoon-core in the dependency it downloaded 
a lot more than just cocoon-core.jar.

> The route that we are going ATM is very good for Ivy because we will
> hand over a first class rep but like you point out the java dependencies
> are Cocoon dependencies and we should manage the our specific ones.

I should have clarified that now that David has pointed us at the 
official Apache repo I'm finding that many many more of the artifacts we 
need are already there.

However, they are mostly coming from the maven repo, not the ivy ones. 
So you point is valid.

>> The latter route will require us to build a crawler, but Thorsten has 
>> already started on that with the Droids lab.
> 
> Yes, but the plugins that I wrote are not yet ready to go prime time. It
> should be easy to make them stable (more if we going to use them here)
> but ATM there are not (needs more eyes and input). 

Yeah, I realise that, good you pointed it out though not everyone here 
is on the labs list.

I'm interested in using Droids in Forrest2, but that is still 
experimental so is not getting a great deal of my time right now.

> The implemented crawler x-m02y07 is more wget style and we can use it as
> base for the cocoon cli. I actually wrote the crawler having our use
> case in mind. 

Cool. We only need wget style crawling to be able to reproduce the CLI 
in Cocoon. We need to support the ability to ignore certain URLS, but 
that's an easy task (I vaguely recall a FAQ on how to do this so we will 
need to support it).

> The critical issue is that we will need time to stabilize this new
> crawler since forrest is heavily based on the static export. 

Good point.

...

>> So, should we continue with managing our own dependencies or should we 
>> jump the short term hurdle and get the ivy branch working with Cocoon 
>> 2.2 snapshots?
> 
> The last option is the one that we would need in the future in any way.
> Still the question is whether we want to put the release on hold for
> another while.

The way I see it is the release is not happening because none of us are 
addressing the remaining issues.

Of course, I'm making it worse by doing the ivy-branch and making this 
suggestion. I'm also making it worse by playing with my Forrest2 
experiments rather than getting 0.8 out the door.

However, for me the delay in getting 0.8 sorted is a symptom of the fact 
that Forrest is getting quite difficult to maintain and I'm focussing on 
trying to address this.

I'm not asking people to help in the Ivy branch, I'd rather see others 
focus on the 0.8 issues (of course the choice is individual). That being 
said, I want to make sure what I am doing on the Ivy branch is 
worthwhile to the community as it moves towards or beyond the 0.8 release.

> I personally think that having only one dependency on cocoon is the only
> think that makes sense for us. I agree to Ross question: why should we
> manage cocoon dependencies?


More opinions please...

Ross

Re: A rethink of the Ivy migration

[Thorsten Scherler <th...@apache.org>]

On Wed, 2007-02-28 at 12:30 +0000, Ross Gardler wrote:
> I've been having a rethink of our approach to the IVY migration.
> 
> Pretty much all of our java dependencies are Cocoon dependencies. So why 
> should we bother to manage them ourselves?

Totally agree. All dependencies should come from cocoon but cocoon would
need to have an ivy.xml for it, right?

The route that we are going ATM is very good for Ivy because we will
hand over a first class rep but like you point out the java dependencies
are Cocoon dependencies and we should manage the our specific ones.

...
> So why don't we use SVN head?
> 
> Because SVN head has no crawler.
> 
> So, we have two choices for progression.
> 
> Continue with the path I have outlined for the IVY migration whereby we 
> use our own local repo to hold the specific version of Cocoon that we 
> know will work for us, or, we can use the move to ivy to force us to 
> upgrade to Cocoon head (something we should probably do for the 0.8 
> release anyway).

That I am actually not sure. Seeing that 0.8 is way overdue updating
cocoon will not help to release faster.

> 
> The latter route will require us to build a crawler, but Thorsten has 
> already started on that with the Droids lab.

Yes, but the plugins that I wrote are not yet ready to go prime time. It
should be easy to make them stable (more if we going to use them here)
but ATM there are not (needs more eyes and input). I just changed the
dependencies management to ivy today to get some practice for our
migration.

Droids just has a couple of dependencies not like cocoon, so I just
needed to add one jar in an ivy repository. Actually like Ross points
out forrest will end with nearly only one dependency to cocoon. The rest
is coming from cocoon.

> 
> I've tried setting it up with just a cocoon-core 2.2-snapshot 
> dependency, there are a couple of unresolved dependencies, but I am sure 
> these can be easily addressed.
> 
> The big advantage in going the latter route is that it will make future 
> upgrades to Cocoon a simple case of running an ant task. The downside is 
> that we will have to write a crawler (using the Droids lab code - which 
> is largely based on the crawler from Nutch if I understand correctly).

I took Nutch and ripped out the plugin/extension point framework and
wrote some PoC plugins. I changed many thinks to make the code simpler
so from Nutch original code is not much left. The first crawler is not
close to the one from nutch but via plugins one could implement the same
functionality (we do not need this for our use case). 

The implemented crawler x-m02y07 is more wget style and we can use it as
base for the cocoon cli. I actually wrote the crawler having our use
case in mind. 

The critical issue is that we will need time to stabilize this new
crawler since forrest is heavily based on the static export. 

The core is the link recognition where we can go different routes. In
the short term we can enhance the parse-html droids plugin with neko
html (similar route as nutch is going) but in the long run we should try
to incorporate a virtual browser like Stefano pointed out on the labs
ml.  

...
> So, should we continue with managing our own dependencies or should we 
> jump the short term hurdle and get the ivy branch working with Cocoon 
> 2.2 snapshots?

The last option is the one that we would need in the future in any way.
Still the question is whether we want to put the release on hold for
another while.

I personally think that having only one dependency on cocoon is the only
think that makes sense for us. I agree to Ross question: why should we
manage cocoon dependencies?

salu2
-- 
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java & XML                consulting, training and solutions