RE: future Ivy development (infallibility-of-metadata would be better from this point)

[Gilles Scokart <gs...@gmail.com>]

Personaly, I agree with Xavier.

A released artefact AND the attached metadata can not change once released.
It's exactly the purpose of a release.

Of curse error happens... (often).  If the user want to take that into
account, he should add a build number to its versions numbers to handle
that.

At most, I could accept an informative flag added to the metadata saying
that a release is deprecated or invalid.  

But for me, once it is released, nothing can be changed.

Gilles 


> -----Original Message-----
> From: Stephane Bailliez [mailto:sbailliez@gmail.com] 
> Sent: Wednesday, November 08, 2006 2:15 PM
> To: ivy-dev@incubator.apache.org
> Subject: Re: future Ivy development
> 
> Antoine Levy-Lambert wrote:
> > Steve Loughran wrote:
> >   
> >>
> >> We need to recognise that infallibility-of-metadata is an 
> unrealistic 
> >> ideal and adapt to it.
> >>     
> > I would more agree with Xavier, and believe that if the metadata in 
> > the POM or ivy.xml is wrong, there should be a new release 
> with a new 
> > version number.
> >   
> 
> No. Because there is a world of difference between people 
> releasing a piece of software and people using it and dealing 
> with dependencies.
> 
> What exists right now is what Steve describes.  What will 
> happen is that when you will need component A you will add it 
> to the repo and add the dependency descriptor. Don't think a 
> minute you will get it right the first time. You won't.
> 
> As for people describing the dependencies of their own piece 
> of software, just look at any Maven project and you will 
> realize that most of them are totally wrong as well, don't 
> care about scope or  simply mixing development and release.
> 
> > Otherwise, you can start tweaking everything everyday no ?
> >   
> Not everyday. But this is why an online repo does not work to 
> me unless there is a revision system behind it instead of 
> binning stuff.
> 
> You will change the metadata. If you want reproductability 
> (and you will want that) then version your repository.
> 
> 

Re: infallibility-of-metadata would be better from this point

[Steve Loughran <st...@apache.org>]

Brett Porter wrote:
> 

> 
> This does mean that we need overall more effecient ways to check for 
> updates in the repository than testing every file once a day, which is 
> something we are moving towards with the repository index.
> 

machine readable atom feed of changes? your app subscribes to the change 
list and marks as invalid changed metadata.

Re: infallibility-of-metadata would be better from this point

[Brett Porter <br...@apache.org>]

On 09/11/2006, at 1:27 AM, Xavier Hanin wrote:

> hat's why I began to thought about using compatible metadata  
> revision, so
> that you never break anything. Whenever you want to change the  
> metadata of a
> module, the only thing you can do is to add new configurations, and  
> maybe
> deprecate others. The metadata revision is identified in the file  
> by an id
> (which could be simple incremental number, or by a timestamp/user  
> uple as
> you indicate) Each configuration declares in which metadata  
> revision it has
> been introduced/deprecated. By default when you depend on a module you
> depend on its first metadata revision, unless explicitly specified
> (specifying a specific revision or the latest one if it makes sense).

This is basically the conclusion we arrived at too (which I now  
recall as being nearly 12 months ago, horrified that we still haven't  
gotten around to implementing it :)

This was kind of the original purpose of the "build number", being  
for re-releases and external packagers. However, I think it needs to  
be a built in concept like you mention, rather than part of the main  
version.

I'd suggest always using the latest metadata, but locking it down to  
a specific revision when you produce the release (like you would with  
a version range) for reproducibility.

This does mean that we need overall more effecient ways to check for  
updates in the repository than testing every file once a day, which  
is something we are moving towards with the repository index.

Cheers,
Brett

Re: infallibility-of-metadata would be better from this point

[Stephane Bailliez <sb...@gmail.com>]

Xavier Hanin wrote:
>
> What I think is really important is to be able to go back to the 
> version of
> metadata used for a particular build, and using a tool like svn for 
> that is
> something I've already considered and seen implemented by some 
> customers. It
> works pretty well, the problem is that then you have to deal with 
> metadata
> versions (as you explain Stephane) and this is not easy. Especially when
> several nodes in your dependency graph require different metadata 
> revisions.
> What is even less easy is to maintain metadata for a public repository 
> where
> you can't be sure the change is an improvement. It can be for one 
> person but
> not for the other.

"improvement" is indeed relative. It is just like fixing bugs.
If an app relies on a bug (eeek) to do things and that later this bug is 
fixed, this may trigger another bug. :)

I would not really look for granular metadata revision to the node level 
(this can get too complicated and is really unecessary to me) but more 
on the whole repository thing.
Basically on your configuration you would say "I depend on r214093 of 
the repository (or a tag name if you wish...))

> That's why I began to thought about using compatible metadata 
> revision, so
> that you never break anything. Whenever you want to change the 
> metadata of a
> module, the only thing you can do is to add new configurations, and maybe
> deprecate others. The metadata revision is identified in the file by 
> an id
> (which could be simple incremental number, or by a timestamp/user uple as
> you indicate) Each configuration declares in which metadata revision 
> it has
> been introduced/deprecated. By default when you depend on a module you
> depend on its first metadata revision, unless explicitly specified
> (specifying a specific revision or the latest one if it makes sense). 
> [...]

This will get way too granular I think.

And just adding configuration is enough to break things too.

I think we have to recognize as Steve said, that metadata can change. 
This is a fact of life. Trying to solve this problem will simply get 
things way more complicated than they actually are or need to be and we 
will get entangled into a gigantic net which will make your life 10 
times more difficult.

And how do you manage dependencies too ? In a corporate environment you 
can be mean and when you upgrade a third party foo 1.1 to 1.2 you could 
very well want to change (for very valid reasons) all other dependency 
descriptors (even third party one) that say that depends on foo 1.1 and 
bump them to foo 1.2 (which avoids unecessary conflict resolution and 
unnecessary clutter).

-- stephane

Re: infallibility-of-metadata would be better from this point

[Gilles Scokart <gs...@gmail.com>]

2006/11/8, Xavier Hanin <xa...@gmail.com>:
> That's why I began to thought about using compatible metadata revision, so
> that you never break anything. Whenever you want to change the metadata of a
> module, the only thing you can do is to add new configurations, and maybe
> deprecate others. The metadata revision is identified in the file by an id
> (which could be simple incremental number, or by a timestamp/user uple as
> you indicate) Each configuration declares in which metadata revision it has
> been introduced/deprecated. By default when you depend on a module you
> depend on its first metadata revision, unless explicitly specified
> (specifying a specific revision or the latest one if it makes sense). The
> compatibility of all metadata revisions ensure that when you get the latest
> metadata revision (i.e. the current one on the repository) you are able to
> know its whole history. Thus you are able to resolve the dependency on the
> metadata revision 1 if you get the revision 2, the only thing that changes
> is that when you declare a dependency on all configurations, it actually
> means all configurations which existed in the metadata revision you
> indicate.
>
> This idea is the result of a few hours of thinking during my trip back from
> california last september, but I hadn't time to think more about it til now,
> nor to experiment with it. So I'm pretty sure you will find a lot of flaws
> to this idea, but it may be a good start for some kind of solution.
>
> Xavier
>
>

I'm not sure to understand what you explains (even after reading it a
few times), but here is what it inspired me when I was on the train
this afternoon (a short travel... so short inspiration).

When we want a build to be reproducable, we classically use an SCM
repository for the source.  The process is to define a tag name, apply
this tag to the source you want to build, and then run the build.

The same principle should be applied when some files (the meta data
files) are comming from outside our source repository.

Of curse, you can not store the tags of all your users into a public
repository.  But the "tag info" can be stored somwhere else (for
example in a report file stored by the user in his own repository
under his build tag).  If the build use this "report" instead of
re-resolving the dependencies, you are sure to repeat the same build.

With a meta-data revision number, it should be achievable.

Re: infallibility-of-metadata would be better from this point

[Xavier Hanin <xa...@gmail.com>]

On 11/8/06, Steve Loughran <st...@apache.org> wrote:
>
> Gilles Scokart wrote:
> > Personaly, I agree with Xavier.
> >
> > A released artefact AND the attached metadata can not change once
> released.
> > It's exactly the purpose of a release.
> >
> > Of curse error happens... (often).  If the user want to take that into
> > account, he should add a build number to its versions numbers to handle
> > that.
> >
> > At most, I could accept an informative flag added to the metadata saying
> > that a release is deprecated or invalid.
> >
> > But for me, once it is released, nothing can be changed.
> >
> > Gilles
> >
>
> this works for an in house project, and I strongly agree with freezing
> binary data.
>
> But what if, say, Sun release J2ee 5.0, and the first edition of the
> metadata written by a third party is bad. Do you go to sun and say
> 'release a new version' for the sake of our repository?. Do you make up
> your own version number, so sun are suddenly fielding support calls
> related to j2ee-5.0.1? What happens when j2ee-5.0.1 itself ships?
>
> Maybe the problem here is that we are using the metadata and the
> artifact version as if they are the same thing. when really we want
> j2ee5.0 described by metadata version 2006-11-08/stevel, the latter
> probably with its own unique URL. But try adding that concept to a
> library manager and see where you get.
>
> Metadata is fallible. In some AI related areas (Agent oriented
> programming), all facts are modelled as beliefs issued by an individual
> at a specific time. Its ok for 2006-11-08/xavier to be different from
> 2006-11-08/steve, which can be different from  2006-11-09/steve. But
> 2006-11-08/steve must remain constant, and ideally, internally consistent.
>
> I agree with your concerns, and think this problem is really difficult to
address and deserves its own discussion, so I thank Gilles to have changed
the mail subject :-)

I've already thought a lot about that, but didn't reach any strong
conclusion.

What I think is really important is to be able to go back to the version of
metadata used for a particular build, and using a tool like svn for that is
something I've already considered and seen implemented by some customers. It
works pretty well, the problem is that then you have to deal with metadata
versions (as you explain Stephane) and this is not easy. Especially when
several nodes in your dependency graph require different metadata revisions.
What is even less easy is to maintain metadata for a public repository where
you can't be sure the change is an improvement. It can be for one person but
not for the other.

That's why I began to thought about using compatible metadata revision, so
that you never break anything. Whenever you want to change the metadata of a
module, the only thing you can do is to add new configurations, and maybe
deprecate others. The metadata revision is identified in the file by an id
(which could be simple incremental number, or by a timestamp/user uple as
you indicate) Each configuration declares in which metadata revision it has
been introduced/deprecated. By default when you depend on a module you
depend on its first metadata revision, unless explicitly specified
(specifying a specific revision or the latest one if it makes sense). The
compatibility of all metadata revisions ensure that when you get the latest
metadata revision (i.e. the current one on the repository) you are able to
know its whole history. Thus you are able to resolve the dependency on the
metadata revision 1 if you get the revision 2, the only thing that changes
is that when you declare a dependency on all configurations, it actually
means all configurations which existed in the metadata revision you
indicate.

This idea is the result of a few hours of thinking during my trip back from
california last september, but I hadn't time to think more about it til now,
nor to experiment with it. So I'm pretty sure you will find a lot of flaws
to this idea, but it may be a good start for some kind of solution.

Xavier

Re: infallibility-of-metadata would be better from this point

[Steve Loughran <st...@apache.org>]

Gilles Scokart wrote:
> Personaly, I agree with Xavier.
> 
> A released artefact AND the attached metadata can not change once released.
> It's exactly the purpose of a release.
> 
> Of curse error happens... (often).  If the user want to take that into
> account, he should add a build number to its versions numbers to handle
> that.
> 
> At most, I could accept an informative flag added to the metadata saying
> that a release is deprecated or invalid.  
> 
> But for me, once it is released, nothing can be changed.
> 
> Gilles 
> 

this works for an in house project, and I strongly agree with freezing 
binary data.

But what if, say, Sun release J2ee 5.0, and the first edition of the 
metadata written by a third party is bad. Do you go to sun and say 
'release a new version' for the sake of our repository?. Do you make up 
your own version number, so sun are suddenly fielding support calls 
related to j2ee-5.0.1? What happens when j2ee-5.0.1 itself ships?

Maybe the problem here is that we are using the metadata and the 
artifact version as if they are the same thing. when really we want 
j2ee5.0 described by metadata version 2006-11-08/stevel, the latter 
probably with its own unique URL. But try adding that concept to a 
library manager and see where you get.

Metadata is fallible. In some AI related areas (Agent oriented 
programming), all facts are modelled as beliefs issued by an individual 
at a specific time. Its ok for 2006-11-08/xavier to be different from 
2006-11-08/steve, which can be different from  2006-11-09/steve. But 
2006-11-08/steve must remain constant, and ideally, internally consistent.