You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@archiva.apache.org by James William Dumay <ja...@atlassian.com> on 2008/08/14 10:09:21 UTC

[comments] metadata-updater consumer in 1.2

Hey guys,
I've been working on making the metadata merging in trunk a little saner by
using the RepositoryMetadataMerge/Reader/Writer instead of those in
MetadataTools.

Part of this work is to remove our use of VersionedReference,
ArtifactReference, etc as at best their logic guesses the groupId,
artifactId and version - a source of many bugs.

What I want to discuss is the possibility to change the way that the
metadata-updater creates and updates metadata.

Currently the updater works by:
1. consuming artifact events
2. converting those paths to ArtifactReference
3. ArtifactReference is passed to MetadataTools.updateMetadata()
4. updateMetadata() converts the ArtifactReference back to a path.
5. Repeats steps 2 to 4 but using a VersionedReference.

How I propose this will work:
1. consume both POM artifact and metadata events
2. create a new maven-metadata.xml if it does not exist (Only if the event
was a POM event)'
4. check if a metadata update is actually needed (ie, if the artifact is
already in the existing metadata, do nothing).
3. Merge metadata given a new POM event or metadata event.

Advantages:
* cuts out the lossy conversion from path -> Artifact/VersionedReference ->
path
* can figure out the correct location, groupId, artifactId and version of
the metadata file to be updated or created by reading POM metadata.
* You would be able to transform a local repository into a server repository
* This implementation will have unit tests.

Disadvantages:
* May not have the entire model available on disk. In this case if the
parent is needed for a new metadata file to be written then we simply do
nothing with this consumer event. Later scans will probably happen at a time
when the parent becomes available.
* Walking the POM inheritance tree may be a little slower as you will need a
read/parse for every time you want to go up one level in the tree.

Thoughts? Questions?

Thanks,
James

Re: [comments] metadata-updater consumer in 1.2

Posted by James William Dumay <ja...@atlassian.com>.
Brett,

On 15/08/2008, at 4:47 PM, Brett Porter wrote:

> Generally in favour, though I agree with Deng and have a couple more  
> questions.
>
> On 14/08/2008, at 8:58 PM, James William Dumay wrote:
>
>>
>> On 14/08/2008, at 6:58 PM, Maria Odea Ching wrote:
>>>>
>>>> How I propose this will work:
>>>> 1. consume both POM artifact and metadata events
>>>> 2. create a new maven-metadata.xml if it does not exist (Only if  
>>>> the event
>>>> was a POM event)'
>>>> 4. check if a metadata update is actually needed (ie, if the  
>>>> artifact is
>>>> already in the existing metadata, do nothing).
>>>> 3. Merge metadata given a new POM event or metadata event.
>>>
>>>
>>> Hmm, what if the metadata is broken or corrupted (ex. there are  
>>> versions
>>> specified in the metadata but doesn't actually exist in the repo)?  
>>> Will the
>>> metadata be checked for that & get updated too?
>>>
>>
>> Removing missing versions is a little bit more difficult. If there  
>> was some differentiation between a managed repository used for  
>> deployment or used as a proxy cache this might be possible.
>
> Isn't that what we have now by the filename differentiation?

Sure, but say if we deleted all maven-metadata files (from remote  
repositories or not) then called this consumer we still wouldn't be  
able to tell this or not.

>
>
>>
>>
>> Metadata in the managed repository shows an incomplete view of one  
>> or more remote repositories that may not have their versions on  
>> disk yet - so in this case I believe we should keep those versions  
>> and not remove them.
>
> I thought we proxied metadata requests first then merged them so we  
> had a full picture?

We still do that - that will not change. What I ment was a version may  
not exist on disk but exist on the remote repository and the metadata  
should still reflect that.

>
>
> I'm a bit worried about losing this functionality (though TBH, I've  
> had trouble getting it to work consistently in the current version  
> anyway) - it's useful to be able to delete artifacts from the file  
> system and have it clean up after itself.

The current implementation is a bit touch and go. Im not sure how this  
behavior should work just yet - the safest bet would be to keep its  
metadata around if it still exists.


>
>
>>
>>
>>>
>>>>
>>>>
>>>> Advantages:
>>>> * cuts out the lossy conversion from path -> Artifact/ 
>>>> VersionedReference ->
>>>> path
>>>> * can figure out the correct location, groupId, artifactId and  
>>>> version of
>>>> the metadata file to be updated or created by reading POM metadata.
>
> Sounds good. Will it fall back to the directory structure if the POM  
> is not present/invalid? (Think jpox 1.1.9).

Yeah, we can do that.

>
>
>>>>
>>>> * You would be able to transform a local repository into a server
>>>> repository
>
> Not sure I understand this one?

The on disk format for remote and local is slightly different. The  
consumer could potentially morph local repo metadata into server  
metadata (Wendy was musing about this on IRC not long ago).

>
>
>>>>
>>>> * This implementation will have unit tests.
>
> tests FTW :)

Tests ^_^

>
>
>>>>
>>>>
>>>> Disadvantages:
>>>> * May not have the entire model available on disk. In this case  
>>>> if the
>>>> parent is needed for a new metadata file to be written then we  
>>>> simply do
>>>> nothing with this consumer event. Later scans will probably  
>>>> happen at a
>>>> time
>>>> when the parent becomes available.
>>>> * Walking the POM inheritance tree may be a little slower as you  
>>>> will need
>>>> a
>>>> read/parse for every time you want to go up one level in the tree.
>>>>
>>>> Thoughts? Questions?
>
> I don't really grok either of these - all the information for the  
> metadata (GAV) is in the uninherited POM, right?
>
> Here are the use cases I was thinking of testing explicitly, btw:
> * make sure versions are deleted when the artifacts are
> * make sure the metadata is recreated when it is deleted and the  
> artifacts are still intact
> * make sure it is updated when an artifact is added via the scan
> * various update/create scenarios for proxying
>
> Cheers,
> Brett
>
>>>>
>>>
>>>
>>> Sounds good to me :)
>>
>> Cool :)
>>
>> James
>
> --
> Brett Porter
> brett@apache.org
> http://blogs.exist.com/bporter/
>


Re: [comments] metadata-updater consumer in 1.2

Posted by Brett Porter <br...@apache.org>.
Generally in favour, though I agree with Deng and have a couple more  
questions.

On 14/08/2008, at 8:58 PM, James William Dumay wrote:

>
> On 14/08/2008, at 6:58 PM, Maria Odea Ching wrote:
>>>
>>> How I propose this will work:
>>> 1. consume both POM artifact and metadata events
>>> 2. create a new maven-metadata.xml if it does not exist (Only if  
>>> the event
>>> was a POM event)'
>>> 4. check if a metadata update is actually needed (ie, if the  
>>> artifact is
>>> already in the existing metadata, do nothing).
>>> 3. Merge metadata given a new POM event or metadata event.
>>
>>
>> Hmm, what if the metadata is broken or corrupted (ex. there are  
>> versions
>> specified in the metadata but doesn't actually exist in the repo)?  
>> Will the
>> metadata be checked for that & get updated too?
>>
>
> Removing missing versions is a little bit more difficult. If there  
> was some differentiation between a managed repository used for  
> deployment or used as a proxy cache this might be possible.

Isn't that what we have now by the filename differentiation?

>
>
> Metadata in the managed repository shows an incomplete view of one  
> or more remote repositories that may not have their versions on disk  
> yet - so in this case I believe we should keep those versions and  
> not remove them.

I thought we proxied metadata requests first then merged them so we  
had a full picture?

I'm a bit worried about losing this functionality (though TBH, I've  
had trouble getting it to work consistently in the current version  
anyway) - it's useful to be able to delete artifacts from the file  
system and have it clean up after itself.

>
>
>>
>>>
>>>
>>> Advantages:
>>> * cuts out the lossy conversion from path -> Artifact/ 
>>> VersionedReference ->
>>> path
>>> * can figure out the correct location, groupId, artifactId and  
>>> version of
>>> the metadata file to be updated or created by reading POM metadata.

Sounds good. Will it fall back to the directory structure if the POM  
is not present/invalid? (Think jpox 1.1.9).

>>>
>>> * You would be able to transform a local repository into a server
>>> repository

Not sure I understand this one?

>>>
>>> * This implementation will have unit tests.

tests FTW :)

>>>
>>>
>>> Disadvantages:
>>> * May not have the entire model available on disk. In this case if  
>>> the
>>> parent is needed for a new metadata file to be written then we  
>>> simply do
>>> nothing with this consumer event. Later scans will probably happen  
>>> at a
>>> time
>>> when the parent becomes available.
>>> * Walking the POM inheritance tree may be a little slower as you  
>>> will need
>>> a
>>> read/parse for every time you want to go up one level in the tree.
>>>
>>> Thoughts? Questions?

I don't really grok either of these - all the information for the  
metadata (GAV) is in the uninherited POM, right?

Here are the use cases I was thinking of testing explicitly, btw:
* make sure versions are deleted when the artifacts are
* make sure the metadata is recreated when it is deleted and the  
artifacts are still intact
* make sure it is updated when an artifact is added via the scan
* various update/create scenarios for proxying

Cheers,
Brett

>>>
>>
>>
>> Sounds good to me :)
>
> Cool :)
>
> James

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/


Re: [comments] metadata-updater consumer in 1.2

Posted by James William Dumay <ja...@atlassian.com>.
On 14/08/2008, at 6:58 PM, Maria Odea Ching wrote:
>>
>> How I propose this will work:
>> 1. consume both POM artifact and metadata events
>> 2. create a new maven-metadata.xml if it does not exist (Only if  
>> the event
>> was a POM event)'
>> 4. check if a metadata update is actually needed (ie, if the  
>> artifact is
>> already in the existing metadata, do nothing).
>> 3. Merge metadata given a new POM event or metadata event.
>
>
> Hmm, what if the metadata is broken or corrupted (ex. there are  
> versions
> specified in the metadata but doesn't actually exist in the repo)?  
> Will the
> metadata be checked for that & get updated too?
>

Removing missing versions is a little bit more difficult. If there was  
some differentiation between a managed repository used for deployment  
or used as a proxy cache this might be possible.

Metadata in the managed repository shows an incomplete view of one or  
more remote repositories that may not have their versions on disk yet  
- so in this case I believe we should keep those versions and not  
remove them.

>
>>
>>
>> Advantages:
>> * cuts out the lossy conversion from path -> Artifact/ 
>> VersionedReference ->
>> path
>> * can figure out the correct location, groupId, artifactId and  
>> version of
>> the metadata file to be updated or created by reading POM metadata.
>> * You would be able to transform a local repository into a server
>> repository
>> * This implementation will have unit tests.
>>
>> Disadvantages:
>> * May not have the entire model available on disk. In this case if  
>> the
>> parent is needed for a new metadata file to be written then we  
>> simply do
>> nothing with this consumer event. Later scans will probably happen  
>> at a
>> time
>> when the parent becomes available.
>> * Walking the POM inheritance tree may be a little slower as you  
>> will need
>> a
>> read/parse for every time you want to go up one level in the tree.
>>
>> Thoughts? Questions?
>
>
> Sounds good to me :)

Cool :)

James

Re: [comments] metadata-updater consumer in 1.2

Posted by Maria Odea Ching <oc...@apache.org>.
On Thu, Aug 14, 2008 at 4:09 PM, James William Dumay <ja...@atlassian.com>wrote:

> Hey guys,
> I've been working on making the metadata merging in trunk a little saner by
> using the RepositoryMetadataMerge/Reader/Writer instead of those in
> MetadataTools.
>
> Part of this work is to remove our use of VersionedReference,
> ArtifactReference, etc as at best their logic guesses the groupId,
> artifactId and version - a source of many bugs.
>
> What I want to discuss is the possibility to change the way that the
> metadata-updater creates and updates metadata.
>
> Currently the updater works by:
> 1. consuming artifact events
> 2. converting those paths to ArtifactReference
> 3. ArtifactReference is passed to MetadataTools.updateMetadata()
> 4. updateMetadata() converts the ArtifactReference back to a path.
> 5. Repeats steps 2 to 4 but using a VersionedReference.
>
> How I propose this will work:
> 1. consume both POM artifact and metadata events
> 2. create a new maven-metadata.xml if it does not exist (Only if the event
> was a POM event)'
> 4. check if a metadata update is actually needed (ie, if the artifact is
> already in the existing metadata, do nothing).
> 3. Merge metadata given a new POM event or metadata event.


Hmm, what if the metadata is broken or corrupted (ex. there are versions
specified in the metadata but doesn't actually exist in the repo)? Will the
metadata be checked for that & get updated too?


>
>
> Advantages:
> * cuts out the lossy conversion from path -> Artifact/VersionedReference ->
> path
> * can figure out the correct location, groupId, artifactId and version of
> the metadata file to be updated or created by reading POM metadata.
> * You would be able to transform a local repository into a server
> repository
> * This implementation will have unit tests.
>
> Disadvantages:
> * May not have the entire model available on disk. In this case if the
> parent is needed for a new metadata file to be written then we simply do
> nothing with this consumer event. Later scans will probably happen at a
> time
> when the parent becomes available.
> * Walking the POM inheritance tree may be a little slower as you will need
> a
> read/parse for every time you want to go up one level in the tree.
>
> Thoughts? Questions?


Sounds good to me :)


>
>
> Thanks,
> James
>

Thanks,
Deng