An new kind of cache for Ivy

[Nicolas Lalevée <ni...@hibnet.org>]

Hi,

I am starting to have Ivy be able to understand some Eclipse updatesite. I am now facing issues with the size of the downloaded metadata. So the next step is to put these files in a cache.

I had the same issue with the obr resolver. I did resolve this but with a kind of work around. I used the repositoryCacheManager associated with the resolver and make the obr xml descriptor an artifact in a special organisation+module. See the method init in OBRResolver [1].

For this kind of metadata, which are a sort of per "repository" or per resolver data rather the module data, we may need a specific cache for repository level data. I think this would also be helpful if one day we want to support some maven repository index [2].

But then I am wondering how should I manage the key for the entry in the cache. I am starting to think of a hash of the url of the resource to store. Would it be sufficient ? Should I also add the resolver name, or the resolver type ?

Nicolas

[1] https://svn.apache.org/repos/asf/ant/ivy/core/trunk/src/java/org/apache/ivy/osgi/obr/OBRResolver.java
[2] http://mirrors.ibiblio.org/pub/mirrors/maven2/dot-index/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: An new kind of cache for Ivy

[Nicolas Lalevée <ni...@hibnet.org>]

Le 14 avr. 2011 à 22:38, Nicolas Lalevée a écrit :

> Hi,
> 
> I am starting to have Ivy be able to understand some Eclipse updatesite. I am now facing issues with the size of the downloaded metadata. So the next step is to put these files in a cache.
> 
> I had the same issue with the obr resolver. I did resolve this but with a kind of work around. I used the repositoryCacheManager associated with the resolver and make the obr xml descriptor an artifact in a special organisation+module. See the method init in OBRResolver [1].
> 
> For this kind of metadata, which are a sort of per "repository" or per resolver data rather the module data, we may need a specific cache for repository level data. I think this would also be helpful if one day we want to support some maven repository index [2].
> 
> But then I am wondering how should I manage the key for the entry in the cache. I am starting to think of a hash of the url of the resource to store. Would it be sufficient ? Should I also add the resolver name, or the resolver type ?

Finally I implemented a new method in the RepositoryCacheManager: downloadRepositoryResource. It will put in cache any resource associated with a repository. Since these resources are associated with a repository, I though that the proper place of their cache is next to the cache of the module's artifact. So no need for another kind of cache.

About the implementation, I reused some code of the DefaultRepositoryCacheManager, so that the store of that kind of resource is stored like any other module artifact. The "organisation" of such resource will be then "_repository_metadata_", and the module name will be the SHA-1 of the resource name (the url for an URLResource) encoded in hexadecimal (just like does git about commits id actually).

I made the caching quite aggressive. If a resource is not found, we are caching that we didn't found it. And we'll check for updates of these resources (or check that a missing resource still miss) not more than every hour. It's obviously configurable, see CacheResourceOptions. Maybe we should also have a "magic" property like "ivy.repository.resource.cache.ttl".

It is still missing some unit test dedicated to that new method. The tests with the OBRResolver, the MirroredURLResolver and the UpdateSiteLoader, which are now using that new caching method, shows that is works quite well.

Nicolas



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org