[jira] Commented: (IVY-399) Flexible Cache Management

["Xavier Hanin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IVY-399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12493914 ] 

Xavier Hanin commented on IVY-399:
----------------------------------

To start working on this issue with a clean view of how the cache is currently implemented in Ivy, I've listed all files which are cached, with their location, content and use:
{noformat}
All default locations are relative to the cache root.

ResolvedIvyFileInCache
======================
default location: 
    resolved-[organisation]-[module]-[revision].xml
content:
    the representation as an ivy file of the module descriptor used for dependency resolution
created by:
    ResolveEngine#resolve(ModuleDescriptor md, ResolveOptions options)
used by:
    PublishEngine#publish(ModuleRevisionId mrid, Collection srcArtifactPattern, String resolverName, PublishOptions options)
        => when no src ivy pattern is provided, the cached file is used
    DeliverEngine#deliver(ModuleRevisionId mrid, String revision, String destIvyPattern, DeliverOptions options)
        => used as basis for the ivy file to deliver
    RetrieveEngine#getConfs(ModuleRevisionId mrid, RetrieveOptions options)
        => to list the configurations of a module to retrieve, when they are not provided in the RetrieveOptions, or provided as '*' wildcard

ResolvedIvyPropertiesInCache
============================
default location:
    resolved-[organisation]-[module]-[revision].properties
content:
    a map of resolved revision and status by dependency, stored as properties
created by:
    ResolveEngine#resolve(ModuleDescriptor md, ResolveOptions options)
used by:
    DeliverEngine#deliver(ModuleRevisionId mrid, String revision, String destIvyPattern, DeliverOptions options)
        => used to replace dynamic versions by static ones, and do recursive delivery according to the dependencies status

ConfigurationResolveReportInCache
=================================
default location (not configurable):
    resolveId + "-" + conf + ".xml" (usually resolvedId is [organisation]-[module])
content:
    xml resolution report for one configuration
created by:
    XmlReportOutputter#output(ConfigurationResolveReport report, String resolveId, String[] confs, File destDir)
used by:
    ConfigurationResolveReport
        => to load previous dependencies to know if they have changed
    RetrieveEngine#determineArtifactsToCopy(ModuleRevisionId mrid, String destFilePattern, RetrieveOptions options)
    IvyReport
    IvyRepositoryReport
    IvyArtifactProperty
    IvyCacheTask (basis of cachepath and cachefileset)
    Main#outputCachePath
        => to know the details of the last resolve operation

IvyFileInCache
==============
default location:
    [organisation]/[module]/ivy-[revision].xml
content:
    an ivy file representation of a downloaded module descriptor, converted into the system namespace
created by:
    BasicResolver#getDependency(DependencyDescriptor dd, ResolveData data)
used by:
    CacheManager#findModuleInCache(ModuleRevisionId mrid, boolean validate)
        => common method used to reload a module information from cache. This method is in turn used by:
            * BasicResolver#parse => to avoid downloading a module descriptor before parsing it (used when the version matcher requires metadata parsing)
            * BasicResolver#getDependency => to avoid downloading a module descriptor when resolving a dependency
            * CacheResolver#getDependency => it's the purpose of the CacheResolver to reuse module descriptors from cache
    BasicResolver#parse
        => to check that the resolver does not point directly to the cache, which is forbidden
    RetrieveEngine#retrieve(ModuleRevisionId mrid, String destFilePattern, RetrieveOptions options)
        => when the retrieve operation is asked to retrieve module descriptors, files from the cache are used

ArchiveFileInCache
==================
default location:
    [organisation]/[module]/[type]s/[artifact]-[revision](.[ext])
content:
    a module artifact
created by:
    BasicResolver#download(Artifact[] artifacts, DownloadOptions options)
used by:
    IvyCachePath#execute()	
    IvyCacheFileset#execute()
        => to know which file should be put in the path or fileset
    RetrieveEngine#retrieve(ModuleRevisionId mrid, String destFilePattern, RetrieveOptions options)
        => to know which files should be copied
    Main#invoke
        => to build the classloader to use for launching
    Main#outputCachePath
        => to report the paths of artifacts
    IvyArtifactReport#writeCacheLocation
        => to add the artifact cache location in the artifact report
    CacheResolver#download
        => to check the file exists in cache
    BasicResolver#parse
        => to delete old artifacts when downloading a module revision which has changed

CachedDataFile
==============
default location:
    [organisation]/[module]/ivydata-[revision].properties
content:
    Artifact Origin (is local + location)
    Resolver Name
    Artifacts Resolver Name (different from Resolver in case of dual resolver)
created by:
    the cache manager to store the information listed in content
used by:
    the cache manager to load information listed in content.
    This information is used by:
    ArtifactOrigin
        => used by the same classes as ArchiveFileInCache
    Resolve and Artifact Resolver
        => used by findModuleInCache, to reassociate a cached module to the resolvers which originally resolved it
        Side Note: this causes problems when trying to reuse a same cache with different settings, causing warning messages 'unknown resolver xxx'
{noformat}

> Flexible Cache Management
> -------------------------
>
>                 Key: IVY-399
>                 URL: https://issues.apache.org/jira/browse/IVY-399
>             Project: Ivy
>          Issue Type: Improvement
>         Environment: ALL
>            Reporter: Eric Crahen
>
> Creating an issue at Xaviers request for improving the approach to cache management
> On 1/29/07, Xavier Hanin <xa...@gmail.com> wrote:
>     Supporting this kind of graph
>     could be interesting, and what makes it difficult for Ivy is that Ivy
>     heavily relies on its cache mechanism, which makes it impossible to do
>     what you want (i.e. never put anything from your local repository to
>     the cache).
> This would be a very powerful feature to add. In 2.0, is there any reason for the cache to have to be so baked into everything? In otherwords, why not implement every resolver and all of the internal management w/ no caching what so ever baked in anywhere? Instead all caching is done in a decorator fashion by wrapping a caching resolver around any other resolver? In otherwords, the core of Ivy only knows about resolvers, no concept of cache exists in the heart of Ivy.
> It seems to me this would be much more flexible, and it would still be very possible to provide the syntactic sugar to make it very simple and even seemless to configure these wrappers by default. At the same time, people who will use the flexibility have the power to set up chains that might go something like.
> (logical chain)
>   localresolver
>   cacheresolver
>     httpresolver url="..."
>   cacheresolver
>     httpresolver url="..."
> There is no longer any need to have things like useLocal flags. Its already expressed that the local resolver is not cached because its just not wrapped in a caching resolver.
> I think this idiom should be applied to both artifact and metadata resolution.
> One cool thing about this, is that in this way, since all caching is simply a type of resolver we'd provide people who don't like the particular method we use to perform caching in the resolver we provide are free to provide their own. This would address lots of the issues that have been raised about caching, consistency, doing anything remotely fancy with local resolvers - right now its very hard to address any of that because caching is not very plugable as it stands.
> I think the only drawback is that it seems like its harder to configure out of the box because most people by default would want to wrap every resolver with a cacheresolver - but like I said, this is easily solvable by providing some simple syntactic sugar. For instance the simplehttpresolver might be the name of an undecorated resolver for power users, and the things named httpresolver would simple be an alias for the cacheresolver wrapped around the simplehttpresolver (or subclass, whatever is the most sensible choice)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.