[jira] Commented: (IVY-399) Flexible Cache Management

["Robert Buck (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IVY-399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514199 ] 

Robert Buck commented on IVY-399:
---------------------------------

I was told that this feature would not make it into 2.0, which I find particularly bothersome in two ways. First the user community overwhelmingly requested this feature to be fixed, as it resolves a number of issues that people have run into in years past. Secondly, I am concerned that with such a major change, that it will not show up in 2.1 (feature release) because it is not a feature -- it is a major architecture change (which would imply to me it would come in at 3.0). This is the very concern I expressed 6 months ago in an email thread to this group. Unless a reasonable workaround is available, I will have to suspend IVY use at VeriSign; there would be little benefit using IVY without this feature over what we already do, which is store all third-party artifacts in Perforce. I am pretty ticked off that the top feature people are asking for is not going to be available, and the less voted for things got priority.

It was suggested that I document what problem this would resolve for us. Here goes, with all ranting and raving included. Makes me feel like Bill O'Reilly.

We presently store all third party artifacts in Perforce along with the source tree. Both get versioned together, and are fully available upon checkout, ready to build and deploy. The rub with this approach is that we are storing the artifacts in Perforce. Let me explain why this is not the best solution. First, we end up in a situation over time where we have all versions of all third-party libraries used in Perforce. Secondly, developers end up checking out third-party libraries that are not necessary to build particular components; you get all or nothing. Thirdly, checkouts involve pulling down out of Perforce several hundred megabytes of third party libraries, which is slow -- and for the common case a developer only needed a small subset of these to fix a single bug and test.

It is my opinion that IVY does not manage third-party artifacts particularly well at the point of the network interface.

Conceptually, the first tier aspect of something like IVY is the relationship between the public repository and the client, that is the network interface. Nobody in their right mind would be using IVY in a single user, isolated scenario -- it is wasted effort.

There are two primary concerns that should be immediately recognizable: the network, and the number of clients. These two factors greatly impact performance and scalability. Digging into the term "client", you should identify what is meant by the term:

a. single host, single sandbox, single product
b. single host, multiple sandbox, single product
c. single host, single sandbox, multiple products, lots of common dependencies
d. single host, multiple sandbox, multiple products, lots of common dependencies
e. multiple hosts (with all variations above)

In the case of (a), IVY provides ZERO benefit over sticking third-party libraries into Perforce. However, to optimize cases (b) through (d), cases which if solved, would provide lots of benefits to end-users (developers), you would need an architecture like this:

[central public repository] (1 TO MANY) [host-specific caching/mirroring repository] (1 to MANY) [local-sandbox caching repository]

Here, multiple sandboxes on a single host share network resolved third-party libraries. If artifacts are found in the host specific caching repository, they are NOT copied into the sandbox-specific cache; do not waste the resources, build time, and risk potential (or frequent) corruption issues.

In the case of (b) through (e), without a caching local repository that mirrors, but only what is demanded from the public repository, and where this is shared by multiple local clients, IVY provides ZERO benefits over sticking third-party libraries into Perforce.

That pretty much sums up my opinion of the current version of IVY and what I'd like to see in IVY. And to summarize, if there is not a solution for this, I will have to recommend that VeriSign use alternative solutions.


> Flexible Cache Management
> -------------------------
>
>                 Key: IVY-399
>                 URL: https://issues.apache.org/jira/browse/IVY-399
>             Project: Ivy
>          Issue Type: Improvement
>         Environment: ALL
>            Reporter: Eric Crahen
>
> Creating an issue at Xaviers request for improving the approach to cache management
> On 1/29/07, Xavier Hanin <xa...@gmail.com> wrote:
>     Supporting this kind of graph
>     could be interesting, and what makes it difficult for Ivy is that Ivy
>     heavily relies on its cache mechanism, which makes it impossible to do
>     what you want (i.e. never put anything from your local repository to
>     the cache).
> This would be a very powerful feature to add. In 2.0, is there any reason for the cache to have to be so baked into everything? In otherwords, why not implement every resolver and all of the internal management w/ no caching what so ever baked in anywhere? Instead all caching is done in a decorator fashion by wrapping a caching resolver around any other resolver? In otherwords, the core of Ivy only knows about resolvers, no concept of cache exists in the heart of Ivy.
> It seems to me this would be much more flexible, and it would still be very possible to provide the syntactic sugar to make it very simple and even seemless to configure these wrappers by default. At the same time, people who will use the flexibility have the power to set up chains that might go something like.
> (logical chain)
>   localresolver
>   cacheresolver
>     httpresolver url="..."
>   cacheresolver
>     httpresolver url="..."
> There is no longer any need to have things like useLocal flags. Its already expressed that the local resolver is not cached because its just not wrapped in a caching resolver.
> I think this idiom should be applied to both artifact and metadata resolution.
> One cool thing about this, is that in this way, since all caching is simply a type of resolver we'd provide people who don't like the particular method we use to perform caching in the resolver we provide are free to provide their own. This would address lots of the issues that have been raised about caching, consistency, doing anything remotely fancy with local resolvers - right now its very hard to address any of that because caching is not very plugable as it stands.
> I think the only drawback is that it seems like its harder to configure out of the box because most people by default would want to wrap every resolver with a cacheresolver - but like I said, this is easily solvable by providing some simple syntactic sugar. For instance the simplehttpresolver might be the name of an undecorated resolver for power users, and the things named httpresolver would simple be an alias for the cacheresolver wrapped around the simplehttpresolver (or subclass, whatever is the most sensible choice)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.