Re: local repositories (was: http://jira.codehaus.org/browse/MNG-3151)

[Brett Porter <br...@apache.org>]

On 23/08/2007, at 7:12 AM, Stephen Connolly wrote:

> Or a better local repository model ;-)

+1. I already started to draft a proposal for the wiki on this, but  
hadn't posted it yet.

>
> One where there are two local repositories (committed and current  
> build) that get layered together.

What I was actually thinking was the lock operating at the individual  
artifact level, and being a read lock as well as a write lock. The  
installation step itself should be very quick for the artifact and  
it's associated metadata because they are already built, so this is  
not a bad lock to have, and I think it's still necessary to have a  
read lock because there's no way to atomically do the merge you refer  
to later on.

One of the advantages of what you were describing is the ability to  
not install anything unless the entire build passes - that has some  
virtues as well, but I consider it a separate feature to fixing the  
local repository concurrency problems.

> If you have layering, then you could have a shared local repository  
> as a third layer to save duplication of files.

Yes, this is definitely something I'd like to see. I think that we  
should be separating the cache from the place that developers install  
their own things. There are a number of reasons this is a good idea:

1) it would simplify a lot of the artifact handling code
2) it lets you clean up easily - you likely keep your cache but nuke  
your local builds
3) you can create separate local repository workspaces on a single  
machine without needing to redownload all the rest of the cache  
(useful for CI servers)
4) you could start to share a cache (though I'd still recommend a  
shared remote repository instead of doing this, it's at least less  
harmful than doing it now)

Cheers,
Brett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: local repositories

[Brett Porter <br...@apache.org>]

On 23/08/2007, at 3:59 PM, Stephen Connolly wrote:

>> One of the advantages of what you were describing is the ability  
>> to not install anything unless the entire build passes - that has  
>> some virtues as well, but I consider it a separate feature to  
>> fixing the local repository concurrency problems.
> I am torn over which way to run on this question.

I think it's something that would be separately configurable. I don't  
think it would be the default. The individual artifacts, once built,  
should be useful whether the rest build or not - but it really  
depends on your project.

> The only issue is how to push from one layer to the next... don't  
> want to end up with another phase between install and deploy (local- 
> deploy???) that has the same locking problems!

I wasn't suggesting any pushing here: install goes into separate  
local installation repository only, deploy pushes to the remote  
repository from there, dependency resolution looks first in the  
installation repository, then in the cache (if that's the way it's  
defined), and then of course in the remote repository to bring into  
the cache. There's no need to migrate things between them.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: local repositories

[Stephen Connolly <st...@one-dash.com>]

Brett Porter wrote:
>
> On 23/08/2007, at 7:12 AM, Stephen Connolly wrote:
>
>> Or a better local repository model ;-)
>
> +1. I already started to draft a proposal for the wiki on this, but 
> hadn't posted it yet.
>
>>
>> One where there are two local repositories (committed and current 
>> build) that get layered together.
>
> What I was actually thinking was the lock operating at the individual 
> artifact level, and being a read lock as well as a write lock. The 
> installation step itself should be very quick for the artifact and 
> it's associated metadata because they are already built, so this is 
> not a bad lock to have, and I think it's still necessary to have a 
> read lock because there's no way to atomically do the merge you refer 
> to later on.
>
> One of the advantages of what you were describing is the ability to 
> not install anything unless the entire build passes - that has some 
> virtues as well, but I consider it a separate feature to fixing the 
> local repository concurrency problems.
I am torn over which way to run on this question.

On CI servers, if we have builds in parallel, and some of those 
components have integration tests with non-consistent run times. i.e. 
may take 5 min or may take 2 min for the tests to complete, you could 
end up with two parallel builds with the second starting 1 min after the 
first and mixing the artifacts they use.  Of course when all plugins 
honour the attached scope resolution for finding artifacts this would 
not be an issue!

A simpler fix would be when a Maven process grabs the write lock for an 
artifact, it holds the write lock until the process ends.

Thus parallel builds of different projects would not interfere, we 
wouldn't need layering (though I'd really like that) and parallel builds 
of the same project would be first come holds everyone else until done.
>
>> If you have layering, then you could have a shared local repository 
>> as a third layer to save duplication of files.
>
> Yes, this is definitely something I'd like to see. I think that we 
> should be separating the cache from the place that developers install 
> their own things. There are a number of reasons this is a good idea:
>
> 1) it would simplify a lot of the artifact handling code
> 2) it lets you clean up easily - you likely keep your cache but nuke 
> your local builds
> 3) you can create separate local repository workspaces on a single 
> machine without needing to redownload all the rest of the cache 
> (useful for CI servers)
> 4) you could start to share a cache (though I'd still recommend a 
> shared remote repository instead of doing this, it's at least less 
> harmful than doing it now)
>
The only issue is how to push from one layer to the next... don't want 
to end up with another phase between install and deploy 
(local-deploy???) that has the same locking problems!
> Cheers,
> Brett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org