Cleanup callbacks in IvyContext?

[Charles Duffy <ch...@dyfis.net>]

Howdy, all --

I'm trying to strengthen Ivy's locking system to make it strong enough to
allow ivy:clean at arbitrary times on systems which can have other actions
making use of the same shared caches.

There are a few requirements to make this happen while still allowing
multiple resolves (and like operations) to occur concurrently. One of those
is maintaining nonexclusive read locks (as opposed to the write locks which
are currently supported), and cleaning them up when necessary. For
ease-of-implementation, I'm currently proposing to only support this
behavior when NIO locks (which implicitly support shared locking semantics)
are enabled.

To clean these locks up without requiring end-user code to be modified, I
propose using the IvyContext stack -- allowing Runnables to be attached to
a stack element, and invoking them implicitly when the stack is popped.


Because converting a read lock to a write lock is inherently prone to race
conditions, we might need to break backwards compatibility with respect to
ivy:clean, allowing this to be called only inside a context where no read
operations have been done -- or breaking read lock semantics by dropping
all read locks before grabbing the write lock.


Thoughts?

Re: Cleanup callbacks in IvyContext?

[Nicolas Lalevée <ni...@hibnet.org>]

Le 9 janv. 2014 à 19:43, Charles Duffy <ch...@dyfis.net> a écrit :

> On Thu, Jan 9, 2014 at 12:17 PM, Nicolas Lalevée <nicolas.lalevee@hibnet.org
>> wrote:
> 
>> Yep, a quite hard problem to resolve. At the ant task level I don't know
>> how you could nail it without asking the end user to call some ivy:lock
>> task.
>> 
> 
> I'd like to argue that for ant tasks, a default behavior of releasing locks
> only on process shutdown should be adequate -- ant invocations not being
> long-lived by nature.

I was just thinking about locking the usual configure - resolve - retrieve - report, but I forgot that to run tests you need a classpath which needs to continue to exist on disk. You're right, releasing at shutdown seems to fit every usual use case.

> IDE plugins alter this a bit... but then, is it really safe to delete
> caches while a running IDE expects them to persist?

I think there will be some trouble with Eclipse users if IvyDE starts using locks. I remember IvyDE bugs on Windows where users would complain they cannot delete files because IvyDE via Eclipse was holding a reference to them. Since it is in an IDE and humans have only one brain, let's assume they only do one thing at a time, they are not ivy:cleaning via ant and running something in Eclipse at the same time, at least not on a daily basis. If they do, then thing will expectedly break, and the break is quite understandable.
And actually Eclipse is having a layer of cache upon the filesystem, which make errors on file search if the cache is not in sync, generate unsolvable bugs [1]. So I guess it will be OK if Eclipse show errors on such concurrent use case, users should be used to it.

To come back to the original topic, and to answer the use case about allowing to "safely" run an ivy:clean next to an ivy:resolve-and-others, for instance for the use case of a CI server, the IvyContext will not have an enough wide scope. The jvm shutdown hook should be better used, right ? And it should be optional so that in IDE like in IvyDE, locks can be disabled and won't be an issue.

>> I myself don't have much experience with file based lock mechanism between
>> different process. For instance what happens if a process holding a lock is
>> killed -9, or whatever which make the jvm not even able to trigger its
>> shutdown hooks ?
>> 
> 
> I would only propose to support this with NIO locks in use. These are
> implicitly released on file descriptor release, even when this is caused by
> SIGKILL, power termination, etc.

I didn't know, that is nice.

Nicolas

[1] https://issues.apache.org/jira/browse/IVYDE-302
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Cleanup callbacks in IvyContext?

[Charles Duffy <ch...@dyfis.net>]

On Thu, Jan 9, 2014 at 12:17 PM, Nicolas Lalevée <nicolas.lalevee@hibnet.org
> wrote:

> Yep, a quite hard problem to resolve. At the ant task level I don't know
> how you could nail it without asking the end user to call some ivy:lock
> task.
>

I'd like to argue that for ant tasks, a default behavior of releasing locks
only on process shutdown should be adequate -- ant invocations not being
long-lived by nature.

IDE plugins alter this a bit... but then, is it really safe to delete
caches while a running IDE expects them to persist?


> I myself don't have much experience with file based lock mechanism between
> different process. For instance what happens if a process holding a lock is
> killed -9, or whatever which make the jvm not even able to trigger its
> shutdown hooks ?
>

I would only propose to support this with NIO locks in use. These are
implicitly released on file descriptor release, even when this is caused by
SIGKILL, power termination, etc.

Re: Cleanup callbacks in IvyContext?

[Nicolas Lalevée <ni...@hibnet.org>]

Le 8 janv. 2014 à 16:47, Charles Duffy <ch...@dyfis.net> a écrit :

> Apologies about the delay before providing a full reply -- I was
> unexpectedly stuck in Chicago without my laptop.

No worries, I am myself usually random in my response time, due to my sparse free time.

> The problem with the LockStrategy interface is that it's too fine-grained
> for the use case wherein a read lock needs to be held for the duration of a
> high-level operation.
> 
> Consider the case where a user calls ivy:resolve, ivy:retrieve and
> ivy:report in sequence. An ivy:clean run by an external process between any
> of these operations would cause a failure, and would be prevented by any
> proper repository-wide read-lock mechanism.
> 
> The above _does_ indicate a potential problem with the proposal I opened
> this thread with -- inasmuch as it doesn't provide a way to hold locks
> spanning multiple high-level operations while still putting each of those
> operations in a separate IvyContext. I wonder if it might make sense to
> allow a given member in the stack to indicate that it should hold locks its
> children. For ant, being generally a short-lived process, it might make
> sense for the parent context to be the read lock holder.
> 
> Does this clarify the issue I'm trying to solve?

Yep, a quite hard problem to resolve. At the ant task level I don't know how you could nail it without asking the end user to call some ivy:lock task.

I myself don't have much experience with file based lock mechanism between different process. For instance what happens if a process holding a lock is killed -9, or whatever which make the jvm not even able to trigger its shutdown hooks ?

Nicolas

> 
> 
> On Sat, Jan 4, 2014 at 8:06 AM, Nicolas Lalevée
> <ni...@hibnet.org>wrote:
> 
>> 
>> Le 3 janv. 2014 à 22:28, Charles Duffy <ch...@dyfis.net> a écrit :
>> 
>>> Howdy, all --
>>> 
>>> I'm trying to strengthen Ivy's locking system to make it strong enough to
>>> allow ivy:clean at arbitrary times on systems which can have other
>> actions
>>> making use of the same shared caches.
>>> 
>>> There are a few requirements to make this happen while still allowing
>>> multiple resolves (and like operations) to occur concurrently. One of
>> those
>>> is maintaining nonexclusive read locks (as opposed to the write locks
>> which
>>> are currently supported), and cleaning them up when necessary. For
>>> ease-of-implementation, I'm currently proposing to only support this
>>> behavior when NIO locks (which implicitly support shared locking
>> semantics)
>>> are enabled.
>>> 
>>> To clean these locks up without requiring end-user code to be modified, I
>>> propose using the IvyContext stack -- allowing Runnables to be attached
>> to
>>> a stack element, and invoking them implicitly when the stack is popped.
>>> 
>>> 
>>> Because converting a read lock to a write lock is inherently prone to
>> race
>>> conditions, we might need to break backwards compatibility with respect
>> to
>>> ivy:clean, allowing this to be called only inside a context where no read
>>> operations have been done -- or breaking read lock semantics by dropping
>>> all read locks before grabbing the write lock.
>>> 
>>> 
>>> Thoughts?
>> 
>> My first blind though would be that IvyContext seems to be high level to
>> handle locking. Couldn't it be done by improving the LockStrategy interface
>> so that it also handles read locks ?
>> 
>> Nicolas
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
>> For additional commands, e-mail: dev-help@ant.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Cleanup callbacks in IvyContext?

[Charles Duffy <ch...@dyfis.net>]

Apologies about the delay before providing a full reply -- I was
unexpectedly stuck in Chicago without my laptop.

The problem with the LockStrategy interface is that it's too fine-grained
for the use case wherein a read lock needs to be held for the duration of a
high-level operation.

Consider the case where a user calls ivy:resolve, ivy:retrieve and
ivy:report in sequence. An ivy:clean run by an external process between any
of these operations would cause a failure, and would be prevented by any
proper repository-wide read-lock mechanism.

The above _does_ indicate a potential problem with the proposal I opened
this thread with -- inasmuch as it doesn't provide a way to hold locks
spanning multiple high-level operations while still putting each of those
operations in a separate IvyContext. I wonder if it might make sense to
allow a given member in the stack to indicate that it should hold locks its
children. For ant, being generally a short-lived process, it might make
sense for the parent context to be the read lock holder.

Does this clarify the issue I'm trying to solve?


On Sat, Jan 4, 2014 at 8:06 AM, Nicolas Lalevée
<ni...@hibnet.org>wrote:

>
> Le 3 janv. 2014 à 22:28, Charles Duffy <ch...@dyfis.net> a écrit :
>
> > Howdy, all --
> >
> > I'm trying to strengthen Ivy's locking system to make it strong enough to
> > allow ivy:clean at arbitrary times on systems which can have other
> actions
> > making use of the same shared caches.
> >
> > There are a few requirements to make this happen while still allowing
> > multiple resolves (and like operations) to occur concurrently. One of
> those
> > is maintaining nonexclusive read locks (as opposed to the write locks
> which
> > are currently supported), and cleaning them up when necessary. For
> > ease-of-implementation, I'm currently proposing to only support this
> > behavior when NIO locks (which implicitly support shared locking
> semantics)
> > are enabled.
> >
> > To clean these locks up without requiring end-user code to be modified, I
> > propose using the IvyContext stack -- allowing Runnables to be attached
> to
> > a stack element, and invoking them implicitly when the stack is popped.
> >
> >
> > Because converting a read lock to a write lock is inherently prone to
> race
> > conditions, we might need to break backwards compatibility with respect
> to
> > ivy:clean, allowing this to be called only inside a context where no read
> > operations have been done -- or breaking read lock semantics by dropping
> > all read locks before grabbing the write lock.
> >
> >
> > Thoughts?
>
> My first blind though would be that IvyContext seems to be high level to
> handle locking. Couldn't it be done by improving the LockStrategy interface
> so that it also handles read locks ?
>
> Nicolas
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>

Re: Cleanup callbacks in IvyContext?

[Charles Duffy <ch...@dyfis.net>]

This is part of why I wouldn't give it knowledge of locks specifically, but
simply Runnable callbacks to invoke on pop -- not specifically tied to a
single use case. This is not entirely unlike atExit calls available through
the JVM, the standard C library, etc.

Improving LockStrategy is a necessary component, to be sure -- but
something needs to tell it when a read lock is needed, and likewise when
its need is past.
On Jan 4, 2014 8:07 AM, "Nicolas Lalevée" <ni...@hibnet.org>
wrote:

>
> Le 3 janv. 2014 à 22:28, Charles Duffy <ch...@dyfis.net> a écrit :
>
> > Howdy, all --
> >
> > I'm trying to strengthen Ivy's locking system to make it strong enough to
> > allow ivy:clean at arbitrary times on systems which can have other
> actions
> > making use of the same shared caches.
> >
> > There are a few requirements to make this happen while still allowing
> > multiple resolves (and like operations) to occur concurrently. One of
> those
> > is maintaining nonexclusive read locks (as opposed to the write locks
> which
> > are currently supported), and cleaning them up when necessary. For
> > ease-of-implementation, I'm currently proposing to only support this
> > behavior when NIO locks (which implicitly support shared locking
> semantics)
> > are enabled.
> >
> > To clean these locks up without requiring end-user code to be modified, I
> > propose using the IvyContext stack -- allowing Runnables to be attached
> to
> > a stack element, and invoking them implicitly when the stack is popped.
> >
> >
> > Because converting a read lock to a write lock is inherently prone to
> race
> > conditions, we might need to break backwards compatibility with respect
> to
> > ivy:clean, allowing this to be called only inside a context where no read
> > operations have been done -- or breaking read lock semantics by dropping
> > all read locks before grabbing the write lock.
> >
> >
> > Thoughts?
>
> My first blind though would be that IvyContext seems to be high level to
> handle locking. Couldn't it be done by improving the LockStrategy interface
> so that it also handles read locks ?
>
> Nicolas
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>

Re: Cleanup callbacks in IvyContext?

[Nicolas Lalevée <ni...@hibnet.org>]

Le 3 janv. 2014 à 22:28, Charles Duffy <ch...@dyfis.net> a écrit :

> Howdy, all --
> 
> I'm trying to strengthen Ivy's locking system to make it strong enough to
> allow ivy:clean at arbitrary times on systems which can have other actions
> making use of the same shared caches.
> 
> There are a few requirements to make this happen while still allowing
> multiple resolves (and like operations) to occur concurrently. One of those
> is maintaining nonexclusive read locks (as opposed to the write locks which
> are currently supported), and cleaning them up when necessary. For
> ease-of-implementation, I'm currently proposing to only support this
> behavior when NIO locks (which implicitly support shared locking semantics)
> are enabled.
> 
> To clean these locks up without requiring end-user code to be modified, I
> propose using the IvyContext stack -- allowing Runnables to be attached to
> a stack element, and invoking them implicitly when the stack is popped.
> 
> 
> Because converting a read lock to a write lock is inherently prone to race
> conditions, we might need to break backwards compatibility with respect to
> ivy:clean, allowing this to be called only inside a context where no read
> operations have been done -- or breaking read lock semantics by dropping
> all read locks before grabbing the write lock.
> 
> 
> Thoughts?

My first blind though would be that IvyContext seems to be high level to handle locking. Couldn't it be done by improving the LockStrategy interface so that it also handles read locks ?

Nicolas



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org