You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Julian Foad <ju...@btopenworld.com> on 2014/06/23 10:03:13 UTC
Issue #4162: make svn status fix timestamp mismatches in meta-data
Markus Schaber wrote (in thread "controversial issues in the tracker"):
> * 4162 make svn status fix timestamp mismatches in meta-data
> http://subversion.tigris.org/issues/show_bug.cgi?id=4162
>
> The controversial issue here is whether it is a good idea to have "svn
> status" modify the meta data - until now, it intentionally only has a
> read-only lock on the working copy, and changing that may have compatibility
> side effects.
>
> Currently, correcting the metadata for files with updated timestamps (but
> unmodified content) is only available as a side-effect of other commands
> (update, cleanup, etc.) which all have other side-effects.
>
> Thus, alternative solutions may be to pass an option to status which explicitly
> allows modification of the meta data, or create a command (or option to svn
> cleanup) which only fixes the timestamps without other side effects.
>
> My personal suggestion is to add a flag to svn status.
I'm with Stefan Sperling: the user doesn't want to know or care about this. It's just caching. We don't need another flag or command or explicit action to control it. Subversion should simply update the metadata.
I don't know exactly how to do it, but I think it should be done.
- Julian
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Johan Corveleyn <jc...@gmail.com>.
On Mon, Jun 23, 2014 at 2:08 PM, Stefan Sperling <st...@elego.de> wrote:
> On Mon, Jun 23, 2014 at 01:55:32PM +0200, Branko Čibej wrote:
>> There's a world of difference here. The former is an error that prevents
>> you from using the working copy. The latter case doesn't prevent anything.
>
> This discussion is about a performance issue, not a behavioural one.
>
>> BTW, don't you find it a bit strange to get error (or warning) messages
>> from our libraries that refer to the implementation of our command-line
>> client?
>
> That doesn't sound at all like what Johan is asking for.
>
> If status information returned by the library to clients said "the timestamp
> of this node doesn't match the recorded one" clients could then deal with
> this in any manner they wished.
Indeed. BTW, I just copy-pasted the error / warning that you get when
you run into working copy locks (that warning explicitly mentions 'svn
cleanup'). I don't know if this warning comes from the library or from
the command line client. If the former, then I agree it's not ideal as
it is.
Another suggestion in this context: a '--report' option for 'svn
status' would also be some help for users (or their administrators) to
diagnose (performance) problems. It could generate a more extensive
report of the state of the working copy, including the amount of
mismatching metadata.
Anyone who has done 1st line support for svn users for a while already
knows the phrase: "try running 'svn cleanup' and see if the situation
improves", but it's always a shot in the dark when users complain
about svn performance. Right now there is no (simple, free,
ubiquitous) tool that can report on the amount of metadata-mismatch.
You just have to guess (or perform some kind of clever query on wc.db
yourself ...(?)). Would be nice if one could just say "run 'svn status
--report' and send me the report, so we can see what's causing the
problem".
Maybe that's a kind of tool that doesn't belong in core svn (but OTOH,
if users have to download/install a separate tool for this, they
probably won't), but anyway, putting it out here.
--
Johan
Re: Issue #4162: make svn status fix timestamp mismatches in
meta-data
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 23, 2014 at 01:55:32PM +0200, Branko Čibej wrote:
> There's a world of difference here. The former is an error that prevents
> you from using the working copy. The latter case doesn't prevent anything.
This discussion is about a performance issue, not a behavioural one.
> BTW, don't you find it a bit strange to get error (or warning) messages
> from our libraries that refer to the implementation of our command-line
> client?
That doesn't sound at all like what Johan is asking for.
If status information returned by the library to clients said "the timestamp
of this node doesn't match the recorded one" clients could then deal with
this in any manner they wished.
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Branko Čibej <br...@wandisco.com>.
On 23.06.2014 13:34, Johan Corveleyn wrote:
> Why not? We can give them a hint, right? Exactly like "Working copy
> 'bla' locked. Run 'svn cleanup' to remove locks." we could say
> "Working copy 'bla' has 10% incorrect metadata. Run 'svn cleanup
> --sync-metadata' to optimize your working copy".
There's a world of difference here. The former is an error that prevents
you from using the working copy. The latter case doesn't prevent anything.
BTW, don't you find it a bit strange to get error (or warning) messages
from our libraries that refer to the implementation of our command-line
client?
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Johan Corveleyn <jc...@gmail.com>.
On Mon, Jun 23, 2014 at 12:03 PM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
>> Sent: maandag 23 juni 2014 11:41
>> To: Bert Huijben
>> Cc: Julian Foad; Markus Schaber; Subversion Dev
>> Subject: Re: Issue #4162: make svn status fix timestamp mismatches in meta-
>> data
>>
>> On Mon, Jun 23, 2014 at 11:08 AM, Bert Huijben <be...@qqmail.nl> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Julian Foad [mailto:julianfoad@btopenworld.com]
>> >> Sent: maandag 23 juni 2014 10:03
>> >> To: Markus Schaber
>> >> Cc: Subversion Dev (dev@subversion.apache.org)
>> >> Subject: Issue #4162: make svn status fix timestamp mismatches in meta-
>> >> data
>> >>
>> >> Markus Schaber wrote (in thread "controversial issues in the tracker"):
>> >>
>> >> > * 4162 make svn status fix timestamp mismatches in meta-data
>> >> > http://subversion.tigris.org/issues/show_bug.cgi?id=4162
>> >> >
>> >> > The controversial issue here is whether it is a good idea to have "svn
>> >> > status" modify the meta data - until now, it intentionally only has a
>> >> > read-only lock on the working copy, and changing that may have
>> >> compatibility
>> >> > side effects.
>> >> >
>> >> > Currently, correcting the metadata for files with updated timestamps
>> (but
>> >> > unmodified content) is only available as a side-effect of other
>> commands
>> >> > (update, cleanup, etc.) which all have other side-effects.
>> >> >
>> >> > Thus, alternative solutions may be to pass an option to status which
>> >> explicitly
>> >> > allows modification of the meta data, or create a command (or option to
>> >> svn
>> >> > cleanup) which only fixes the timestamps without other side effects.
>> >> >
>> >> > My personal suggestion is to add a flag to svn status.
>> >>
>> >> I'm with Stefan Sperling: the user doesn't want to know or care about this.
>> >> It's just caching. We don't need another flag or command or explicit action
>> to
>> >> control it. Subversion should simply update the metadata.
>> >
>> >
>> > Changing the recorded timestamps would require obtaining a write lock
>> while performing a status walk... which is a breaking change for any client
>> that wants to do things concurrently.
>> >
>> > -1 on 'just doing it'
>> >
>> > There is a good reason the current code only updates timestamps when it
>> has a write lock. Otherwise it would break concurrent operations.
>> >
>> >
>> > Note that 1.9 already supports an api to fix timestamps at the api level: you
>> can now run cleanup without breaking the write locks.
>> >
>> >
>>
>>
>> Another option is to let 'svn status' report on the amount of
>> mismatching timestamps, possibly controlled by some verbosity flag.
>> That doesn't require a write lock, and 'svn status' can gather this
>> information for free (at the cost of some memory for keeping the
>> stats) during its status walk: it already enters a specific code path
>> when a file has mismatching metadata yet its content is unmodified
>> (see the patch to subversion/libsvn_wc/questions.c mentioned in [1]).
>>
>> At least this has the possibility to give the user some feedback on
>> the amount of "metadata-mismatch" in his working copy. Right now, the
>> user could be working for a long time with a working copy that has
>> almost 100% mismatching metadata. It will be dog-slow, and the user
>> will not know why.
>>
>> Note that timestamp-mismatch does not only happen because of SVNKit
>> (see [1]), but also for instance when a user copies his entire working
>> copy accross filesystems. This happens quite frequently: when users
>> get a new pc, they often want to just copy their old working copy to
>> the new disk. At least on Windows, this yields 100% timestamp
>> mismatches because all last-mod-timestamps of the files are changed
>> (also on unix, unless they use -p or a similar flag to preserve
>> last-mod-timestamps).
>>
>> Something like "Warning: working copy has 95% incorrect metadata,
>> resulting in slow performance. Please run 'svn bla' to make it fast
>> again." might help.
>>
>> [1] http://issues.tmatesoft.com/issue/SVNKIT-315
>
> Why should we try to fix an SvnKit problem (of not recording the actual timestamp, but only an arbitrary rounded value) in Subversion itself?
Bert, did you read my mail?
Specifically:
>> Note that timestamp-mismatch does not only happen because of SVNKit
>> (see [1]), but also for instance when a user copies his entire working
>> copy accross filesystems. This happens quite frequently: when users
>> get a new pc, they often want to just copy their old working copy to
>> the new disk. At least on Windows, this yields 100% timestamp
>> mismatches because all last-mod-timestamps of the files are changed
>> (also on unix, unless they use -p or a similar flag to preserve
>> last-mod-timestamps).
Most colleagues of mine that run into this problem run into it because
of the above, not because of SVNKit.
Yes, if working copies are expensive to build up, and they contain
some local mods, people will usually copy the working copy entirely,
instead of performing a new checkout.
And apart from that: I am frankly quite tired of this long-standing
attitude saying "it happens outside of our little island, so it's not
our problem". The simple fact is: *our* users experience perforrmance
problems (note that SVNKit doesn't get hurt by this itself -- only
native clients) ... what can *we* do to improve the situation?
Pointing fingers doesn't help anyone.
Anyway, as I said above, there are plenty of other possible causes of
100% mismatching metadata.
> I don't think we should expose a value like that in our higher layers... Ordinary users wouldn't know what to do with it anyway.
Why not? We can give them a hint, right? Exactly like "Working copy
'bla' locked. Run 'svn cleanup' to remove locks." we could say
"Working copy 'bla' has 10% incorrect metadata. Run 'svn cleanup
--sync-metadata' to optimize your working copy".
> And doing something like running a recursive 'svn commit' (or any other status performing code with a write lock) on your working copy will already fix all your timestamps for you.
How often do real users perform a recursive 'svn commit' from the root
of their working copy, especially when they're using an IDE
integration of something like that?
--
Johan
RE: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
> Sent: maandag 23 juni 2014 11:41
> To: Bert Huijben
> Cc: Julian Foad; Markus Schaber; Subversion Dev
> Subject: Re: Issue #4162: make svn status fix timestamp mismatches in meta-
> data
>
> On Mon, Jun 23, 2014 at 11:08 AM, Bert Huijben <be...@qqmail.nl> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Julian Foad [mailto:julianfoad@btopenworld.com]
> >> Sent: maandag 23 juni 2014 10:03
> >> To: Markus Schaber
> >> Cc: Subversion Dev (dev@subversion.apache.org)
> >> Subject: Issue #4162: make svn status fix timestamp mismatches in meta-
> >> data
> >>
> >> Markus Schaber wrote (in thread "controversial issues in the tracker"):
> >>
> >> > * 4162 make svn status fix timestamp mismatches in meta-data
> >> > http://subversion.tigris.org/issues/show_bug.cgi?id=4162
> >> >
> >> > The controversial issue here is whether it is a good idea to have "svn
> >> > status" modify the meta data - until now, it intentionally only has a
> >> > read-only lock on the working copy, and changing that may have
> >> compatibility
> >> > side effects.
> >> >
> >> > Currently, correcting the metadata for files with updated timestamps
> (but
> >> > unmodified content) is only available as a side-effect of other
> commands
> >> > (update, cleanup, etc.) which all have other side-effects.
> >> >
> >> > Thus, alternative solutions may be to pass an option to status which
> >> explicitly
> >> > allows modification of the meta data, or create a command (or option to
> >> svn
> >> > cleanup) which only fixes the timestamps without other side effects.
> >> >
> >> > My personal suggestion is to add a flag to svn status.
> >>
> >> I'm with Stefan Sperling: the user doesn't want to know or care about this.
> >> It's just caching. We don't need another flag or command or explicit action
> to
> >> control it. Subversion should simply update the metadata.
> >
> >
> > Changing the recorded timestamps would require obtaining a write lock
> while performing a status walk... which is a breaking change for any client
> that wants to do things concurrently.
> >
> > -1 on 'just doing it'
> >
> > There is a good reason the current code only updates timestamps when it
> has a write lock. Otherwise it would break concurrent operations.
> >
> >
> > Note that 1.9 already supports an api to fix timestamps at the api level: you
> can now run cleanup without breaking the write locks.
> >
> >
>
>
> Another option is to let 'svn status' report on the amount of
> mismatching timestamps, possibly controlled by some verbosity flag.
> That doesn't require a write lock, and 'svn status' can gather this
> information for free (at the cost of some memory for keeping the
> stats) during its status walk: it already enters a specific code path
> when a file has mismatching metadata yet its content is unmodified
> (see the patch to subversion/libsvn_wc/questions.c mentioned in [1]).
>
> At least this has the possibility to give the user some feedback on
> the amount of "metadata-mismatch" in his working copy. Right now, the
> user could be working for a long time with a working copy that has
> almost 100% mismatching metadata. It will be dog-slow, and the user
> will not know why.
>
> Note that timestamp-mismatch does not only happen because of SVNKit
> (see [1]), but also for instance when a user copies his entire working
> copy accross filesystems. This happens quite frequently: when users
> get a new pc, they often want to just copy their old working copy to
> the new disk. At least on Windows, this yields 100% timestamp
> mismatches because all last-mod-timestamps of the files are changed
> (also on unix, unless they use -p or a similar flag to preserve
> last-mod-timestamps).
>
> Something like "Warning: working copy has 95% incorrect metadata,
> resulting in slow performance. Please run 'svn bla' to make it fast
> again." might help.
>
> [1] http://issues.tmatesoft.com/issue/SVNKIT-315
Why should we try to fix an SvnKit problem (of not recording the actual timestamp, but only an arbitrary rounded value) in Subversion itself?
I don't think we should expose a value like that in our higher layers... Ordinary users wouldn't know what to do with it anyway.
And doing something like running a recursive 'svn commit' (or any other status performing code with a write lock) on your working copy will already fix all your timestamps for you.
If you have 95% invalid recorded timestamps you have bigger problems... And those problems should (if they are related to some Subversion client) be fixed at the origin (SmartSVN?), not at all other locations.
(FYI: Recent versions of SmartSVN now use JavaHL, see http://www.wandisco.com/smartsvn/changelog.txt)
Bert
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Johan Corveleyn <jc...@gmail.com>.
On Mon, Jun 23, 2014 at 11:08 AM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Julian Foad [mailto:julianfoad@btopenworld.com]
>> Sent: maandag 23 juni 2014 10:03
>> To: Markus Schaber
>> Cc: Subversion Dev (dev@subversion.apache.org)
>> Subject: Issue #4162: make svn status fix timestamp mismatches in meta-
>> data
>>
>> Markus Schaber wrote (in thread "controversial issues in the tracker"):
>>
>> > * 4162 make svn status fix timestamp mismatches in meta-data
>> > http://subversion.tigris.org/issues/show_bug.cgi?id=4162
>> >
>> > The controversial issue here is whether it is a good idea to have "svn
>> > status" modify the meta data - until now, it intentionally only has a
>> > read-only lock on the working copy, and changing that may have
>> compatibility
>> > side effects.
>> >
>> > Currently, correcting the metadata for files with updated timestamps (but
>> > unmodified content) is only available as a side-effect of other commands
>> > (update, cleanup, etc.) which all have other side-effects.
>> >
>> > Thus, alternative solutions may be to pass an option to status which
>> explicitly
>> > allows modification of the meta data, or create a command (or option to
>> svn
>> > cleanup) which only fixes the timestamps without other side effects.
>> >
>> > My personal suggestion is to add a flag to svn status.
>>
>> I'm with Stefan Sperling: the user doesn't want to know or care about this.
>> It's just caching. We don't need another flag or command or explicit action to
>> control it. Subversion should simply update the metadata.
>
>
> Changing the recorded timestamps would require obtaining a write lock while performing a status walk... which is a breaking change for any client that wants to do things concurrently.
>
> -1 on 'just doing it'
>
> There is a good reason the current code only updates timestamps when it has a write lock. Otherwise it would break concurrent operations.
>
>
> Note that 1.9 already supports an api to fix timestamps at the api level: you can now run cleanup without breaking the write locks.
>
>
Another option is to let 'svn status' report on the amount of
mismatching timestamps, possibly controlled by some verbosity flag.
That doesn't require a write lock, and 'svn status' can gather this
information for free (at the cost of some memory for keeping the
stats) during its status walk: it already enters a specific code path
when a file has mismatching metadata yet its content is unmodified
(see the patch to subversion/libsvn_wc/questions.c mentioned in [1]).
At least this has the possibility to give the user some feedback on
the amount of "metadata-mismatch" in his working copy. Right now, the
user could be working for a long time with a working copy that has
almost 100% mismatching metadata. It will be dog-slow, and the user
will not know why.
Note that timestamp-mismatch does not only happen because of SVNKit
(see [1]), but also for instance when a user copies his entire working
copy accross filesystems. This happens quite frequently: when users
get a new pc, they often want to just copy their old working copy to
the new disk. At least on Windows, this yields 100% timestamp
mismatches because all last-mod-timestamps of the files are changed
(also on unix, unless they use -p or a similar flag to preserve
last-mod-timestamps).
Something like "Warning: working copy has 95% incorrect metadata,
resulting in slow performance. Please run 'svn bla' to make it fast
again." might help.
[1] http://issues.tmatesoft.com/issue/SVNKIT-315
--
Johan
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Branko Čibej <br...@wandisco.com>.
On 23.06.2014 11:43, Julian Foad wrote:
> Branko Čibej wrote:
>> Bert Huijben wrote:
>>> Changing the recorded timestamps would require obtaining a write lock while
>>> performing a status walk... which is a breaking change for any client that
>>> wants to do things concurrently. -1 on 'just doing it' There is a good reason
>>> the current code only updates timestamps when it has a write lock. Otherwise
>>> it would break concurrent operations.
>> I absolutely agree. 'svn status' has never write-locked the working
>> copy. There are clients out there that rely on this documented fact.
>> One of them is TortoiseSVN.
>
> I don't dispute that, but there seems to be an assumption there that if it takes out a lock on this metadata then that lock will necessarily have to be "the [present] WC write lock" which will therefore block operations such as 'update'. Is it inconceivable that some different level, granularity or kind of locking strategy could be used?
We already have fine-grained locking on the WC: we lock subtrees of the
working copy, not the whole working copy. One can perform an update on
one subtree and status on another subtree, in parallel, as long as the
subtrees are disjunct; and the locks themselves are multiple-reader,
single-writer.
Inventing more fine-grained locking is of course possible, but increases
the problem of avoiding deadlocks quite out of proportion to the benefit
gained.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Julian Foad <ju...@btopenworld.com>.
Branko Čibej wrote:
> Bert Huijben wrote:
>> Changing the recorded timestamps would require obtaining a write lock while
>> performing a status walk... which is a breaking change for any client that
>> wants to do things concurrently. -1 on 'just doing it' There is a good reason
>> the current code only updates timestamps when it has a write lock. Otherwise
>> it would break concurrent operations.
>
> I absolutely agree. 'svn status' has never write-locked the working
> copy. There are clients out there that rely on this documented fact.
> One of them is TortoiseSVN.
I don't dispute that, but there seems to be an assumption there that if it takes out a lock on this metadata then that lock will necessarily have to be "the [present] WC write lock" which will therefore block operations such as 'update'. Is it inconceivable that some different level, granularity or kind of locking strategy could be used?
- Julian
Re: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Branko Čibej <br...@wandisco.com>.
On 23.06.2014 11:08, Bert Huijben wrote:
>
>> -----Original Message-----
>> From: Julian Foad [mailto:julianfoad@btopenworld.com]
>> Sent: maandag 23 juni 2014 10:03
>> To: Markus Schaber
>> Cc: Subversion Dev (dev@subversion.apache.org)
>> Subject: Issue #4162: make svn status fix timestamp mismatches in meta-
>> data
>>
>> Markus Schaber wrote (in thread "controversial issues in the tracker"):
>>
>>> * 4162 make svn status fix timestamp mismatches in meta-data
>>> http://subversion.tigris.org/issues/show_bug.cgi?id=4162
>>>
>>> The controversial issue here is whether it is a good idea to have "svn
>>> status" modify the meta data - until now, it intentionally only has a
>>> read-only lock on the working copy, and changing that may have
>> compatibility
>>> side effects.
>>>
>>> Currently, correcting the metadata for files with updated timestamps (but
>>> unmodified content) is only available as a side-effect of other commands
>>> (update, cleanup, etc.) which all have other side-effects.
>>>
>>> Thus, alternative solutions may be to pass an option to status which
>> explicitly
>>> allows modification of the meta data, or create a command (or option to
>> svn
>>> cleanup) which only fixes the timestamps without other side effects.
>>>
>>> My personal suggestion is to add a flag to svn status.
>> I'm with Stefan Sperling: the user doesn't want to know or care about this.
>> It's just caching. We don't need another flag or command or explicit action to
>> control it. Subversion should simply update the metadata.
>
> Changing the recorded timestamps would require obtaining a write lock while performing a status walk... which is a breaking change for any client that wants to do things concurrently.
>
> -1 on 'just doing it'
>
> There is a good reason the current code only updates timestamps when it has a write lock. Otherwise it would break concurrent operations.
I absolutely agree. 'svn status' has never write-locked the working
copy. There are clients out there that rely on this documented fact. One
of them is TortoiseSVN.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
RE: Issue #4162: make svn status fix timestamp mismatches in meta-data
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Julian Foad [mailto:julianfoad@btopenworld.com]
> Sent: maandag 23 juni 2014 10:03
> To: Markus Schaber
> Cc: Subversion Dev (dev@subversion.apache.org)
> Subject: Issue #4162: make svn status fix timestamp mismatches in meta-
> data
>
> Markus Schaber wrote (in thread "controversial issues in the tracker"):
>
> > * 4162 make svn status fix timestamp mismatches in meta-data
> > http://subversion.tigris.org/issues/show_bug.cgi?id=4162
> >
> > The controversial issue here is whether it is a good idea to have "svn
> > status" modify the meta data - until now, it intentionally only has a
> > read-only lock on the working copy, and changing that may have
> compatibility
> > side effects.
> >
> > Currently, correcting the metadata for files with updated timestamps (but
> > unmodified content) is only available as a side-effect of other commands
> > (update, cleanup, etc.) which all have other side-effects.
> >
> > Thus, alternative solutions may be to pass an option to status which
> explicitly
> > allows modification of the meta data, or create a command (or option to
> svn
> > cleanup) which only fixes the timestamps without other side effects.
> >
> > My personal suggestion is to add a flag to svn status.
>
> I'm with Stefan Sperling: the user doesn't want to know or care about this.
> It's just caching. We don't need another flag or command or explicit action to
> control it. Subversion should simply update the metadata.
Changing the recorded timestamps would require obtaining a write lock while performing a status walk... which is a breaking change for any client that wants to do things concurrently.
-1 on 'just doing it'
There is a good reason the current code only updates timestamps when it has a write lock. Otherwise it would break concurrent operations.
Note that 1.9 already supports an api to fix timestamps at the api level: you can now run cleanup without breaking the write locks.
Bert