You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Shahaf <d....@daniel.shahaf.name> on 2011/07/25 00:22:44 UTC
Purging pristines
I've noticed several times that 'svn cleanup' trims down the
.svn/pristine/ hierarchy:
[[[
svn/t1% ls .svn/pristine/*/*base | wc -l
2046
svn/t1% $svn cleanup
svn/t1% ls .svn/pristine/*/*base | wc -l
1835
]]]
Are pristines purged at other times? If not, should we release-note
a recommendation to run 'cleanup' periodically?
RE: Purging pristines
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Markus Schaber [mailto:m.schaber@3s-software.com]
> Sent: donderdag 28 juli 2011 11:20
> To: Julian Foad; Daniel Shahaf
> Cc: dev@subversion.apache.org
> Subject: AW: Purging pristines
>
> Hi,
>
> Von: Julian Foad [mailto:julian.foad@wandisco.com]
>
> > And IMO we should work on purging unused pristines automatically, for
> > 1.7.x if possible.
>
> I agree here that automatic purging of pristines is very useful, although you
> can do 1.7.0 without it for now (as manual purging is possible).
>
> > I'd like to see purging at a fairly fine-grained level. Does anyone have
> > insight about how to do this? What's in my head is that every WC API
> > operation that could possibly leave unused pristines should attempt to
> > purge unused pristines before returning. The operations that could leave
> > unused pristines are basically 'update' and 'switch', although there may
> > be some lower level WC API calls that can do so too. The set of pristines
> > to be purged could be either all that are currently unused, or just those
> > that became unused during the call.
>
> AFAICS, a "commit" operation may also produce unused pristines.
And at least crop, exclude, revert and delete can also produce unused pristines.
(Conceptually the first two by changing the BASE tree, the other two by changing the WORKING tree).
When we release pristines doesn't have to be the same between 'svn' and the other clients if we just create a libsvn_client function that cleans up pristines.
(And which doesn't break locks)
I think we should have at least some pristine cleanup which doesn't require user confirmation to verify if it is safe to steal locks.
I'm -1 on recommending users to script a 'svn cleanup' in their common usage patterns.
Bert
Re: AW: Purging pristines
Posted by Julian Foad <ju...@wandisco.com>.
Markus Schaber wrote:
> Von: Julian Foad [mailto:julian.foad@wandisco.com]
> > And IMO we should work on purging unused pristines automatically, for
> > 1.7.x if possible.
>
> I agree here that automatic purging of pristines is very useful,
> although you can do 1.7.0 without it for now (as manual purging is
> possible).
>
> > I'd like to see purging at a fairly fine-grained level. Does anyone have
> > insight about how to do this? What's in my head is that every WC API
> > operation that could possibly leave unused pristines should attempt to
> > purge unused pristines before returning. The operations that could leave
> > unused pristines are basically 'update' and 'switch', although there may
> > be some lower level WC API calls that can do so too. The set of pristines
> > to be purged could be either all that are currently unused, or just those
> > that became unused during the call.
>
> AFAICS, a "commit" operation may also produce unused pristines.
Oh yes, thanks.
- Julian
> > When I mentioned this before there was some sentiment expressed that the
> > delay involved in purging a large set of files could be unacceptably large
> > to a GUI client, but I think automatic purging is important and so we have
> > to find a way to make it acceptable, whether that involves making it even
> > finer grained or adding progress notification for it or something else.
>
> Progress notification is always good. :-)
>
> And maybe it could be made optional (but enabled by default), so
> clients which perform several operations in a row or are performance
> critical can delay the purging.
>
> Markus.
AW: Purging pristines
Posted by Markus Schaber <m....@3s-software.com>.
Hi,
Von: Julian Foad [mailto:julian.foad@wandisco.com]
> And IMO we should work on purging unused pristines automatically, for
> 1.7.x if possible.
I agree here that automatic purging of pristines is very useful, although you can do 1.7.0 without it for now (as manual purging is possible).
> I'd like to see purging at a fairly fine-grained level. Does anyone have
> insight about how to do this? What's in my head is that every WC API
> operation that could possibly leave unused pristines should attempt to
> purge unused pristines before returning. The operations that could leave
> unused pristines are basically 'update' and 'switch', although there may
> be some lower level WC API calls that can do so too. The set of pristines
> to be purged could be either all that are currently unused, or just those
> that became unused during the call.
AFAICS, a "commit" operation may also produce unused pristines.
> When I mentioned this before there was some sentiment expressed that the
> delay involved in purging a large set of files could be unacceptably large
> to a GUI client, but I think automatic purging is important and so we have
> to find a way to make it acceptable, whether that involves making it even
> finer grained or adding progress notification for it or something else.
Progress notification is always good. :-)
And maybe it could be made optional (but enabled by default), so clients which perform several operations in a row or are performance critical can delay the purging.
Markus.
Re: Purging pristines
Posted by Julian Foad <ju...@wandisco.com>.
Daniel Shahaf wrote:
> Do we need to add an 'svn cleanup --include-externals' flag? It's an
> API change so if we need it we need to add it now.
I suggest that although the issue is annoying it's not critical enough
to hold up 1.7.0, and we should be making the WC management code delete
its unused pristines automatically in a point release soon so we do not
need to make 'cleanup' recurse into externals.
- Julian
Re: Purging pristines
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Do we need to add an 'svn cleanup --include-externals' flag? It's an
API change so if we need it we need to add it now.
Konstantin Kolinko wrote on Mon, Oct 03, 2011 at 15:50:30 +0400:
> 2. The problem is worse than it seemed:
> svn cleanup does not help for complex wcs, because it does not
> recurse into externals!
>
>
> 3. Note, that the space wasted by unused pristines is not proportional
> to change size, but proportional to (number of changed files) *
> (average size of a file). It grows fast.
Re: Purging pristines
Posted by Philip Martin <ph...@wandisco.com>.
"Bert Huijben" <be...@qqmail.nl> writes:
>> -----Original Message-----
>> From: Philip Martin [mailto:philip.martin@wandisco.com]
>>
>> It's not hard if you have some sort of shell programming language, look
>> for lines starting X in the output of 'svn st --ignore-externals' and
>> run cleanup on the paths in those lines. I suppose it's harder to do
>> that sort of thing on Windows than Unix.
>
> That assumes that directory externals are directly below versioned
> directories, which doesn't have to be the case.
In that case run 'svn status' and look for lines "Performing status on
external item" and run cleanup on the paths in those lines.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Purging pristines
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Philip Martin [mailto:philip.martin@wandisco.com]
> Sent: maandag 3 oktober 2011 14:33
> To: Konstantin Kolinko
> Cc: dev
> Subject: Re: Purging pristines
>
> Konstantin Kolinko <kn...@gmail.com> writes:
>
> > It is a nightmare to have to list all my dir externals explicitly to
> > run cleanup on them.
>
> It's not hard if you have some sort of shell programming language, look
> for lines starting X in the output of 'svn st --ignore-externals' and
> run cleanup on the paths in those lines. I suppose it's harder to do
> that sort of thing on Windows than Unix.
That assumes that directory externals are directly below versioned
directories, which doesn't have to be the case.
Bert
Re: Purging pristines
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Konstantin Kolinko wrote on Mon, Oct 03, 2011 at 17:27:16 +0400:
> 2011/10/3 Philip Martin <ph...@wandisco.com>:
> > Konstantin Kolinko <kn...@gmail.com> writes:
> >
> >> It is a nightmare to have to list all my dir externals explicitly to
> >> run cleanup on them.
> >
> > It's not hard if you have some sort of shell programming language, look
> > for lines starting X in the output of 'svn st --ignore-externals' and
> > run cleanup on the paths in those lines. I suppose it's harder to do
> > that sort of thing on Windows than Unix.
> >
>
> Yes, I am mostly on Windows.
>
> I know how to do tricks, but it is harder to teach users that they
> have to use shell programming here.
Easy workaround: include a shell/batch script in the root of trunk that
runs cleanup for all externals in trunk.
Re: Purging pristines
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/10/3 Philip Martin <ph...@wandisco.com>:
> Konstantin Kolinko <kn...@gmail.com> writes:
>
>> It is a nightmare to have to list all my dir externals explicitly to
>> run cleanup on them.
>
> It's not hard if you have some sort of shell programming language, look
> for lines starting X in the output of 'svn st --ignore-externals' and
> run cleanup on the paths in those lines. I suppose it's harder to do
> that sort of thing on Windows than Unix.
>
Yes, I am mostly on Windows.
I know how to do tricks, but it is harder to teach users that they
have to use shell programming here.
2011/10/3 Daniel Shahaf <d....@daniel.shahaf.name>:
> Do we need to add an 'svn cleanup --include-externals' flag? It's an
> API change so if we need it we need to add it now.
I personally would prefer that svn cleanup recursed into externals automatically
(as does svn up etc,) and to have "--exclude-externals" flag.
TortoiseSVN already has this recursing feature, so either it is already in API,
or Stefan has implemented a clever workaround around it.
Best regards,
Konstantin Kolinko
Re: Purging pristines
Posted by Philip Martin <ph...@wandisco.com>.
Konstantin Kolinko <kn...@gmail.com> writes:
> It is a nightmare to have to list all my dir externals explicitly to
> run cleanup on them.
It's not hard if you have some sort of shell programming language, look
for lines starting X in the output of 'svn st --ignore-externals' and
run cleanup on the paths in those lines. I suppose it's harder to do
that sort of thing on Windows than Unix.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Purging pristines
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/7/28 Daniel Shahaf <da...@elego.de>:
> Julian Foad wrote on Thu, Jul 28, 2011 at 10:09:28 +0100:
>> On Mon, 2011-07-25, Daniel Shahaf wrote:
>> > I've noticed several times that 'svn cleanup' trims down the
>> > .svn/pristine/ hierarchy:
>> [...]
>> > Are pristines purged at other times?
>>
>> No.
>>
>> > If not, should we release-note
>> > a recommendation to run 'cleanup' periodically?
>>
>> Yes.
>>
>
> r1151946.
>
>> And IMO we should work on purging unused pristines automatically, for
>> 1.7.x if possible.
>>
Continuing this thread from end of July...
1. There is no issue in issue tracker for this problem (purging unused
pristines).
and Release notes say "see issue #XXX"
2. The problem is worse than it seemed:
svn cleanup does not help for complex wcs, because it does not
recurse into externals!
If I use TortoiseSVN 1.7, there is a checkbox "[x] Include externals"
n this command and it is checked on by default.
But in the command line client there is no support for that: neither
it does recurse automatically, nor it has options to enable recursion
(it does not support --depth argument or anything like
--exclude-externals, --include-externals).
This cleanup issue is known but is treated as enhancement and is unscheduled:
http://subversion.tigris.org/issues/show_bug.cgi?id=2325
It is a nightmare to have to list all my dir externals explicitly to
run cleanup on them.
3. Note, that the space wasted by unused pristines is not proportional
to change size, but proportional to (number of changed files) *
(average size of a file). It grows fast.
I am running nightly build built from 1.7.x branch by TortoiseSVN,
svn, version 1.7.0-dev (under development)
compiled Sep 28 2011, 00:15:41
Best regards,
Konstantin Kolinko
Re: Purging pristines
Posted by Daniel Shahaf <da...@elego.de>.
Julian Foad wrote on Thu, Jul 28, 2011 at 10:09:28 +0100:
> On Mon, 2011-07-25, Daniel Shahaf wrote:
> > I've noticed several times that 'svn cleanup' trims down the
> > .svn/pristine/ hierarchy:
> [...]
> > Are pristines purged at other times?
>
> No.
>
> > If not, should we release-note
> > a recommendation to run 'cleanup' periodically?
>
> Yes.
>
r1151946.
> And IMO we should work on purging unused pristines automatically, for
> 1.7.x if possible.
>
+1
Re: Purging pristines
Posted by Julian Foad <ju...@wandisco.com>.
On Mon, 2011-07-25, Daniel Shahaf wrote:
> I've noticed several times that 'svn cleanup' trims down the
> .svn/pristine/ hierarchy:
[...]
> Are pristines purged at other times?
No.
> If not, should we release-note
> a recommendation to run 'cleanup' periodically?
Yes.
And IMO we should work on purging unused pristines automatically, for
1.7.x if possible.
I'd like to see purging at a fairly fine-grained level. Does anyone
have insight about how to do this? What's in my head is that every WC
API operation that could possibly leave unused pristines should attempt
to purge unused pristines before returning. The operations that could
leave unused pristines are basically 'update' and 'switch', although
there may be some lower level WC API calls that can do so too. The set
of pristines to be purged could be either all that are currently unused,
or just those that became unused during the call.
When I mentioned this before there was some sentiment expressed that the
delay involved in purging a large set of files could be unacceptably
large to a GUI client, but I think automatic purging is important and so
we have to find a way to make it acceptable, whether that involves
making it even finer grained or adding progress notification for it or
something else.
Thoughts?
- Julian