WCNG - Upgrading working copies

[Daniel Becroft <dj...@gmail.com>]

Hi,

I'm currently trialling the nightly builds that Stefan is providing for
TSVN, which include the new WCNG implementation. So far, it all looks good.
It takes a few minutes extra to checkout a working copy, but the day-to-day
performance difference (to me) is not noticeable. Good stuff!

One thing I did notice, though, is the performance of the 'svn upgrade'
functionality. It seems to take, on average, 3x longer to upgrade an
existing working copy than to just delete and checkout again.

Checkout (1.6.x): 9 minutes
Checkout (1.7.x): 14 minutes
Upgrade (1.7.x): 43 minutes

Is this kind of performance hit during an upgrade expected? Is the
recommended upgrade option expected to be to use 'svn upgrade' or do get a
fresh checkout?

If you need stats regarding the working copy, I can provide them.

Cheers,
---
Daniel Becroft

Re: WCNG - Upgrading working copies

[Philip Martin <ph...@wandisco.com>]

Mark Phippard <ma...@gmail.com> writes:

> It seems like Daniel is more asking if this might be indicative of a
> bug.  Does it ever make sense that svn upgrade would take that long?
> Without any need to fetch pristines from the repository as checkout
> needs to do, it seems hard to believe that it should take that long.
> I can imagine scenarios where we have to calc new checksums or open
> and re-read entry data over and over again but 43 minutes?
>
> Daniel, is your working copy on a local hard drive?

Upgrade has had no performance tuning and it's possible that upgrade
executes more SQLite queries than checkout.  The number of queries has a
significant performance effect on network disks:

http://svn.haxx.se/dev/archive-2010-10/0350.shtml

Upgrade copies the pristines, so it's reading and writing from the same
working copy.  Checkout reads pristines from the network and writes to
disk, this could be faster depending on the performance of disk and
network.

Performance on network disks is a concern.

-- 
Philip

Re: WCNG - Upgrading working copies

[Stefan Sperling <st...@elego.de>]

On Fri, Nov 26, 2010 at 08:58:48AM -0500, Mark Phippard wrote:
> On Fri, Nov 26, 2010 at 8:02 AM, Stefan Sperling <st...@elego.de> wrote:
> > On Fri, Nov 26, 2010 at 02:05:55PM +1000, Daniel Becroft wrote:
> >> Hi,
> >>
> >> I'm currently trialling the nightly builds that Stefan is providing for
> >> TSVN, which include the new WCNG implementation. So far, it all looks good.
> >> It takes a few minutes extra to checkout a working copy, but the day-to-day
> >> performance difference (to me) is not noticeable. Good stuff!
> >>
> >> One thing I did notice, though, is the performance of the 'svn upgrade'
> >> functionality. It seems to take, on average, 3x longer to upgrade an
> >> existing working copy than to just delete and checkout again.
> >>
> >> Checkout (1.6.x): 9 minutes
> >> Checkout (1.7.x): 14 minutes
> >> Upgrade (1.7.x): 43 minutes
> >>
> >> Is this kind of performance hit during an upgrade expected? Is the
> >> recommended upgrade option expected to be to use 'svn upgrade' or do get a
> >> fresh checkout?
> >>
> >> If you need stats regarding the working copy, I can provide them.
> >
> > Please keep benchmarking and reporting your results.
> > The current state of wc-ng development is, AFAIK, that we're still
> > working out some final steps in the implementation. We're not yet at
> > the point where we're heavily trying to optimize the new code.
> > But any report about slowness will help once we've reached that point.
> > Keep them coming.
> 
> It seems like Daniel is more asking if this might be indicative of a
> bug.  Does it ever make sense that svn upgrade would take that long?
> Without any need to fetch pristines from the repository as checkout
> needs to do, it seems hard to believe that it should take that long.
> I can imagine scenarios where we have to calc new checksums or open
> and re-read entry data over and over again but 43 minutes?
> 
> Daniel, is your working copy on a local hard drive?

Yes, these performance problems are certainly bugs.
Neels has made more performance measurements recently, similar to the
ones he shared on this list a few weeks ago.
The current performance is pretty disappointing, not just for upgrade.

I suppose you're right that we should be starting to look into these
problems ASAP. They can be fixed without having op_depth and the
conflict store finished.

Stefan

Re: WCNG - Upgrading working copies

[Daniel Becroft <dj...@gmail.com>]

On Fri, Nov 26, 2010 at 11:58 PM, Mark Phippard <ma...@gmail.com> wrote:

> On Fri, Nov 26, 2010 at 8:02 AM, Stefan Sperling <st...@elego.de> wrote:
> > On Fri, Nov 26, 2010 at 02:05:55PM +1000, Daniel Becroft wrote:
> >> Hi,
> >>
> >> I'm currently trialling the nightly builds that Stefan is providing for
> >> TSVN, which include the new WCNG implementation. So far, it all looks
> good.
> >> It takes a few minutes extra to checkout a working copy, but the
> day-to-day
> >> performance difference (to me) is not noticeable. Good stuff!
> >>
> >> One thing I did notice, though, is the performance of the 'svn upgrade'
> >> functionality. It seems to take, on average, 3x longer to upgrade an
> >> existing working copy than to just delete and checkout again.
> >>
> >> Checkout (1.6.x): 9 minutes
> >> Checkout (1.7.x): 14 minutes
> >> Upgrade (1.7.x): 43 minutes
> >>
> >> Is this kind of performance hit during an upgrade expected? Is the
> >> recommended upgrade option expected to be to use 'svn upgrade' or do get
> a
> >> fresh checkout?
> >>
> >> If you need stats regarding the working copy, I can provide them.
> >
> > Please keep benchmarking and reporting your results.
> > The current state of wc-ng development is, AFAIK, that we're still
> > working out some final steps in the implementation. We're not yet at
> > the point where we're heavily trying to optimize the new code.
> > But any report about slowness will help once we've reached that point.
> > Keep them coming.
>
> It seems like Daniel is more asking if this might be indicative of a
> bug.  Does it ever make sense that svn upgrade would take that long?
> Without any need to fetch pristines from the repository as checkout
> needs to do, it seems hard to believe that it should take that long.
> I can imagine scenarios where we have to calc new checksums or open
> and re-read entry data over and over again but 43 minutes?
>

Daniel, is your working copy on a local hard drive?
>

Hi Mark,

Yes, the working copy is on a local drive. It also has:

 - 398 .svn folders (some of these are 7 or 8 level deep)
 - 21,206 *.svn-base files

I'm ran another 'svn upgrade' process through Process Monitor (from
sysinternals), and have found something interesting.

It seems that for every file that gets added (e.g. trunk/A/B/C/alpha.txt),
it reads (or scans) the entire contents of the .svn/tmp/wcng/.svn/wc.db file
three times (each one on 1024 byte increments). At the moment, this file is
approx. 5Mb, and that's 2s in total between accesses to the .svn-base files.
I can't see any mention of entries, or pristine in there, so I'm not sure
what it's actually doing. THe last successful upgrade I ran (the 43m one)
gave an 11Mb file, so I'm not even half-way through. At this point procmon
is reporting 14m events, so I had to kill it.

When I get a chance, I'll run it again on a smaller WC.


Cheers,
Daniel B.

Re: WCNG - Upgrading working copies

[Mark Phippard <ma...@gmail.com>]

On Fri, Nov 26, 2010 at 8:02 AM, Stefan Sperling <st...@elego.de> wrote:
> On Fri, Nov 26, 2010 at 02:05:55PM +1000, Daniel Becroft wrote:
>> Hi,
>>
>> I'm currently trialling the nightly builds that Stefan is providing for
>> TSVN, which include the new WCNG implementation. So far, it all looks good.
>> It takes a few minutes extra to checkout a working copy, but the day-to-day
>> performance difference (to me) is not noticeable. Good stuff!
>>
>> One thing I did notice, though, is the performance of the 'svn upgrade'
>> functionality. It seems to take, on average, 3x longer to upgrade an
>> existing working copy than to just delete and checkout again.
>>
>> Checkout (1.6.x): 9 minutes
>> Checkout (1.7.x): 14 minutes
>> Upgrade (1.7.x): 43 minutes
>>
>> Is this kind of performance hit during an upgrade expected? Is the
>> recommended upgrade option expected to be to use 'svn upgrade' or do get a
>> fresh checkout?
>>
>> If you need stats regarding the working copy, I can provide them.
>
> Please keep benchmarking and reporting your results.
> The current state of wc-ng development is, AFAIK, that we're still
> working out some final steps in the implementation. We're not yet at
> the point where we're heavily trying to optimize the new code.
> But any report about slowness will help once we've reached that point.
> Keep them coming.

It seems like Daniel is more asking if this might be indicative of a
bug.  Does it ever make sense that svn upgrade would take that long?
Without any need to fetch pristines from the repository as checkout
needs to do, it seems hard to believe that it should take that long.
I can imagine scenarios where we have to calc new checksums or open
and re-read entry data over and over again but 43 minutes?

Daniel, is your working copy on a local hard drive?

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: WCNG - Upgrading working copies

[Stefan Sperling <st...@elego.de>]

On Fri, Nov 26, 2010 at 02:05:55PM +1000, Daniel Becroft wrote:
> Hi,
> 
> I'm currently trialling the nightly builds that Stefan is providing for
> TSVN, which include the new WCNG implementation. So far, it all looks good.
> It takes a few minutes extra to checkout a working copy, but the day-to-day
> performance difference (to me) is not noticeable. Good stuff!
> 
> One thing I did notice, though, is the performance of the 'svn upgrade'
> functionality. It seems to take, on average, 3x longer to upgrade an
> existing working copy than to just delete and checkout again.
> 
> Checkout (1.6.x): 9 minutes
> Checkout (1.7.x): 14 minutes
> Upgrade (1.7.x): 43 minutes
> 
> Is this kind of performance hit during an upgrade expected? Is the
> recommended upgrade option expected to be to use 'svn upgrade' or do get a
> fresh checkout?
> 
> If you need stats regarding the working copy, I can provide them.

Please keep benchmarking and reporting your results.
The current state of wc-ng development is, AFAIK, that we're still
working out some final steps in the implementation. We're not yet at
the point where we're heavily trying to optimize the new code.
But any report about slowness will help once we've reached that point.
Keep them coming.

Thanks,
Stefan