Optimizing SVN switch

[Bill Soudan <bi...@soudan.net>]

Hi all,

I'm in the process of changing us over from CVS to SVN at work, mainly to
take advantage of SVN's cheap branching.  We're planning on modifying our
workflow to use a one-branch-per-bug model, which will simplify our lives
considerably during the code review and merging processes.  Of course,
we'll make heavy use of the 'svn switch' operation.

I'm a bit disappointed in the performance of 'svn switch' on a working
copy over an NFS mount.  We're running on (unfortunately) somewhat old SUN
hardware.  Our repository consists of approximately 2300 files and
directories, and a switch from one branch to a nearly similar branch takes
over 4 minutes.

stracing the svn client reveals the following points of note:

  0.0000s - some sort of wc check
  0.1682s - tree lock, read entries files
  3.5427s - open .svn/tmp/tempfile.tmp
  3.5448s - scan wc directories, getdents64
  5.7612s - contact repository over socket, do stuff
  5.9175s - read wcprops for each file & dir & then set to 'END'
141.????s - do actual wc updates
143.1844s - read wcprops for each file & dir & then set to new value
267.9598s - write new entries files
275.6327s - close & rm .svn/tmp/tempfile.tmp, unlock tree
277.2784s - operation complete

The wcprop operations account for ~259s of the 277s.  Therefore, the
direction I'm headed is to try an eliminate as many of the wcprop
operations as I can.  I don't know a lot yet about the guts of Subversion,
but my current thoughts:

1) Can wcprops be eliminated entirely?  Why aren't they just stuffed into 
the entries file?  My guess is that the file is seperate for a good 
reason, but if so, what is it?

2) Can switch just skip the step that sets all wcprops to END?  The second
'set wcprop' step just overwrites the old version anyway.  That change
would shave off 135s, about 50% of the time.

3) Can some of the wcprops operations be eliminated?  There's lots of 
redundant calls during each iteration (format is read over and over again, 
lots of lstat64s on the same file, etc.)

I'm planning on spending more time on this, but in the meantime, any 
feedback or pointers would be appreciated.

Thanks,
Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Optimizing SVN switch

[Bill Soudan <bi...@soudan.net>]

On Fri, 9 Apr 2004, C. Michael Pilato wrote:

> We have two walks because the wc-props are a cache system, and bad
> stuff happens if the cache is wrong.  So, we clear the cache
> (painstakingly) before the switch so we don't wind up with a race
> condition should something break during the switch.  Then afterwards,
> the REPORT response gives us back all the new cache values, which we
> then write to disk.

Sounds reasonable, though in my mind, if the other access methods can get
by without this operation, it seems as though there should be a way for
ra_dav to do the same.  What's inherently different about ra_dav that
gives it a race condition the others don't have?

> We could lose the second of these walks and amortize the cost of
> having no cache across the set of future operations (the first we
> needed the Version Resource URLs for a given path, we'd have to ask
> the server for it and cache it).
> 
> We could probably speed up the first of the walks by just 'rm-ing' the
> wcprops files instead of reading and writing empty ones, though I'm
> just guessing at that (and don't know what kind of other pains that
> might cause).

Could svn just maintain the wcprops in the 'entries' file along with the
other attributes?  There's a full tree entries read right before the first
wcprop walk, and then there's a full tree entries write after the second
wcprop walk.  There doesn't appear to be anything special happening in
between either pair.

For my repository/hardware/configuration, the entries write step is only 7
seconds vs. the over 2 minutes it takes to manage all the individual small
files.  Assuming the wcprops walks are entirely I/O bound, managing the
wcprops in the entries files would reduce the total time to approximately
20 seconds from over 4 minutes.

Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Optimizing SVN switch

["C. Michael Pilato" <cm...@collab.net>]

Bill Soudan <bi...@soudan.net> writes:

> I don't know much about the SVN internals yet but it looks like
> there's two 'resource walks' here, one where it removes all the
> wcprops (thus wrinig END to each file) which takes ~136s and then
> the second where it sets the updated key/value which takes ~124s.

You're absolutely right:

> >   5.9175s - read wcprops for each file & dir & then set to 'END'

Yes, this is the "clean 'em out" walk.

> > 143.1844s - read wcprops for each file & dir & then set to new value

And this is the "set 'em up" walk.

We have two walks because the wc-props are a cache system, and bad
stuff happens if the cache is wrong.  So, we clear the cache
(painstakingly) before the switch so we don't wind up with a race
condition should something break during the switch.  Then afterwards,
the REPORT response gives us back all the new cache values, which we
then write to disk.

We could lose the second of these walks and amortize the cost of
having no cache across the set of future operations (the first we
needed the Version Resource URLs for a given path, we'd have to ask
the server for it and cache it).

We could probably speed up the first of the walks by just 'rm-ing' the
wcprops files instead of reading and writing empty ones, though I'm
just guessing at that (and don't know what kind of other pains that
might cause).


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Optimizing SVN switch

[Bill Soudan <bi...@soudan.net>]

On Fri, 9 Apr 2004, C. Michael Pilato wrote:

> Bill Soudan <bi...@soudan.net> writes:
> 
> > For now, I'll switch over to svnserve.  What should I do from here?  File
> > an enhancement request for ra_dav?  Or is this dramatic enough to be 
> > considered a bug?
> 
> Is the problem that we use wcprops stored separately, or is 'svn
> switch' over ra-dav still sending the giant "resource walk" at the end
> of the switch REPORT response (which writes every wc-prop in the whole
> tree) ?

Re: Optimizing SVN switch

["C. Michael Pilato" <cm...@collab.net>]

Bill Soudan <bi...@soudan.net> writes:

> For now, I'll switch over to svnserve.  What should I do from here?  File
> an enhancement request for ra_dav?  Or is this dramatic enough to be 
> considered a bug?

Is the problem that we use wcprops stored separately, or is 'svn
switch' over ra-dav still sending the giant "resource walk" at the end
of the switch REPORT response (which writes every wc-prop in the whole
tree) ?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Optimizing SVN switch

[Bill Soudan <bi...@soudan.net>]

On Fri, 9 Apr 2004 kfogel@collab.net wrote:

> Bill Soudan <bi...@soudan.net> writes:
>
> > 1) Can wcprops be eliminated entirely?  Why aren't they just stuffed into 
> > the entries file?  My guess is that the file is seperate for a good 
> > reason, but if so, what is it?
> 
> Only ra_dav uses the wcprops.  An interesting experiment, if you have
> the setup to do it, would be to try the same operations over
> svnserve/ra_svn and see how the timings compare.  That could tell us a
> lot.

Night and day.  Great news, I didn't expect this to be an easy fix.  
Timings for svn+rsh vs. http:

~/svn_rsh$ time svn switch svn+rsh://
0.70u 5.40s 0:11.60 52.5%

~/svn_http$ time svn switch http://
3.31u 156.62s 4:11.74 63.5%

For now, I'll switch over to svnserve.  What should I do from here?  File
an enhancement request for ra_dav?  Or is this dramatic enough to be 
considered a bug?

Most obvious fix to me would be to just maintain the wcprops in memory
during the operation and store them in the entries file in order to reduce
the huge number of file operations.  Or, if possible, eliminate ra_dav's
need for them entirely.  Keeping them in seperate files is just enormously
expensive for working copies on an NFS mount, a not exactly uncommon
situation.

Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Optimizing SVN switch

[kf...@collab.net]

Bill Soudan <bi...@soudan.net> writes:
> 1) Can wcprops be eliminated entirely?  Why aren't they just stuffed into 
> the entries file?  My guess is that the file is seperate for a good 
> reason, but if so, what is it?
> 
> 2) Can switch just skip the step that sets all wcprops to END?  The second
> 'set wcprop' step just overwrites the old version anyway.  That change
> would shave off 135s, about 50% of the time.
>
> 3) Can some of the wcprops operations be eliminated?  There's lots of 
> redundant calls during each iteration (format is read over and over again, 
> lots of lstat64s on the same file, etc.)

Only ra_dav uses the wcprops.  An interesting experiment, if you have
the setup to do it, would be to try the same operations over
svnserve/ra_svn and see how the timings compare.  That could tell us a
lot.

Thanks for doing this investigation...

-K

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org