You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by François Beausoleil <fb...@ftml.net> on 2004/11/25 15:29:53 UTC

Unbounded memory usage in large WC move operations ?

Hello everyone !

I believe there might be some problems with very large working copies.

I checked out my whole vendor-drop repository location to a WC, because 
I want to do reorganizations, and I wanted them to be a single atomic 
commit.

My vendor drop is organized like this:
/              <-- repos root
   vendor/      <-- WC root
     log4j/
       1.2.8/
       current/
     velocity/
       1.4/
       current/

I want to rename all current/ to trunk/ and move all version tags to a 
tags folder.  Additionaly, I want to reorganize so that some vendor 
drops are under a common root.  For example, all Apache projects go 
together, Jakarta projects go together, etc.

Server and client are on the same physical machines, and the machine is 
a P4 3.0 Ghz, 1 Gb RAM and 80 Gb 7200 RPM HD.  OS is Windows XP SP2, and 
SVN is 1.1.1 [svn, version 1.1.0 (r11180)].

Checkout was fine.  Took lots of time, but that was to be expected, 
given the size of the checkout.

The problem occured when I started moving.  I executed the following 
from the root of my WC:
   $ svn mkdir jakarta
   $ svn move log4j jakarta

Believe it or not, the SVN client grew to use almost 300 Mb of RAM.  I 
have a picture of the Windows Task Manager with the memory and CPU graph 
showing:
   http://fbeausoleil.ftml.net/svn-large-move-mem-usage.gif

The whole move operation took something like 20 or 30 minutes (didn't 
keep track - I expected things to go much faster).  It looks like the 
move operation operated in three distinct phases:

1. WC locking phase seemed to use an unbounded amount of memory - 300 MB 
for my WC.  This took almost half of the runtime;
2. After the peak had been reached, memory usage stayed constant, or 
near enough.  This part took about half of the runtime too;
3. WC unlocking began, and finished pretty quickly.  Seemed like 30 
seconds to a minute.  Memory usage quickly dropped back to zero at that 
point.

I don't have exact statistics, but my WC has 53 libraries, each with a 
current/ and, on average, two version tags (so 3 * 53 = 159 vendor 
drops), and each vendor drop contains hundreds of files and folders.

What I suspect is going on here is that SVN is locking the whole WC, 
instead of locking the parent (vendor/) folder, and the two children it 
is interested in (log4j/ and jakarta/).

Is that to be expected ?  Am I missing some locking behavior here ? 
Should I forget about one single atomic commit, and do everything 
server-side instead ?  I know it'll be much faster.  Using the bindings 
[a reason to learn how to do it, I guess ;)], is it possible to do 
multiple server-side moves ?

Thanks !
François

Re: Unbounded memory usage in large WC move operations ?

Posted by Philip Martin <ph...@codematters.co.uk>.
François Beausoleil <fb...@ftml.net> writes:

> 1. WC locking phase seemed to use an unbounded amount of memory - 300
> MB for my WC.  This took almost half of the runtime;

Yes.  Most operations hold all the entries files in memory, it's a
known limitation of the current libsvn_wc implementation.

-- 
Philip Martin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org