You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mats Nilsson <ma...@xware.se> on 2003/06/18 11:20:59 UTC
Scalability problem using svn_load_dirs.pl during
Hi.
I tried to use svn_load_dirs to archive some external stuff that we use.
Sadly svn ran out of memory in the process.
Granted, the machine that I use is not particularly powerful, and I've
seen many accounts on this list of more able machines running into the
wall.
When it processes the second version of the external library, it quite
quickly goes through the harvesting of information, and the first call
to svn:
svn add -N --targets ....
But when it calls the next command (svn rm --targets ...), the svn grows
until all memory and swap is exhausted
I've searched issuezilla for scalability/performance issues, but nothing
that explicitly pertains to "svn rm".
Specifics:
Machine
RH9.0, 256 Megs of memory, 512 Megs of swap, 750 MHz
Subversion
0.24.1 (built from tarball)
In case anyone wants to try to reproduce:
I downloaded
http://deuce.doc.wustl.edu/old_distribution/ACE-5.3.zip
http://deuce.doc.wustl.edu/old_distribution/ACE-5.3.1.zip
And then unpacked into
~/tmp/ace-5.3 and
~/tmp/ace-5.3.1
mkdir -p ~/tmp/ace-5.3 ~/tmp/ace-5.3.1
cd ~/tmp/ace-5.3
unzip ~/ACE-5.3.zip
cd ~/tmp/ace-5.3.1
unzip ~/ACE-5.3.1.zip
cd ~/tmp
svnadmin create /usr/local/svn/repos
svn mkdir file:///usr/local/svn/repos/libs -m ""
svn mkdir file:///usr/local/svn/repos/libs/ACE -m ""
svn mkdir file:///usr/local/svn/repos/libs/ACE/trunk -m ""
svn mkdir file:///usr/local/svn/repos/libs/ACE/tags -m ""
perl ~/subversion/tools/client-side/svn_load_dirs.pl
file:///usr/local/libs/ACE trunk -t 'tags/@.*@' ace*
And then I accept all defaults.
The first version goes in nicely, and it has 6348 files in 248
directories. About 60Mb unpacked.
The second version has 4899 files in 230 directories. A few directories
have hundreds of files each.
The difference to the first version consists of a few directory and file
additions, roughly one hundred changed files in mostly one directory,
and files removed from two directory trees.
As stated above, during the second round, svn rm eats all my memory.
Is this related to other performance/scalability issues?
Mats
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Scalability problem using svn_load_dirs.pl during
Posted by cm...@collab.net.
"Mats Nilsson" <ma...@xware.se> writes:
> Maybe I'm confused by the different size columns in 'top'.
>
> SIZE says 600M, RSS says 230M when the machine halts.
>
> I ran svn commit outside svn_load_dirs and it grows quite rapidly while
> committing all the 1503 deletes in the same directory
> trunk/ACE_wrappers/man/man.3/.
>
> So is this another pool/subpool issue, or just another issue of having
> many files in the same directory?
>
> Mats
>
> >
> > And it doesn't fail for me. svn does grow to 'only' 200Meg when it
> > does the commit, though, so something's obviously wrong.
> >
> > Robert
I'm working on this bug right now, actually. There was definitely a
pool usage error in 'svn commit'. Am testing a fix for it now.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Scalability problem using svn_load_dirs.pl during
Posted by Mats Nilsson <ma...@xware.se>.
Maybe I'm confused by the different size columns in 'top'.
SIZE says 600M, RSS says 230M when the machine halts.
I ran svn commit outside svn_load_dirs and it grows quite rapidly while
committing all the 1503 deletes in the same directory
trunk/ACE_wrappers/man/man.3/.
So is this another pool/subpool issue, or just another issue of having
many files in the same directory?
Mats
>
> And it doesn't fail for me. svn does grow to 'only' 200Meg when it
> does the commit, though, so something's obviously wrong.
>
> Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Scalability problem using svn_load_dirs.pl during
Posted by Robert Pluim <rp...@bigfoot.com>.
Robert Pluim writes:
> Mats Nilsson writes:
>
> > > > Is that related to issue 860 (httpd memory use proportional to file
> > > > size)?
> > >
> > > Doesn't look like it at first glance (but you're running over file://,
> > > right?).
> >
> > Right, I'm filing a new issue now, after some IRC with sussman.
>
> OK. I'm just following your recipe here to see how it fails for me.
And it doesn't fail for me. svn does grow to 'only' 200Meg when it
does the commit, though, so something's obviously wrong.
Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Scalability problem using svn_load_dirs.pl during
Posted by Robert Pluim <rp...@bigfoot.com>.
Mats Nilsson writes:
> > > Is that related to issue 860 (httpd memory use proportional to file
> > > size)?
> >
> > Doesn't look like it at first glance (but you're running over file://,
> > right?).
>
> Right, I'm filing a new issue now, after some IRC with sussman.
OK. I'm just following your recipe here to see how it fails for me.
Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Scalability problem using svn_load_dirs.pl during
Posted by Mats Nilsson <ma...@xware.se>.
> I see the client memory usage growing here when I commit a mass
> deletion, but it's not that excessive, ~40M for 1000 file delete. How
> bad is it for you?
>
The machine grinds to a stop when svn has consumed 600M.
> > Is that related to issue 860 (httpd memory use proportional to file
> > size)?
>
> Doesn't look like it at first glance (but you're running over file://,
> right?).
Right, I'm filing a new issue now, after some IRC with sussman.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Scalability problem using svn_load_dirs.pl during
Posted by Robert Pluim <rp...@bigfoot.com>.
Mats Nilsson writes:
> Yes!
>
> Svn stays at about 6M (instead of hitting the wall at 600M).
> Thanks!
Cool. Could someone with commit privileges check my patch? (I've
reattached it with a log message).
> Next problem when it gets down to "svn commit", which also eats all my
> memory (a little slower than svn rm, though).
I see the client memory usage growing here when I commit a mass
deletion, but it's not that excessive, ~40M for 1000 file delete. How
bad is it for you?
> Is that related to issue 860 (httpd memory use proportional to file
> size)?
Doesn't look like it at first glance (but you're running over file://,
right?).
Robert
* svn/subversion/libsvn_client/delete.c (svn_client_delete): Actually
use the subpool that we were already creating.
RE: Scalability problem using svn_load_dirs.pl during
Posted by Mats Nilsson <ma...@xware.se>.
Yes!
Svn stays at about 6M (instead of hitting the wall at 600M).
Thanks!
Next problem when it gets down to "svn commit", which also eats all my
memory (a little slower than svn rm, though).
Is that related to issue 860 (httpd memory use proportional to file
size)?
Mats
> -----Original Message-----
> From: Robert Pluim [mailto:rpluim@bigfoot.com]
> Sent: den 18 juni 2003 14:11
> To: Mats Nilsson
> Cc: dev@subversion.tigris.org
> Subject: Re: Scalability problem using svn_load_dirs.pl during
>
>
> Mats Nilsson writes:
> > Hi.
> >
> > I tried to use svn_load_dirs to archive some external
> stuff that we use.
> > Sadly svn ran out of memory in the process.
> >
> > Granted, the machine that I use is not particularly
> powerful, and I've
> > seen many accounts on this list of more able machines
> running into the
> > wall.
> >
> > When it processes the second version of the external
> library, it quite
> > quickly goes through the harvesting of information, and
> the first call
> > to svn:
> > svn add -N --targets ....
> > But when it calls the next command (svn rm --targets ...),
> the svn grows
> > until all memory and swap is exhausted
>
> My-oh-my, svn_client_delete creates a subpool, and then never uses
> it. Mats, does the following patch help?
>
> Robert
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Scalability problem using svn_load_dirs.pl during
Posted by Robert Pluim <rp...@bigfoot.com>.
Mats Nilsson writes:
> Hi.
>
> I tried to use svn_load_dirs to archive some external stuff that we use.
> Sadly svn ran out of memory in the process.
>
> Granted, the machine that I use is not particularly powerful, and I've
> seen many accounts on this list of more able machines running into the
> wall.
>
> When it processes the second version of the external library, it quite
> quickly goes through the harvesting of information, and the first call
> to svn:
> svn add -N --targets ....
> But when it calls the next command (svn rm --targets ...), the svn grows
> until all memory and swap is exhausted
My-oh-my, svn_client_delete creates a subpool, and then never uses
it. Mats, does the following patch help?
Robert