You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ben Collins-Sussman <su...@collab.net> on 2002/10/21 20:03:26 UTC

issue 860 investigation

So issue #860 has been giving me nightmares.  It looks like it's a
regression bug, whereby mod_dav_svn is no longer scaling.  The bug
submitter claims that the server's memory use grows in proportion to
the amount of data it's writing to the server.

Here's my initial findings, just playing around with today's svn:


Issue 680 Initial Experiments
=============================

Using httpd-2.0.43 (released) and svn r3430.

Created a 100M random-data file:

   dd if=/dev/urandom of=foo count=100000 bs=1024


* svn add; commit over ra_dav:  svn peaks at 6M, httpd peaks at 17M
* svn add; commit over ra_local:   svn binary peaks at 22M

==> This isn't ideal, but it's not horrible either.  It also makes a
lot of sense.  You can see that libsvn_fs seems to be using 17M RAM to
commit the 100M file.

* checkout new working copy over ra_dav:  svn peaks at 5M, httpd at 12M
* checkout new working copy over ra_local:  svn binary peaks at 120M

==> This is absolutely freaky.  Can anyone explain this???!

The other thing that's odd is that the original bug report claims that
the memory lossage happens during mod_dav_svn *commit*.  

Why can't I reproduce the problem?  Is it simply because I need to
actually try committing a 2GB file, not a 100M file?








---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: issue 860 investigation

Posted by Ben Collins-Sussman <su...@collab.net>.
Michael Price <mp...@atl.lmco.com> writes:

> For what its worth, all of my testing is done on a FreeBSD box, not Linux.
> 

Me too... comrade.  :-)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: issue 860 investigation

Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Michael Price <mp...@atl.lmco.com> writes:
> I believe I'm the person behind that bug report. I'll perform more testing
> against the latest svn tonight and see what I get.
> 
> For what its worth, all of my testing is done on a FreeBSD box, not Linux.

Ben's also on a FreeBSD box, so at least you don't have *that*
variable to worry about.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: issue 860 investigation

Posted by Michael Price <mp...@atl.lmco.com>.
Ben Collins-Sussman writes:
 > OK, I repeated the test, committing a 1GB file of random data.
 > 
 > The client consumed a constant 6M as usual, and the httpd child grew
 > to 120M.  In the case of committing a 100M file, the httpd child grew
 > to 17M.  So it seems that we have a genuine memory leak that seems to
 > scale linearly.
 > 
 > I'm rebuilding everything with pool debugging, to see if the leak is
 > somewhere in mod_dav_svn or libsvn_fs.

I believe I'm the person behind that bug report. I'll perform more testing
against the latest svn tonight and see what I get.

For what its worth, all of my testing is done on a FreeBSD box, not Linux.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: issue 860 investigation

Posted by Ben Collins-Sussman <su...@collab.net>.
Branko Čibej <br...@xbc.nu> writes:

> Seriously: Do what I suggested on IRC. Repeat the test with a 200M and
> 300M file, and look at the trends. It's entirely possible that the
> commit memory usage increases exponentially, that's why you need more
> than one data point if you want to find out what's happening.

OK, I repeated the test, committing a 1GB file of random data.

The client consumed a constant 6M as usual, and the httpd child grew
to 120M.  In the case of committing a 100M file, the httpd child grew
to 17M.  So it seems that we have a genuine memory leak that seems to
scale linearly.

I'm rebuilding everything with pool debugging, to see if the leak is
somewhere in mod_dav_svn or libsvn_fs.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: issue 860 investigation

Posted by Branko Čibej <br...@xbc.nu>.
Ben Collins-Sussman wrote:

>So issue #860 has been giving me nightmares.  It looks like it's a
>regression bug, whereby mod_dav_svn is no longer scaling.  The bug
>submitter claims that the server's memory use grows in proportion to
>the amount of data it's writing to the server.
>
>Here's my initial findings, just playing around with today's svn:
>
>
>Issue 680 Initial Experiments
>=============================
>
>Using httpd-2.0.43 (released) and svn r3430.
>
>Created a 100M random-data file:
>
>   dd if=/dev/urandom of=foo count=100000 bs=1024
>
Naughty, naughty. That's not 100M. Use count=100 bs=1M :-)

>* svn add; commit over ra_dav:  svn peaks at 6M, httpd peaks at 17M
>* svn add; commit over ra_local:   svn binary peaks at 22M
>
>==> This isn't ideal, but it's not horrible either.  It also makes a
>lot of sense.  You can see that libsvn_fs seems to be using 17M RAM to
>commit the 100M file.
>
>* checkout new working copy over ra_dav:  svn peaks at 5M, httpd at 12M
>* checkout new working copy over ra_local:  svn binary peaks at 120M
>
>==> This is absolutely freaky.  Can anyone explain this???!
>
Yikes! The only explanation I can think of is that libsvn_ra_local 
itself is leaking like a sieve.

>The other thing that's odd is that the original bug report claims that
>the memory lossage happens during mod_dav_svn *commit*.  
>
>Why can't I reproduce the problem?  Is it simply because I need to
>actually try committing a 2GB file, not a 100M file?
>
Either that, or close the bug and open another one. :-)

Seriously: Do what I suggested on IRC. Repeat the test with a 200M and 
300M file, and look at the trends. It's entirely possible that the 
commit memory usage increases exponentially, that's why you need more 
than one data point if you want to find out what's happening.

-- 
Brane Čibej   <br...@xbc.nu>   http://www.xbc.nu/brane/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org