You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Karl Fogel <kf...@newton.ch.collab.net> on 2002/11/04 22:06:53 UTC
RFV on issue 860 (httpd memory usage)
Anyone who's feeling squirrely, please take a look at
http://subversion.tigris.org/issues/show_bug.cgi?id=860
Toward the end, you'll see Ben Collins-Sussman's summary of the
current situation, about how the bug's severity has been reduced, yet
there is still some mysterious linearity going on.
I'm going to be looking at this too, but this is one of those
situations where many eyes might help. Especially those with
experience tracing Berkeley DB memory usage and/or APR pools.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by mark benedetto king <bk...@inquira.com>.
On Mon, Nov 04, 2002 at 04:06:53PM -0600, Karl Fogel wrote:
> Anyone who's feeling squirrely, please take a look at
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=860
>
> Toward the end, you'll see Ben Collins-Sussman's summary of the
> current situation, about how the bug's severity has been reduced, yet
> there is still some mysterious linearity going on.
>
> I'm going to be looking at this too, but this is one of those
> situations where many eyes might help. Especially those with
> experience tracing Berkeley DB memory usage and/or APR pools.
>
> -Karl
>
Have you tried running valgrind with leak-checking?
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
kbohling@birddog.com writes:
> I'll see if I can manage to get a version of apache installed and try it over
> ra_local and ra_dav tomorrow with that client. I'll also try a 1GB
> file see if that makes it visible.
Thanks for the experimenting! I've annotated the issue to point to
this thread (starting with your message above).
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by kb...@birddog.com.
(I can't tell for sure, but it sure seems that way).
Quoting "Kirby C. Bohling" <kb...@birddog.com>:
> On Mon, 2002-11-04 at 16:06, Karl Fogel wrote:
> > Anyone who's feeling squirrely, please take a look at
> >
> > http://subversion.tigris.org/issues/show_bug.cgi?id=860
> >
> > Toward the end, you'll see Ben Collins-Sussman's summary of the
> > current situation, about how the bug's severity has been reduced, yet
> > there is still some mysterious linearity going on.
> >
> > I'm going to be looking at this too, but this is one of those
> > situations where many eyes might help. Especially those with
> > experience tracing Berkeley DB memory usage and/or APR pools.
> >
>
> Karl,
>
> At the end of the issue, it talks about it might be apache or Berkeley
> DB. However, at the top it mentions that this happens with ra_dav or
> ra_local. So I'm failing to see how it could be apache if it does it
> over ra_local.
>
..Snip...
I looked into this, and did the following.
Using RH 8.0, fresh install, installed everything.
Download 0.15.0, untar.
./configure --enable-maintainer-mode --without-apache --without-apxs
make
svnadmin create /home/kbohling/repos
svn mkdir file:///home/kbohling/repos/trunk
svn co file:///home/kbohling/repos/trunk
dd if=/dev/urandom of=stuff count=200k bs=1024
svn add stuff
svn commit -m "Memory hog"
ldd of lt-svn seems to say it was using db4, not db4.1.
I was sitting watching top in another window. It claimed a SIZE of 5220 for
almost the entire commit. Right near the end, it went up to 5280, the spiked
very briefly to 5336 or so. I never ever saw it hit, either of the sizes
mentioned in the issue entry. I'm not sure what this means. I generally don't
build the apache client/server pieces.
What'd I do differently? I'm missing the verbose memory stuff, and I'm missing
the mod_dav_svn stuff. Does that help at all? The issue claimed the problem
exists over ra_local, but I sure don't see it on my RH machine. I thought I was
supposed to see it slowly gain to about 17MB in memory, but I definitely didn't
see that.
I'll see if I can manage to get a version of apache installed and try it over
ra_local and ra_dav tomorrow with that client. I'll also try a 1GB file see if
that makes it visible.
Thanks,
Kirby
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Daniel Berlin <db...@dberlin.org>.
On 4 Nov 2002, Kirby C. Bohling wrote:
> On Mon, 2002-11-04 at 16:06, Karl Fogel wrote:
> > Anyone who's feeling squirrely, please take a look at
> >
> > http://subversion.tigris.org/issues/show_bug.cgi?id=860
> >
> > Toward the end, you'll see Ben Collins-Sussman's summary of the
> > current situation, about how the bug's severity has been reduced, yet
> > there is still some mysterious linearity going on.
> >
> > I'm going to be looking at this too, but this is one of those
> > situations where many eyes might help. Especially those with
> > experience tracing Berkeley DB memory usage and/or APR pools.
> >
>
> Karl,
>
> At the end of the issue, it talks about it might be apache or Berkeley
> DB. However, at the top it mentions that this happens with ra_dav or
> ra_local. So I'm failing to see how it could be apache if it does it
> over ra_local.
>
> I'm kinda curious if this is a WAD (Works as Designed) issue with
> Berkeley DB. You're writting a lot of information to a lot of different
> log files that you're not committing. I don't know enough about
> Berkeley's infrastructure to comment, but has anybody asked them about
> the overhead of having an active transaction that crosses 100-1000 log
> files (log files are still 1M right?), and committing it has to write a
> compressed version of the file out to the logs right so your talking
> between 100-1000 log files (depends on the data and the compression).
> So to take up the 28MB difference between 100M and 900M of data. 900M
> of random data compresses into ~900 1M log files. So is right about 32K
> of overhead per open log file in a transaction assuming no compression.
> 32K sounds like a reasonable number (pretty high given BDB's embedded
> usage, but maybe), so it might be there.
>
> Maybe modify the log file size on the creation to be 10M and see if the
> 100M starts acting like 10M that's my bet.
>
> Maybe Daniel Berlin knows, I'm pretty sure he's the one who set
> straight some things for me about BDB on the list in the past.
Hmmm.
I took at look the berkeley source for log stuff (Now i remember why i
couldn't remember it. It makes me shiver. I must have blocked it out. Not
that it's not well designed, just hard to read.)
It looks like putting into the log, including new log file creation if
necessary, frees all the memory it allocates.
IE it only keeps a live structure for the *current* log in memory at a
given time, regardless of whether the transaction spans logs or not.
It also doesn't mmap log files (it only seems to call os_mapfile, which
is what maps files if it can, when opening the memory pool), so the
memory isn't coming from there either.
--Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Kirby C. Bohling" <kb...@birddog.com>.
On Mon, 2002-11-04 at 16:06, Karl Fogel wrote:
> Anyone who's feeling squirrely, please take a look at
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=860
>
> Toward the end, you'll see Ben Collins-Sussman's summary of the
> current situation, about how the bug's severity has been reduced, yet
> there is still some mysterious linearity going on.
>
> I'm going to be looking at this too, but this is one of those
> situations where many eyes might help. Especially those with
> experience tracing Berkeley DB memory usage and/or APR pools.
>
Karl,
At the end of the issue, it talks about it might be apache or Berkeley
DB. However, at the top it mentions that this happens with ra_dav or
ra_local. So I'm failing to see how it could be apache if it does it
over ra_local.
If this happens when using ra_local (debugging apache intimidates me...
sorry I'm a wimp), I would take a look at it. Leaking memory is
something I grok well enough, that I feel like I can contribute.
Between strace and LD_PRELOAD (on brk and mmap) it should be possible
(maybe even easy) to find a backtrace of where the memory is being
allocated from then send in backtraces that are the worst offenders.
Maybe I'll finally remember to use that commit access to commit it to
dev/tools too.
I'm kinda curious if this is a WAD (Works as Designed) issue with
Berkeley DB. You're writting a lot of information to a lot of different
log files that you're not committing. I don't know enough about
Berkeley's infrastructure to comment, but has anybody asked them about
the overhead of having an active transaction that crosses 100-1000 log
files (log files are still 1M right?), and committing it has to write a
compressed version of the file out to the logs right so your talking
between 100-1000 log files (depends on the data and the compression).
So to take up the 28MB difference between 100M and 900M of data. 900M
of random data compresses into ~900 1M log files. So is right about 32K
of overhead per open log file in a transaction assuming no compression.
32K sounds like a reasonable number (pretty high given BDB's embedded
usage, but maybe), so it might be there.
Maybe modify the log file size on the creation to be 10M and see if the
100M starts acting like 10M that's my bet.
Maybe Daniel Berlin knows, I'm pretty sure he's the one who set
straight some things for me about BDB on the list in the past.
Thanks,
Kirby
> -Karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
Real Programmers view electronic multimedia files with a hex editor.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
"Glenn A. Thompson" <gt...@cdr.net> writes:
> >No, I don't think so. Revision 3589 added the scratchpool, but didn't
> >change anyone to actually use it yet. The change that brought the
> >memory usage down was probably revision 3469.
>
> OK, I'll look at this change
Great!
> >Note how Ben says in the issue that most the memory is not showing up
> >in the pool statistics at all. His 40MB figure is coming from 'top'!
> >As far as the pool debug output goes, pool usage hovers at about 3MB.
> >That's the main clue that it might be a problem with our Berkeley DB
> >usage...
>
> OH yeah, understand. I just trying to grasp the 1/3 number.
> Cool.
> I'll keep my eyes open.
Thanks, looking forward to any ideas...
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>No, I don't think so. Revision 3589 added the scratchpool, but didn't
>change anyone to actually use it yet. The change that brought the
>memory usage down was probably revision 3469.
>
>
OK, I'll look at this change
>Note how Ben says in the issue that most the memory is not showing up
>in the pool statistics at all. His 40MB figure is coming from 'top'!
>As far as the pool debug output goes, pool usage hovers at about 3MB.
>That's the main clue that it might be a problem with our Berkeley DB
>usage...
>
>
OH yeah, understand. I just trying to grasp the 1/3 number.
Cool.
I'll keep my eyes open.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
Kirby C. Bohling writes:
> Can you still reproduce the memory 40MB of VM consumed by the commit
> which was troublesome to find? It doesn't appear so from the time
> output from the commit (only 5MB resident VM). You originally posted
> the bug. That was the one I was completely unsuccessful at reproducing.
> I know Ben Collins-Sussman is also a FreeBSD user, and he was originally
> working on the issue 860. I'm curious if anybody's seen it recently
> under any other OS.
The bug described in issue 860 isn't much of an issue anymore on my
platform so I haven't been worrying about it. If there is still a bug
there then its too small for me to spend time tracking down.
This issue looks to be different enough from 860 for me to file a new
issue. Its 985. http://subversion.tigris.org/issues/show_bug.cgi?id=985
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Kirby C. Bohling" <kb...@birddog.com>.
On Wed, 2002-11-13 at 00:31, Michael Price wrote:
> Kirby C. Bohling writes:
> > Could you send me more info on your configuration? What did you pass
> > to configure, what revision of svn, what version OS/Compiler/libc your
> > using? Which BDB version are you using?
>
> FreeBSD 4.7-STABLE, svn revision 3772, apr/apr-util from today, db 4.1.24
> with patches 1 2 4 & 6, neon 0.23.5.
>
Check, thanks. I'll download a copy. The last FreeBSD version I have
is 4.4 I believe.
> > I tried to duplicate it, and I couldn't on a fresh RedHat 8.0 machine.
> > I've got plenty of spare hardware, and plenty of bandwidth to reproduce
> > this on x86 hardware with your Free OS of choice. I checked in smaller
> > file (about 200MB), but I never saw more the 5.4MB of allocated VM at
> > any point during my commit, and it moved about add about 40K right at
> > the end of the commit. Which is fundamentally different behaviour then
> > what the issue tracker describes.
>
> You did notice in the script that the program used too much memory on
> "checkout" and not "commit" didn't you?
>
No I missed that, I was too focused on the description of 860 to be
bothered by details... :-(. It sounds like this is a different
problem, that possibly has been addressed.
Can you still reproduce the memory 40MB of VM consumed by the commit
which was troublesome to find? It doesn't appear so from the time
output from the commit (only 5MB resident VM). You originally posted
the bug. That was the one I was completely unsuccessful at reproducing.
I know Ben Collins-Sussman is also a FreeBSD user, and he was originally
working on the issue 860. I'm curious if anybody's seen it recently
under any other OS.
Thanks,
Kirby
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
Real Programmers view electronic multimedia files with a hex editor.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
Kirby C. Bohling writes:
> Could you send me more info on your configuration? What did you pass
> to configure, what revision of svn, what version OS/Compiler/libc your
> using? Which BDB version are you using?
FreeBSD 4.7-STABLE, svn revision 3772, apr/apr-util from today, db 4.1.24
with patches 1 2 4 & 6, neon 0.23.5.
> I tried to duplicate it, and I couldn't on a fresh RedHat 8.0 machine.
> I've got plenty of spare hardware, and plenty of bandwidth to reproduce
> this on x86 hardware with your Free OS of choice. I checked in smaller
> file (about 200MB), but I never saw more the 5.4MB of allocated VM at
> any point during my commit, and it moved about add about 40K right at
> the end of the commit. Which is fundamentally different behaviour then
> what the issue tracker describes.
You did notice in the script that the program used too much memory on
"checkout" and not "commit" didn't you?
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Kirby C. Bohling" <kb...@birddog.com>.
On Tue, 2002-11-12 at 23:57, Michael Price wrote:
> This isn't httpd related but it is issue 860 related. There is an svn
> memory leak and/or memory issue. The script below should explain better
> than I can.
>
> Note that the program crashed in the last example due to the hard memory
> limits I set (as the subsequent 'ulimit -a' shows).
>
... Snip interesting script output...
Michael,
Could you send me more info on your configuration? What did you pass
to configure, what revision of svn, what version OS/Compiler/libc your
using? Which BDB version are you using?
I tried to duplicate it, and I couldn't on a fresh RedHat 8.0 machine.
I've got plenty of spare hardware, and plenty of bandwidth to reproduce
this on x86 hardware with your Free OS of choice. I checked in smaller
file (about 200MB), but I never saw more the 5.4MB of allocated VM at
any point during my commit, and it moved about add about 40K right at
the end of the commit. Which is fundamentally different behaviour then
what the issue tracker describes.
I'll try this script when I get home, but with 0.15.0 I couldn't
duplicate the issue on a RedHat Linux box when I didn't build
mod_dav_svn.
I would love to help track this down, but I can't repoduce it yet. So
more information on your configuration would be mighty useful to getting
me in a position where I can start investigation of the problem.
Thanks,
Kirby
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
Real Programmers view electronic multimedia files with a hex editor.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Ben Collins-Sussman <su...@collab.net>.
Michael Price <mp...@atl.lmco.com> writes:
> Ben Collins-Sussman writes:
> > Are you using today's svn? Because last week cmpilato fixed a
> > checkout bug that looked a *lot* like this. Memory during checkouts
> > and updates were essentially growing quickly and without bound.
>
> Used revision 3772.
Grrr, hm. I think the bug cmpilato fixed involved updating a whole
bunch of files.
>
> apr and apr-util are from today.
>
> db version 4.1.24 with patches 1, 2, 4, and 6.
This may be an interesting variable too. Not many of us are using db
4.1 yet.
Can you file this as an issue for us?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
Ben Collins-Sussman writes:
> Are you using today's svn? Because last week cmpilato fixed a
> checkout bug that looked a *lot* like this. Memory during checkouts
> and updates were essentially growing quickly and without bound.
Used revision 3772.
apr and apr-util are from today.
db version 4.1.24 with patches 1, 2, 4, and 6.
neon version 0.23.5
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Ben Collins-Sussman <su...@collab.net>.
Are you using today's svn? Because last week cmpilato fixed a
checkout bug that looked a *lot* like this. Memory during checkouts
and updates were essentially growing quickly and without bound.
Michael Price <mp...@atl.lmco.com> writes:
> This isn't httpd related but it is issue 860 related. There is an svn
> memory leak and/or memory issue. The script below should explain better
> than I can.
>
> Note that the program crashed in the last example due to the hard memory
> limits I set (as the subsequent 'ulimit -a' shows).
>
> Michael
>
> Script started on Wed Nov 13 00:03:13 2002
> (Warning: MANPATH environment variable set)
> mprice@taco ~/test> dd if=/dev/urandom of=big-ass-file bs=1024 count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes transferred in 432.467511 secs (2482827 bytes/sec)
> mprice@taco ~/test> svnadmin create repo
> mprice@taco ~/test> svn co file:///$HOME/test/repo wc
> Checked out revision 0.
> mprice@taco ~/test> mv big-ass-file wc
> mprice@taco ~/test> cd wc
> mprice@taco ~/test/wc> svn add big-ass-file
> A (bin) big-ass-file
> mprice@taco ~/test/wc> /usr/bin/time -l svn commit -m "Initial big-ass commit."
> Adding (bin) big-ass-file
> Transmitting file data .
> Committed revision 1.
> 1737.10 real 469.79 user 93.13 sys
> 5148 maximum resident set size
> 1156 average shared memory size
> 3271 average unshared data size
> 128 average unshared stack size
> 3655 page reclaims
> 18 page faults
> 0 swaps
> 65866 block input operations
> 226469 block output operations
> 0 messages sent
> 0 messages received
> 0 signals received
> 132763 voluntary context switches
> 20309 involuntary context switches
> mprice@taco ~/test/wc> cd ..
> mprice@taco ~/test> rm -fr wc
> mprice@taco ~/test> cd repo/db
> mprice@taco ~/test/repo/db> db_archive | xargs /bin/rm
> mprice@taco ~/test/repo/db> cd ..
> mprice@taco ~/test/repo> cd ..
> mprice@taco ~/test> /usr/bin/time -l svn co file:///$HOME/test/repo wc
> time: command terminated abnormally
> 155.78 real 68.93 user 7.11 sys
> 461872 maximum resident set size
> 1158 average shared memory size
> 184068 average unshared data size
> 128 average unshared stack size
> 118627 page reclaims
> 18 page faults
> 0 swaps
> 8750 block input operations
> 16605 block output operations
> 0 messages sent
> 0 messages received
> 0 signals received
> 21830 voluntary context switches
> 9903 involuntary context switches
> Abort trap (core dumped)
> mprice@taco ~/test> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) 458752
> file size (blocks, -f) unlimited
> max locked memory (kbytes, -l) 65536
> max memory size (kbytes, -m) 524288
> open files (-n) 1024
> pipe size (512 bytes, -p) 1
> stack size (kbytes, -s) 65536
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1024
> virtual memory (kbytes, -v) 524288
> mprice@taco ~/test> exit
>
> Script done on Wed Nov 13 00:50:37 2002
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
This isn't httpd related but it is issue 860 related. There is an svn
memory leak and/or memory issue. The script below should explain better
than I can.
Note that the program crashed in the last example due to the hard memory
limits I set (as the subsequent 'ulimit -a' shows).
Michael
Script started on Wed Nov 13 00:03:13 2002
(Warning: MANPATH environment variable set)
mprice@taco ~/test> dd if=/dev/urandom of=big-ass-file bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 432.467511 secs (2482827 bytes/sec)
mprice@taco ~/test> svnadmin create repo
mprice@taco ~/test> svn co file:///$HOME/test/repo wc
Checked out revision 0.
mprice@taco ~/test> mv big-ass-file wc
mprice@taco ~/test> cd wc
mprice@taco ~/test/wc> svn add big-ass-file
A (bin) big-ass-file
mprice@taco ~/test/wc> /usr/bin/time -l svn commit -m "Initial big-ass commit."
Adding (bin) big-ass-file
Transmitting file data .
Committed revision 1.
1737.10 real 469.79 user 93.13 sys
5148 maximum resident set size
1156 average shared memory size
3271 average unshared data size
128 average unshared stack size
3655 page reclaims
18 page faults
0 swaps
65866 block input operations
226469 block output operations
0 messages sent
0 messages received
0 signals received
132763 voluntary context switches
20309 involuntary context switches
mprice@taco ~/test/wc> cd ..
mprice@taco ~/test> rm -fr wc
mprice@taco ~/test> cd repo/db
mprice@taco ~/test/repo/db> db_archive | xargs /bin/rm
mprice@taco ~/test/repo/db> cd ..
mprice@taco ~/test/repo> cd ..
mprice@taco ~/test> /usr/bin/time -l svn co file:///$HOME/test/repo wc
time: command terminated abnormally
155.78 real 68.93 user 7.11 sys
461872 maximum resident set size
1158 average shared memory size
184068 average unshared data size
128 average unshared stack size
118627 page reclaims
18 page faults
0 swaps
8750 block input operations
16605 block output operations
0 messages sent
0 messages received
0 signals received
21830 voluntary context switches
9903 involuntary context switches
Abort trap (core dumped)
mprice@taco ~/test> ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 458752
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) 524288
open files (-n) 1024
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65536
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) 524288
mprice@taco ~/test> exit
Script done on Wed Nov 13 00:50:37 2002
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
Michael Price writes:
> 1. Allocate on the heap a huge array of elements of size X.
> A good choice for X is 16 bytes.
And just to be clear (before everyone jumps on this) change that "array"
to "linked list".
Was typing on auto-pilot.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Michael Price <mp...@atl.lmco.com>.
Karl Fogel writes:
> "Glenn A. Thompson" <gt...@cdr.net> writes:
> > What change made it 1/3 the size it used to be? Was it just the
> > trail.c changes for scratchpool? Or was there something else?
>
> No, I don't think so. Revision 3589 added the scratchpool, but didn't
> change anyone to actually use it yet. The change that brought the
> memory usage down was probably revision 3469.
>
> Note how Ben says in the issue that most the memory is not showing up
> in the pool statistics at all. His 40MB figure is coming from 'top'!
> As far as the pool debug output goes, pool usage hovers at about 3MB.
> That's the main clue that it might be a problem with our Berkeley DB
> usage...
I feel kind of funny pointing this out since I'm the person behind this
issue, but 'top' isn't a reliable means of determining how much memory you
have allocated at the moment.
Every major Un*x OS I know of (including FreeBSD) uses a caching mechanism
behind (m|c|re)alloc so that memory you free() is rarely actually returned
to the OS (regardless of what the FreeBSD malloc() manual page states).
This effect is easy to see. Write a short program that does the following:
1. Allocate on the heap a huge array of elements of size X.
A good choice for X is 16 bytes.
2. Wait for user input to continue.
3. Free the memory.
4. Wait for user input to continue.
5. Allocate on the heap a huge array of elements of size 2X or 3X.
6. Wait for user input to continue.
7. Free the memory.
8. Wait for user input to end the program.
Have 'top' running in another window. What you will see is that the first
batch of memory you allocate appears to still be in use after you free it
and at the end of the program you still seem to have everything allocated.
Most versions of (m|c|re)alloc have a list of variable sized chunks of
memory they keep allocated after a free() under the assumption you will
allocate a chunk about the same size again. These chunks usuallly vary by
16 or 32 bytes up to around 256 after which they no longer cache or pass
you off to mmap behind your back. Allocate a slightly different sized
chunk and you allocate more memory rather than using the cache of already
allocated memory.
So, perhaps we should make sure this isn't what we are seeing before we
bang our heads against the wall trying to track down our "leak". It might
not be there or may be even smaller than we imagine.
Michael
P.S. You can browse the libstdc++ source included with gcc for an example
of exactly how much caching your memory system may be doing behind
your back.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
"Glenn A. Thompson" <gt...@cdr.net> writes:
> What change made it 1/3 the size it used to be? Was it just the
> trail.c changes for scratchpool? Or was there something else?
No, I don't think so. Revision 3589 added the scratchpool, but didn't
change anyone to actually use it yet. The change that brought the
memory usage down was probably revision 3469.
Note how Ben says in the issue that most the memory is not showing up
in the pool statistics at all. His 40MB figure is coming from 'top'!
As far as the pool debug output goes, pool usage hovers at about 3MB.
That's the main clue that it might be a problem with our Berkeley DB
usage...
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>>
> What change made it 1/3 the size it used to be? Was it just the
> trail.c changes for scratchpool? Or was there something else?
I guess what I'm really saying is that I don't see the new sratchpool
being used. So what code was fixed? text_delta.c?
Uh what you talkin 'bout Willis I don't see the changes:-)
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: RFV on issue 860 (httpd memory usage)
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
hey,
Karl Fogel wrote:
>Anyone who's feeling squirrely, please take a look at
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=860
>
>Toward the end, you'll see Ben Collins-Sussman's summary of the
>current situation, about how the bug's severity has been reduced, yet
>there is still some mysterious linearity going on.
>
>
What change made it 1/3 the size it used to be? Was it just the trail.c
changes for scratchpool? Or was there something else?
Thanks,
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org