You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Jan Hendrik <ja...@bigfoot.com> on 2003/10/23 08:40:31 UTC
repos corruption revisited: memory leak?
Hi all out there!
Maybe this can cast some light onto my repos corruption problem.
To me it looks that without obvious reason at times SVN (or
Berkeley) requests all memory in the world (and in space, too) and
fail:
Since the repository became notoriously corrupted here (W2K Pro
SP1, SVN .30, TSVN .19) I withdrew it from production and only
kept a "private" repos with two working copies fed with changes
from different "real" work stations. It contains text files only, all
binary files (mostly JPG & GIF) were dumpfiltered. Both the
working copies and the repos are on my local machine, the repos
is accessed via Apache/http.
This worked without any trouble for several days for about 35
update/commit cycles, some large (up to 700 files in one commit),
some normal (50-100 files), some minor (less than 10 files). The
repos that had become practically unusable in production thereby
raised from rev. 61 (last in production) to rev. 97. Rev. 97
successfully committed about 40 files.
The attempt to update the other working copy to rev. 97 with these
40 files resulted in repos corruption.
After recovering the repos (Apache stopped) immediate corruption
repeated. After several more recoveries I gave up (with the vow to
never ever again use something below version 2 or 3 <g>). The
repos was by no means accessible anymore except for svnadmin
recover. Even the attempt to relocate to file:/// access was futile.
During these futile attempts I monitored memory use, especially of
virtual memory: it skyrocketed to its ceiling and remained there
well after the SVN server had timed out. CPU usage was at an
average of 50-70%. RAM is 144 MB, pagefile 460 MB (fixed).
The next morning after another recover the repository worked again!
No cold or warm reboot in-between but hibernation only! And
memory consumption was low: RAM usage did not go above
80MB, pagefile not above 220 MB.
Can this be a memory leak? As a non-programmer I have no real
idea what this is, but it seems so unlikely to me that a small
commit works, but directly thereafter the update of the other
working copy resulting from this not. And not once, but repeatedly
and constantly. And that _ANY_ command to the repos ends in
corruption while the HDD rotates like mad. After 35 update/commit
cycles of quite different sizes had worked flawlessly.
And that a hibernation "solves" the problem. I assume from this
that hibernation has some refreshing effect on the memory.
Hibernation is controlled by the BIOS here, not by W2K. It's a
notebook from the days before Windows learned power saving and
ACPI.
More physical memory is still on the agenda. But this skyrocketing
of memory usage (on the other machine with 1GB pagefile I got
Windows errors about too little virtual memory) implies that even
several GB physical RAM would not be sufficient in such a case.
Perforce btw suggests 1.5 KB RAM per file or 150 MB sufficient for
100,000 files. With 4500 files I am miles below of this and I would
not expect that SVN would perform that worse.
Jan Hendrik
---------------------------------------
Freedom quote:
The object and practice of liberty lies
in the limitation of governmental power.
-- Gen. Douglas MacArthur
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by Jan Hendrik <ja...@bigfoot.com>.
Concerning Re: repos corruption revisited: mem
John Peacock wrote on 23 Oct 2003, 11:01, at least in part:
> He is running on dangerously low physical RAM machines. Windows (as a
> rule) does not gracefully handle low memory situations. The swap on
> Windows is not employed in quite the same fashion as the swap on *nix
> boxen (meaning that frequently more swap is slower than less).
> Between the Berkeley DLL's and Apache itself, I suspect he has
> virtually no free RAM available. If he configured Apache to start
> fewer child processes, he might be in better shape.
>
> I think he is simply swapping himself to death and triggering edge
> conditions in Berkeley because of it.
This might give the clue why the repos corruption is much more
likely on the P4 machine than on my old P200 notebook: the P4
has 1 GB pagefile, the notebook 460MB only. On the notebook
more virtual memory gave me the overall impression of slower
operation, 460MB seemed to be a good value though. In the setting
dialog Win2K recommends 216MB only, however, else it is said to
set 2.5 times of RAM and others suggest taking 4GB if available.
Of course the P4 shall become the server - if I ever get into
production again! ;-) T
Jan Hendrik
---------------------------------------
Freedom quote:
The explorers of the modern era are the enterpreneurs;
men with vision; with the courage to take risks,
and faith enough to brave the unknown.
-- Ronald Reagan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by Jan Hendrik <ja...@bigfoot.com>.
Hi John, Ben, Mark, J Wynia & all others!
Sorry for getting back so late, but I was busy and wanted to do
some more tests with your valuable information on a cold-booted
machine before ...
Concerning Re: repos corruption revisited: mem
John Peacock wrote on 23 Oct 2003, 11:38, at least in part:
> Ben Collins-Sussman wrote:
>
> > When I fire up my httpd on Linux, I get about 8 mpm-prefork
> > children, each about 3 megs in size. After even the most gigantic
> > svn operations, the largest child never grows bigger than 10 megs.
>
> But he's running Apache under Windows, which is a very different beast
> from Linux. I don't run Apache under Windows, so perhaps someone who
> does could see how much memory the child processes typically use.
According to Apache manual on Windows it only starts two
processes, the rest are threads. Here the first process uses
3,628KB at startup, plus 1,620 KB in virtual memory, and 3
threads. This remains quite constant over time.
The other process runs 252 threads throughout, memory starting
7,032 KB, plus 6,652 in vertual memory. Both these values raise to
about 21/22 MB when committing, little difference if 22 or 2100 files
are committed. Or with more files even less, 15/16MB. At one
place the manual speaks of 50 threads as default for Windows, at
another of 64. 250 is what was set in httpd.conf by the installer.
> And I am also pointing out that Windows doesn't behave well under low
> memory conditions (and I would consider 128MB to be very low memory
> for a Windows server). If he is running a browser and editor, etc. on
> the machine, he could quickly exhaust physical RAM. For example, just
> this moment, Mozilla 1.4 is consuming ~60MB and a Macromedia flash
> player app is taking 16MB on top of that.
For someone who worked well with 32MB under Win95 running big
apps like Word97 and Homesite4/5 concurrently plus filemanager,
MS IE, Pegasus Mail it is difficult to believe 128MB are very low!
Yes, adding 64MB made things much smoother, while another
32MB to the maximum of 144MB on this machine (or has, by
chance, someone tried more on the Dell Inspiron 3000/M200ST?
Some vendors offer 128MB modules for this model, too, though
Kingston and Dell say these are for Inspiron 3200 upwards only)
did not make a recognizable difference. However, the 512MB
module for the P4 machine will be ordered this week, also a
cardbus 10/100 NIC for my notebook to open another possible
bottle neck. So in about 10 days I should know if memory is the
basic issue ...
> > memory consumption was low: RAM usage did not go above
> > 80MB, pagefile not above 220 MB.
>
> That't not the individual child processes, that is total RAM
> consumption by _all_ processes. He also didn't identify what other
> processes were running the day before he had his problems. I am also
> concerned that he is using hibernation with resident server processes.
Right, it were all processes. Mainly Word97, Homesite5, Opera6,
Pegasus Mail, TotalCommander. That's my usual workbench and
also was the days before the problems. However, today's tests
where on a clean, cold-booted machine running nothing but
cmd.exe (shell), taskmanager, Apache, Kerio Firewall. And SVN of
course. I attach a table with what taskmanager told immediately
after booting, during separate commits of 63 changed, 22 added,
2100 added, 1874 changed (one line added each) files.
I do not pretend to be able to really interpret the data. Of course I
see that the first commit caused about 25 MB to be swapped and
that already small commits need about 50MB memory. SVN.exe
had 5.6MB plus 4.6MB virtual in these and 11-18MB plus 8.8-
19MB virtual memory in use during the huge commit. Interesting
that now as I am writing this, with Homesite, Pegasus, Opera,
TotalCommander running, with 92MB I have more available RAM
than after booting. Virtual memory is at 200MB now though.
> I don't know for sure, but I think it would be best to add RAM as a
> first pass. If and until someone else can replicate this, his specific
> environment is a more important clue that the software he is running,
> IMNSHO.
Doing so in a couple of days ... would be glad if that ends this
problem.
Jan Hendrik
---------------------------------------
Freedom quote:
If concern for human poverty and suffering were one's primary motive,
one would seek to discover their cause. One would not fail to ask:
Why did some nations develop, while others did not?
Why have some nations achieved material abundance,
while others have remained stagnant in sub-human misery?
History and specifically the unprecedented prosperity-explosion
of the 19th century would give an immediate answer:
capitalism is the only system that enables men to produce abundance -
and the key to capitalism is individual freedom.
-- Ayn Rand, Requiem for Man
Re: repos corruption revisited: memory leak?
Posted by Mark <ma...@msdhub.com>.
My system is Win2K/Apache2/SVN(0.29). Apache is doing nothing besides
serving svn repositories (pretty small). The machine has 384mb RAM.
Apache has 2 processes on my box as well. One has 1.2MB memory (plus 1.6MB
in swap) and the other has 6.0MB (plus 9.9MB in swap). When I checkout the
biggest repo I have (938 files/1.1MB), the 6MB apache process grows to 6.9MB
and then drops back down after the checkout is complete.
Mark
----- Original Message -----
From: "J Wynia" <jw...@pragmapool.com>
To: <us...@subversion.tigris.org>
Sent: Thursday, October 23, 2003 9:49 AM
Subject: Re: repos corruption revisited: memory leak?
> I run Apache 2/PHP/SVN on WinXP (AMD 1600/1GB). Apache launches 2
> processes when it starts up. One usually uses about 7.8MB of RAM and the
> other varies. Right now it's idling at about 21MB. If I run a hard working
> PHP script, it can spiral to hundreds of megs of RAM. For most SVN
> activity, I don't really see much of a spike at all.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by J Wynia <jw...@pragmapool.com>.
I run Apache 2/PHP/SVN on WinXP (AMD 1600/1GB). Apache launches 2
processes when it starts up. One usually uses about 7.8MB of RAM and the
other varies. Right now it's idling at about 21MB. If I run a hard working
PHP script, it can spiral to hundreds of megs of RAM. For most SVN
activity, I don't really see much of a spike at all.
John Peacock said:
> Ben Collins-Sussman wrote:
>
>> When I fire up my httpd on Linux, I get about 8 mpm-prefork children,
>> each about 3 megs in size. After even the most gigantic svn
>> operations, the largest child never grows bigger than 10 megs.
>
> But he's running Apache under Windows, which is a very different beast
> from
> Linux. I don't run Apache under Windows, so perhaps someone who does
> could see
> how much memory the child processes typically use.
>
> And I am also pointing out that Windows doesn't behave well under low
> memory
> conditions (and I would consider 128MB to be very low memory for a Windows
> server). If he is running a browser and editor, etc. on the machine, he
> could
> quickly exhaust physical RAM. For example, just this moment, Mozilla 1.4
> is
> consuming ~60MB and a Macromedia flash player app is taking 16MB on top of
> that.
>
>>
>> How could this be a problem with swap? He's saying that child httpd
>> processes are growing up to 80 megs and beyond. Are you suggesting
>> that if his machine had a gig of RAM, that his httpd processes
>> *wouldn't* grow to 80 megs and beyond?
>
> No, he didn't say that:
>
>> memory consumption was low: RAM usage did not go above
>> 80MB, pagefile not above 220 MB.
>>
>
> That't not the individual child processes, that is total RAM consumption
> by
> _all_ processes. He also didn't identify what other processes were
> running the
> day before he had his problems. I am also concerned that he is using
> hibernation with resident server processes.
>
> I don't know for sure, but I think it would be best to add RAM as a first
> pass.
> If and until someone else can replicate this, his specific environment is
> a more
> important clue that the software he is running, IMNSHO.
>
> John
>
> --
> John Peacock
> Director of Information Research and Technology
> Rowman & Littlefield Publishing Group
> 4501 Forbes Boulevard
> Suite H
> Lanham, MD 20706
> 301-459-3366 x.5010
> fax 301-429-5748
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
--
J Wynia
Pragmapool, Inc.
www.pragmapool.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by John Peacock <jp...@rowman.com>.
Ben Collins-Sussman wrote:
> When I fire up my httpd on Linux, I get about 8 mpm-prefork children,
> each about 3 megs in size. After even the most gigantic svn
> operations, the largest child never grows bigger than 10 megs.
But he's running Apache under Windows, which is a very different beast from
Linux. I don't run Apache under Windows, so perhaps someone who does could see
how much memory the child processes typically use.
And I am also pointing out that Windows doesn't behave well under low memory
conditions (and I would consider 128MB to be very low memory for a Windows
server). If he is running a browser and editor, etc. on the machine, he could
quickly exhaust physical RAM. For example, just this moment, Mozilla 1.4 is
consuming ~60MB and a Macromedia flash player app is taking 16MB on top of that.
>
> How could this be a problem with swap? He's saying that child httpd
> processes are growing up to 80 megs and beyond. Are you suggesting
> that if his machine had a gig of RAM, that his httpd processes
> *wouldn't* grow to 80 megs and beyond?
No, he didn't say that:
> memory consumption was low: RAM usage did not go above
> 80MB, pagefile not above 220 MB.
>
That't not the individual child processes, that is total RAM consumption by
_all_ processes. He also didn't identify what other processes were running the
day before he had his problems. I am also concerned that he is using
hibernation with resident server processes.
I don't know for sure, but I think it would be best to add RAM as a first pass.
If and until someone else can replicate this, his specific environment is a more
important clue that the software he is running, IMNSHO.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by Ben Collins-Sussman <su...@collab.net>.
John Peacock <jp...@rowman.com> writes:
> He is running on dangerously low physical RAM machines. Windows (as a
> rule) does not gracefully handle low memory situations. The swap on
> Windows is not employed in quite the same fashion as the swap on *nix
> boxen (meaning that frequently more swap is slower than less).
> Between the Berkeley DLL's and Apache itself, I suspect he has
> virtually no free RAM available. If he configured Apache to start
> fewer child processes, he might be in better shape.
>
> I think he is simply swapping himself to death and triggering edge
> conditions in Berkeley because of it.
When I fire up my httpd on Linux, I get about 8 mpm-prefork children,
each about 3 megs in size. After even the most gigantic svn
operations, the largest child never grows bigger than 10 megs.
How could this be a problem with swap? He's saying that child httpd
processes are growing up to 80 megs and beyond. Are you suggesting
that if his machine had a gig of RAM, that his httpd processes
*wouldn't* grow to 80 megs and beyond?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by John Peacock <jp...@rowman.com>.
Ben Collins-Sussman wrote:
>
> So I'm completely at a loss to to understand why you're seeing these
> huge httpd memory sizes. The only difference is that your apache is
> running on win32, and mine on Linux.
If I might remind you of his original reports (from earlier this month):
>>How much swap do you have configured for each machine?
>
> 1 GB on the P4 128 MB RAM box.
>
> 460 MB on the old 144MB RAM machine (tried more once, but
> slowed down overall operation).
>
He is running on dangerously low physical RAM machines. Windows (as a rule)
does not gracefully handle low memory situations. The swap on Windows is not
employed in quite the same fashion as the swap on *nix boxen (meaning that
frequently more swap is slower than less). Between the Berkeley DLL's and
Apache itself, I suspect he has virtually no free RAM available. If he
configured Apache to start fewer child processes, he might be in better shape.
I think he is simply swapping himself to death and triggering edge conditions in
Berkeley because of it.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: repos corruption revisited: memory leak?
Posted by Ben Collins-Sussman <su...@collab.net>.
"Jan Hendrik" <ja...@bigfoot.com> writes:
> Can this be a memory leak? As a non-programmer I have no real
> idea what this is, but it seems so unlikely to me that a small
> commit works, but directly thereafter the update of the other
> working copy resulting from this not. And not once, but repeatedly
> and constantly. And that _ANY_ command to the repos ends in
> corruption while the HDD rotates like mad. After 35 update/commit
> cycles of quite different sizes had worked flawlessly.
I've been testing for memory leaks all week, using very large trees
(the 340meg mozilla tree, with 42,000 smallish files), and I can't
*ever* make httpd grow large.
When I do an import of the mozilla tree, or a checkout, update,
commit... httpd always stays at a tiny 10 megs or less. We found some
*client* memory leaks (which are now fixed in svn 0.32), but never a
server-side leak.
So I'm completely at a loss to to understand why you're seeing these
huge httpd memory sizes. The only difference is that your apache is
running on win32, and mine on Linux.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org