svnserve update + commit hangs system

[Markus Rännare <ma...@coffeestainstudios.com>]

Hi,

The last weeks we have been seeing troubles with our svn server (svn:// 
server).
The repository is quite large (158gb of data), and many files are quite big.

The problem if a person commits a large file (I've maily tested files in 
the size 200-500mb) to the server as someone else makes a update, then 
the whole server computer grinds to a halt and becomes unresponsive. So 
unresponsive that pressing the numlock button on the server keyboard 
doesn't light the keyboard light, and if the screen is running, the same 
screen is just shown with no signs of showing any responsiveness.
This has resulted in me running to the server-room rebooting the server 
all day long today.

Does anyone have any idea how to debug this issue? I tried getting a 
log, (giving log-file to svnserve), that didn't give me much.

Any help would be GREATLY appriciated.

Kind regards/
Markus Rännare

Re: svnserve update + commit hangs system

[David Chapman <dc...@acm.org>]

On 8/10/2011 8:12 AM, Markus Rännare wrote:
> Hi,
>
> The last weeks we have been seeing troubles with our svn server 
> (svn:// server).
> The repository is quite large (158gb of data), and many files are 
> quite big.
>
> The problem if a person commits a large file (I've maily tested files 
> in the size 200-500mb) to the server as someone else makes a update, 
> then the whole server computer grinds to a halt and becomes 
> unresponsive. So unresponsive that pressing the numlock button on the 
> server keyboard doesn't light the keyboard light, and if the screen is 
> running, the same screen is just shown with no signs of showing any 
> responsiveness.
> This has resulted in me running to the server-room rebooting the 
> server all day long today.
>
> Does anyone have any idea how to debug this issue? I tried getting a 
> log, (giving log-file to svnserve), that didn't give me much.
>
> Any help would be GREATLY appriciated.
>
> Kind regards/
> Markus Rännare
>
>

How much memory does the server have, and is it being entirely consumed 
when the server misbehaves in this way?  Is the disk drive screaming 
while this occurs?  These are signs that you don't have enough memory 
and the operating system is page faulting.

Can you run the "top" command on a terminal window connected to the 
server, or logged into the console screen?  It will tell you how much 
memory is free and how much swap space is being used.  There will 
probably be a swap process at the top of the process list if you are 
page faulting to any degree.  The svn:// server process may also appear 
here; you can look at its process time and memory consumption.

Even if you're not running out of memory, you might be able to get more 
information by watching processes with the "top" command.

I am of course assuming a Linux/Unix machine as the server; under 
Windows you can get the same thing by running the Task Manager.  In both 
cases you want the system monitor running before a problem occurs.  If 
you can reproduce it reliably, so much the better - you don't have to 
worry about waking up the screen from screensaver state.  Just launch 
the monitor process, trigger the problem, and then watch on your console 
screen.

-- 
     David Chapman         dcchapman@acm.org
     Chapman Consulting -- San Jose, CA