Re: [Issue 1563] New - TTimo's mysterious 'stalling' checkout

[Brandon Ehle <az...@yahoo.com>]

>------- Additional comments from sussman@tigris.org Tue Oct 21 10:04:32 -0700 2003 -------
>Timothee Besset is having a sometimes-repeatable scalability/perf problem with
>his public repository located at
>
>   https://zerowing.idsoftware.com:666/radiant
>
>Even though the mozilla tree is much larger, I've not been able to
>reproduce this at all in import/checkout tests.  I'm listing all the
>symptoms here in this issue, in hopes that maybe somebody else can
>reproduce.
>
>* server is running httpd-2.0.47, subversion 0.31 tag, bdb 4.0
>
>* tries to checkout /trunk, about 100M of data.  No directory has an
>  absurd number of children, no ridiculously large files either.
>
>* server starts to generate the skelta (a full list of the working
>  copy tree), client immediately begins to parse the response and GET
>  each file.
>
>* at some random point, server suddenly stops sending data to client.
>  gdb reveals that it's deep within svn_repos_dir_delta(), within fs
>  open_path(), within bdb_read_rep(), in __db_cursorchk() or __db_get().
>
>* when we 'continue' and re-break gdb, mod_dav_svn has definitely
>  moved on to a different open_path() call, but still the client is
>  blocking, waiting for data.  (And of course, the client has stopped
>  fetching files too.)
>
>* apparently httpd starts growing very large during this 'silent' period.
>  Way larger than 7 or 10M as usual, but up to 26M and growing.
>
>* eventually the client just times-out waiting for more response, and
>  TTimo ends up killing httpd.
>
>* when TTimo retries the checkout, it doesn't always reproduce.  But when it
>does, it's not stalling at the same point.
>  
>
Is he running apache with mpm-prefork?



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: [Issue 1563] New - TTimo's mysterious 'stalling' checkout

[Brandon Ehle <az...@yahoo.com>]

Timothee Besset wrote:

>Well, would you have more precise informations about the problems you had
>with prefork, was it reported somewhere? Any precisions on the symptoms? I
>might switch to worker (I do have to recompile httpd for that, right?),
>but I still have to find a good explanation on the various threading
>models..
>
>  
>
Well, for the mpm stuff, there is this link
http://httpd.apache.org/docs-2.0/mpm.html

If I remember correctly, when I started to track down why the hang was 
occurring, the first thing I did was recompile httpd, apr, apr-util, 
neon, mod_dav, and mod_dav_svn with optimizations on and -g turned on as 
well.  If this is all new to you, the first thing I'd recommend is 
setting up a script to run lsstack (pstack) on all the apache processes 
(and threads if you were using worker or threadpool).  If it doesn't 
show any symbols in your log, then you should recompile at least 
mod_dav_svn with -g.

When I was tracking down my hang, I ran apache with lsstack logging to a 
file in 1 second intervals and then I went and hung the server.  The 
resulting log was more than enough information to take a guess at what 
was going wrong.

lsstack for Linux (comes with some distros nowadays)
http://sourceforge.net/projects/lsstack/



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org