You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Gerald Richter <ri...@ecos.de> on 2002/06/03 04:31:48 UTC
Problem/Crash doing large checkout via DAV
Hi,
I have imported a large directory into subversion (42MB, 43000Files), when I
try to do a checkout of this directory via DAV it checksout the first
100-150 files and then aborts. Doing the same via a file:// URL works.
To check this out further I have updated subversion to the newest rev (was
2059 at this time), created a new repository and have done the import, then
I do the checkout and get the following:
-------------
venus:/usr/msrc # svn co http://localhost:8000/svn/bb2000/site_perl
A site_perl/msg.61404.00001.tmp~
A site_perl/tempfile.61200.00001.tmp
A site_perl/msg.61452.00001.tmp~
A site_perl/tempfile.61216.00001.tmp
A site_perl/5.6.1
A site_perl/5.6.1/Expect.pod
A site_perl/5.6.1/Bubble.pm
...
A site_perl/5.6.1/DBD/File.pm
A site_perl/5.6.1/DBD/CSV.pm
A site_perl/5.6.1/i586-linux
subversion/libsvn_ra_dav/fetch.c:1001
svn_error: #21093 : <RA layer request failed>
could not fetch directory entries
subversion/libsvn_ra_dav/util.c:80
svn_error: #21093 : <RA layer request failed>
PROPFIND of /svn/bb2000/site_perl/$svn/bc/1/5.6.1/i586-linux: Could not
read status line: connection was closed by server.
-------------
Then I tried to continue with an update and get
-------------
venus:/usr/msrc # cd site_perl/5.6.1/i586-linux
venus:/usr/msrc/site_perl/5.6.1/i586-linux # svn up
subversion/libsvn_ra_dav/util.c:316
svn_error: #21093 : <RA layer request failed>
REPORT request failed on /svn/bb2000/site_perl/5.6.1/i586-linux
subversion/libsvn_ra_dav/util.c:300
svn_error: #21093 : <RA layer request failed>
The REPORT request returned invalid XML in the response: XML parse error
at line 1: xmlParseStartTag: invalid element name
.. (/svn/bb2000/site_perl/5.6.1/i586-linux)
venus:/usr/msrc/site_perl/5.6.1/i586-linux #
-------------
The svn co doesn't leave anything in the Apache error_log, but the svn up
does:
[Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] A failure occurred
during the completion and response generation for the update report. [404,
#21064]
[Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] (2)No such file or
directory: file not found: transaction , path /5.6.1/i586-linux' [404,
#21064]
Any idea what's happeing here?
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by "Stephen C. Tweedie" <sc...@redhat.com>.
Hi,
On Thu, Jun 06, 2002 at 07:39:07AM +0200, Sander Striker wrote:
> Can you try doubling the max locks/etc in repos/db/DB_CONFIG? I'm wondering
> if that will have any impact.
Remember that it doesn't have any impact unless you run a magic
db4_recover sequence after modifying the DB_CONFIG. (We _really_ need
a way of automating this!)
If it's a db4 locking issue, then using ra_local might show up the
same effect (and would probably be easier to debug.)
--Stephen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
>
> Can you try doubling the max locks/etc in repos/db/DB_CONFIG? I'm
wondering
> if that will have any impact.
>
I have increased the locks from 2000 to 4000, run a db_recover -e and done a
svn co -r 0 http://...
svn up
and it worked!
The svn up still get the httd process to 136MB (and it stays at this size),
but that's much better the more the 220 MB (were the machine run out of
memory) for the normal "svn co".
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Problem/Crash doing large checkout via DAV
Posted by Sander Striker <st...@apache.org>.
> From: Gerald Richter [mailto:richter@ecos.de]
> Sent: 06 June 2002 07:28
> > You can work around the problem by checking-out revision zero and
> > then updating the working copy
> >
> > % svn co -r 0 -d wc http://host/repo
> > % svn up wc
> >
> > These commands use much less memory in the server process (about
> > twenty megabytes instead of two or three hundred in my tests.)
> >
>
> That makes things better, now httpd grows to 130MB during the update, but I
> still get only 2/3 of the files out of the repository then the server closes
> the connection (without any error message in syslog or error_log).
>
> BTW. I my repository only contains 4200 files and not 42,000 as I originaly
> wrote. Must have been too tired when I read this extra zero... (but really
> 45MByte)
>
> After this update the DB Files are soemhow locked and the only thing that
> had help is to run a db_recover on the db files, otherwise any access to the
> repository simply hangs.
>
> When I now try to run a svn up in the half checked out working copy, I get
> errors about file not found in transaction. Seems there is still something
> left from the failed update, also I have rebooted the machine in between.
>
> I try to investigate further. Any ideas are welcome.
Can you try doubling the max locks/etc in repos/db/DB_CONFIG? I'm wondering
if that will have any impact.
> Gerald
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
>
> You can work around the problem by checking-out revision zero and
> then updating the working copy
>
> % svn co -r 0 -d wc http://host/repo
> % svn up wc
>
> These commands use much less memory in the server process (about
> twenty megabytes instead of two or three hundred in my tests.)
>
That makes things better, now httpd grows to 130MB during the update, but I
still get only 2/3 of the files out of the repository then the server closes
the connection (without any error message in syslog or error_log).
BTW. I my repository only contains 4200 files and not 42,000 as I originaly
wrote. Must have been too tired when I read this extra zero... (but really
45MByte)
After this update the DB Files are soemhow locked and the only thing that
had help is to run a db_recover on the db files, otherwise any access to the
repository simply hangs.
When I now try to run a svn up in the half checked out working copy, I get
errors about file not found in transaction. Seems there is still something
left from the failed update, also I have rebooted the machine in between.
I try to investigate further. Any ideas are welcome.
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Philip Martin <ph...@codematters.co.uk>.
"Gerald Richter" <ri...@ecos.de> writes:
> >
> > Gerald, could memory usage be what is causing your check-out to fail?
> > I can easily drive my machine (512M RAM, 1G swap) first into swap and
> > then beyond its swap limit using a repository with a few hundred files
> > in one directory. When the Linux OOM killer kills the Apache httpd
> > process I see a message in the syslog but not in the Apache log.
> >
>
> Yes, that is it!!
>
> While doing the PROPFIND for the directory with 350 file, the Apache process
> grows to over 220MB and then is killed by the Linux OOM killer (my test
> machine has only 128KB of real memory). When I run a filesystem checkout,
> the svn command never grows over 14MB, so it seems a issue of
> mod_dav/mod_dav_svn?
>
> What is mod_dav doing that it needs 750KB per file for a PROPFIND? That
> really seems very very much?
>
> I like svn very much and every day I use it and discover new features I like
> it even more, but this seems to be a real show stopper...
You can work around the problem by checking-out revision zero and
then updating the working copy
% svn co -r 0 -d wc http://host/repo
% svn up wc
These commands use much less memory in the server process (about
twenty megabytes instead of two or three hundred in my tests.)
--
Philip
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
>
> Gerald, could memory usage be what is causing your check-out to fail?
> I can easily drive my machine (512M RAM, 1G swap) first into swap and
> then beyond its swap limit using a repository with a few hundred files
> in one directory. When the Linux OOM killer kills the Apache httpd
> process I see a message in the syslog but not in the Apache log.
>
Yes, that is it!!
While doing the PROPFIND for the directory with 350 file, the Apache process
grows to over 220MB and then is killed by the Linux OOM killer (my test
machine has only 128KB of real memory). When I run a filesystem checkout,
the svn command never grows over 14MB, so it seems a issue of
mod_dav/mod_dav_svn?
What is mod_dav doing that it needs 750KB per file for a PROPFIND? That
really seems very very much?
I like svn very much and every day I use it and discover new features I like
it even more, but this seems to be a real show stopper...
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Philip Martin <ph...@codematters.co.uk>.
Greg Stein <gs...@lyra.org> writes:
> On Tue, Jun 04, 2002 at 10:18:52AM -0500, Ben Collins-Sussman wrote:
> > "Gerald Richter" <ri...@ecos.de> writes:
> >...
> > > (EVerything is on the same machine, so there is no network traffic
> > > time involved)
> >
> > I *thought* that the server would be streaming the humongous
> > 40,000-entry response, so client-side parsing can start right
> > away... we have a brigade for that now, or something. Maybe gstein
> > can comment?
>
> Actually, PROPFIND is not (yet) streamy. I switched over REPORT reponses a
> while back, and content has always been streamy.
>
> So if *one* directory had 40k files, then there could be problems :-)
PROPFIND certainly doesn't scale well at present.
When checking-out a repository over ra_dav with 100 files in a single
directory the PROPFIND takes less than 1 second, with 200 files it's
about 2 seconds, with 300 files it's 5 seconds, with 400 files it's 14
seconds and with 600 files it is 45 seconds. Memory usage in the
server doesn't appear to scale well either. Is switching to a streamy
interface likely to fix these problems?
Gerald, could memory usage be what is causing your check-out to fail?
I can easily drive my machine (512M RAM, 1G swap) first into swap and
then beyond its swap limit using a repository with a few hundred files
in one directory. When the Linux OOM killer kills the Apache httpd
process I see a message in the syslog but not in the Apache log.
I created the files in each repository with a single commit, using the
tools/dev/stress.pl script. The commit doesn't scale well either, but
that does not seem to be ra_dav related as the poor scaling occurs
over ra_local as well. The bit that doesn't scale is whatever happens
after 'Transmitting file data ........'
To create a repository with 200 files using stress.pl I use
% sw/subversion/svn/tools/dev/stress.pl -c -F200 -N1 -D1 -n0
% svn co -d wc http://localhost:8888/repostress
--
Philip
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Jun 04, 2002 at 10:18:52AM -0500, Ben Collins-Sussman wrote:
> "Gerald Richter" <ri...@ecos.de> writes:
>...
> > (EVerything is on the same machine, so there is no network traffic
> > time involved)
>
> I *thought* that the server would be streaming the humongous
> 40,000-entry response, so client-side parsing can start right
> away... we have a brigade for that now, or something. Maybe gstein
> can comment?
Actually, PROPFIND is not (yet) streamy. I switched over REPORT reponses a
while back, and content has always been streamy.
So if *one* directory had 40k files, then there could be problems :-)
However, Gerald followed up to say that is in all directories recursively.
That should not be a problem. We send a PROPFIND for just a directory and
its immediate children. Note that mod_dav will (normally) reject a "depth
infinity" PROPFIND.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
> >
> > Yes, it's quite possible that it's a time-out. Neon (the client-side
> > http library) is probably timing out after 90 secs...
> >
>
> Ok, I try to compile neon with a bigger timeout and see what will
happen...
>
I have set the SOCKET_READ_TIMEOUT from 120 to 12000, but this doesn't
change anything, since it had stop before 120secs are over, this isn't very
suprising...
> >
> > Regardless, repeat the expermiment while using Ethereal to capture the
> > data. When the client dies, take a look at the HTTP dialogue and see
> > how much data was sent.
> >
>
The client sends the PROPFIND for the directory and server doesn't send any
answer to back to the client. Just closeing the connection after a while. I
have rerun this serveral times, the time it takes changes, so it doesn't
seems like a timeout. Would have been too easy :-(
Also ethereal shows that nothing is send back to the client, the access_log,
says status 207 and 900 Bytes for that PORPFIND request. There is nothing in
the error_log. Strange....
If this doesn't makes sense to anybody, seems like I have to try the
debugger when I get some time....
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
>
> Yes, it's quite possible that it's a time-out. Neon (the client-side
> http library) is probably timing out after 90 secs...
>
Ok, I try to compile neon with a bigger timeout and see what will happen...
> > Is it normal that a PROPFIND for a large directory (about 40MB & 40,000
> > files) takes such a long time.
>
> Ay-ya-yay! 40,000 entries in one directory!!
>
Not directly, the directory contains 350 files, but the number of all files
in that directory and all it's subdirectories are about 40,000
>
> Regardless, repeat the expermiment while using Ethereal to capture the
> data. When the client dies, take a look at the HTTP dialogue and see
> how much data was sent.
>
I do this a little bit later an let you know what is happeing
Thanks for your response
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Ben Collins-Sussman <su...@collab.net>.
"Gerald Richter" <ri...@ecos.de> writes:
> Is it possible that this is a simple time-out? The Apache TimeOut is
> set to 300, but maybe there are some other time-out values either server or
> client side?
Yes, it's quite possible that it's a time-out. Neon (the client-side
http library) is probably timing out after 90 secs...
> Is it normal that a PROPFIND for a large directory (about 40MB & 40,000
> files) takes such a long time.
Ay-ya-yay! 40,000 entries in one directory!!
> Does the svn server recurse through the subdirs at this point or is
> it a part of the error that it takes such a long time?
I believe that for a DAV-based checkout, a PROPFIND "get all props"
request is sent on the directory. The server then sends a big list of
entries as an xml body-response. The client parses the xml, and
starts doing GETs and PROPFINDs on each individual file.
> (EVerything is on the same machine, so there is no network traffic
> time involved)
I *thought* that the server would be streaming the humongous
40,000-entry response, so client-side parsing can start right
away... we have a brigade for that now, or something. Maybe gstein
can comment?
Regardless, repeat the expermiment while using Ethereal to capture the
data. When the client dies, take a look at the HTTP dialogue and see
how much data was sent.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
> >
> > Any logs/debug infos I can enable to get more infos?
>
> Well, yeah... if the bug is perfectly reproducible, the best thing you
> can do is run httpd inside gdb and figger out the problem!
>
"figger out the problem" is always the best thing you can do!
> Make sure you compile both httpd and svn with --enable-maintaine-mode,
> and then read the section in our HACKING file on "Debugging the
> server".
>
Also I am familar with debuging Apache modules, you know debugging an more
or less unknown software of this size is not an 5 minute task...
So maybe you can give me some more hints. I rerun the checkout via DAV and
noticed when it comes to the point where it fails it does a PROPFIND on the
directory i586-linux. This directory is very large and the server seems to
go through the data (I see a lot of disk access), after 90 sec I get the
error. Is it possible that this is a simple time-out? The Apache TimeOut is
set to 300, but maybe there are some other time-out values either server or
client side?
Is it normal that a PROPFIND for a large directory (about 40MB & 40,000
files) takes such a long time. Does the svn server recurse through the
subdirs at this point or is it a part of the error that it takes such a long
time? (EVerything is on the same machine, so there is no network traffic
time involved)
Thanks
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Ben Collins-Sussman <su...@collab.net>.
"Gerald Richter" <ri...@ecos.de> writes:
> >
> > I'm not sure what's up here... assuming that the repository was
> > created via import over ra_local, maybe there's a bogus character in a
> > filename that's choking up http?
> >
>
> No, it was created via import over DAV.
>
> Any logs/debug infos I can enable to get more infos?
Well, yeah... if the bug is perfectly reproducible, the best thing you
can do is run httpd inside gdb and figger out the problem!
Make sure you compile both httpd and svn with --enable-maintaine-mode,
and then read the section in our HACKING file on "Debugging the
server".
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Gerald Richter <ri...@ecos.de>.
>
> I'm not sure what's up here... assuming that the repository was
> created via import over ra_local, maybe there's a bogus character in a
> filename that's choking up http?
>
No, it was created via import over DAV.
Any logs/debug infos I can enable to get more infos?
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by Ben Collins-Sussman <su...@collab.net>.
cmpilato@collab.net writes:
> "Gerald Richter" <ri...@ecos.de> writes:
>
> > The svn co doesn't leave anything in the Apache error_log, but the svn up
> > does:
> >
> > [Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] A failure occurred
> > during the completion and response generation for the update report. [404,
> > #21064]
> > [Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] (2)No such file or
> > directory: file not found: transaction , path /5.6.1/i586-linux' [404,
> > #21064]
> >
> >
> > Any idea what's happeing here?
>
> I'm not sure, but my first guess is that perhaps this is related to a
> recent change in the formatting of the "magic URLs" that get
> transmitted between client and server. Those are "old style" URLs,
> and probably just reflect places in our code base where we aren't
> gracefully updating those to "new style" URLs like we are supposed to
> be doing.
>
> Sussman, whatcha think?
Ummm, nope, there are no "new style" URLs being generated anywhere,
except in a private patch I'm testing on my own box. :-)
I'm not sure what's up here... assuming that the repository was
created via import over ra_local, maybe there's a bogus character in a
filename that's choking up http?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Problem/Crash doing large checkout via DAV
Posted by cm...@collab.net.
"Gerald Richter" <ri...@ecos.de> writes:
> The svn co doesn't leave anything in the Apache error_log, but the svn up
> does:
>
> [Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] A failure occurred
> during the completion and response generation for the update report. [404,
> #21064]
> [Sun Jun 02 11:54:09 2002] [error] [client 127.0.0.1] (2)No such file or
> directory: file not found: transaction , path /5.6.1/i586-linux' [404,
> #21064]
>
>
> Any idea what's happeing here?
I'm not sure, but my first guess is that perhaps this is related to a
recent change in the formatting of the "magic URLs" that get
transmitted between client and server. Those are "old style" URLs,
and probably just reflect places in our code base where we aren't
gracefully updating those to "new style" URLs like we are supposed to
be doing.
Sussman, whatcha think?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org