You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Noel Yap <ya...@yahoo.com> on 2002/10/09 21:31:45 UTC
problems with large commits
I'm not sure exactly what resource is used up or
whether it's on the client or the server, but large
imports or large commits fail (I forgot to copy the
exact error).
I was wondering if it would be possible to chop up the
commits into several operations but still have it
still be one transaction.
One way I can think of to get this to happen is if
svnadmin had a way to aggregate a bunch of
transactions into one.
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Jeff Stuart <fu...@weboverdrive.com>.
I've also had this happen a while ago with LARGE (IE 4750+ files)
checkout. What happens it turns out is that I was running out of memory
and the kernel killed the process for me. My httpd procs were getting
up into the 250+ mb range. :( I haven't had a chance to check it
RECENTLY (IE this was over a week ago). Nor have I had a chance to test
it with the latest CVS of APR and APR-UTIL.
On Wed, 2002-10-09 at 17:31, Noel Yap wrote:
> I'm not sure exactly what resource is used up or
> whether it's on the client or the server, but large
> imports or large commits fail (I forgot to copy the
> exact error).
>
> I was wondering if it would be possible to chop up the
> commits into several operations but still have it
> still be one transaction.
>
> One way I can think of to get this to happen is if
> svnadmin had a way to aggregate a bunch of
> transactions into one.
>
> Thanks,
> Noel
>
> __________________________________________________
> Do you Yahoo!?
> Faith Hill - Exclusive Performances, Videos & More
> http://faith.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Paul Lussier <pl...@lanminds.com>.
In a message dated: Thu, 10 Oct 2002 12:11:54 PDT
Noel Yap said:
>Since I'm pressed for time, can you give me some quick
>pointers as to how to use ethereal:
>1. Should it run on the client or the server?
>2. How should it be executed?
You can run it on either, but probably you want to run it on the
server to make sure the server is even seeing the traffic. If not,
then run it on the client to find out why the http traffic isn't
making it there.
Sorry if that's not clear. I just saw this post and haven't paid
attention to this thread.
I've used ethereal before, so I may be able to help you out with
using it (though I'm by no means an expert with it!).
--
Seeya,
Paul
--
It may look like I'm just sitting here doing nothing,
but I'm really actively waiting for all my problems to go away.
If you're not having fun, you're not doing it right!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Ben Collins-Sussman <su...@collab.net>.
Noel Yap <ya...@yahoo.com> writes:
> --- Ben Collins-Sussman <su...@collab.net> wrote:
> > If this is true, and if you can *still* reprodece
> > the MKCOL failure
> > from the client, then please capture the http
> > dialogue with ethereal
> > or something!
>
> Since I'm pressed for time, can you give me some quick
> pointers as to how to use ethereal:
> 1. Should it run on the client or the server?
> 2. How should it be executed?
On the client, run ethereal as root.
Go to the capture->start menu.
Choose the correct ethernet interface.
Type "port 80" in the 'Filter' field.
Press OK, and a little dialog pops up, waiting for packets.
Run your svn command.
Press OK on the little dialog.
Back in ethereal's main window, right-click on on the first
packet-line, and select "follow TCP stream".
A new window pops up that shows the whole HTTP conversation.
That window also has a "save as" button.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Ben Collins-Sussman <su...@collab.net> wrote:
> If this is true, and if you can *still* reprodece
> the MKCOL failure
> from the client, then please capture the http
> dialogue with ethereal
> or something!
Since I'm pressed for time, can you give me some quick
pointers as to how to use ethereal:
1. Should it run on the client or the server?
2. How should it be executed?
> We've seen cases before where the protocol itself
> messes up, for
> example where apache starts responding before the
> request is
> finished...
I hope I can at least help track down the problem.
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Ben Collins-Sussman <su...@collab.net>.
Noel Yap <ya...@yahoo.com> writes:
> --- Ben Collins-Sussman <su...@collab.net> wrote:
> > > I didn't mention that the error I'm getting occurs
> > > during the "Adding" phase of the commit, not the
> > > "Transmitting". Do you think this is a
> > client-side
> > > thing? Or is it still possible that it's a
> > > server-side thing?
> >
> > Yeah, curious. I think this is an apache or
> > mod_dav_svn bug.
>
> Interesting... I tried doing the commit from the host
> server, still going through DAV, and it's working with
> no problem (at least it got to the "Transmitting"
> phase).
>
> It sounds like the problem is from the client unless
> there's still something I'm not thinking about.
If this is true, and if you can *still* reprodece the MKCOL failure
from the client, then please capture the http dialogue with ethereal
or something!
We've seen cases before where the protocol itself messes up, for
example where apache starts responding before the request is
finished...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Ben Collins-Sussman <su...@collab.net> wrote:
> > I didn't mention that the error I'm getting occurs
> > during the "Adding" phase of the commit, not the
> > "Transmitting". Do you think this is a
> client-side
> > thing? Or is it still possible that it's a
> > server-side thing?
>
> Yeah, curious. I think this is an apache or
> mod_dav_svn bug.
Interesting... I tried doing the commit from the host
server, still going through DAV, and it's working with
no problem (at least it got to the "Transmitting"
phase).
It sounds like the problem is from the client unless
there's still something I'm not thinking about.
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Ben Collins-Sussman <su...@collab.net> wrote:
> Noel Yap <ya...@yahoo.com> writes:
>
> > I didn't mention that the error I'm getting occurs
> > during the "Adding" phase of the commit, not the
> > "Transmitting". Do you think this is a
> client-side
> > thing? Or is it still possible that it's a
> > server-side thing?
>
> Yeah, curious. I think this is an apache or
> mod_dav_svn bug.
>
> Your error says that the http MKCOL request never
> completes. MKCOL is
> just the DAV method for creating a new, empty
> directory. Neon
> complains that the request failed because it 'could
> not read status
> line', i.e. the server dropped the connection. In
> my experience, this
> almost always indicates that the httpd child-worker
> process
> segfaulted.
>
> The thing to do here is start apache in gdb, and
> 'run -X', which
> forces apache to run single-threaded. (This also
> assumes that you've
> compiled both apache and svn with
> --enable-maintainer-mode for
> debugging symbols.)
>
> Now attempt your commit again, and see if/when
> apache dies...
I'll see if I can get this done but, right now, it
doesn't look likely.
The local commit succeeded which supports your theory
(although it doesn't rule out something happening on
the client). If I run into the problem again, I'll
try going through Apache from the repo server. If it
still fails, it's an extremely good indication that it
is what you suspect.
Actually, I'll see if I can set up another repo to
test with. I'll let you know if I get the chance to
do this.
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Ben Collins-Sussman <su...@collab.net>.
Noel Yap <ya...@yahoo.com> writes:
> I didn't mention that the error I'm getting occurs
> during the "Adding" phase of the commit, not the
> "Transmitting". Do you think this is a client-side
> thing? Or is it still possible that it's a
> server-side thing?
Yeah, curious. I think this is an apache or mod_dav_svn bug.
Your error says that the http MKCOL request never completes. MKCOL is
just the DAV method for creating a new, empty directory. Neon
complains that the request failed because it 'could not read status
line', i.e. the server dropped the connection. In my experience, this
almost always indicates that the httpd child-worker process
segfaulted.
The thing to do here is start apache in gdb, and 'run -X', which
forces apache to run single-threaded. (This also assumes that you've
compiled both apache and svn with --enable-maintainer-mode for
debugging symbols.)
Now attempt your commit again, and see if/when apache dies...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Philip Martin <ph...@codematters.co.uk>.
Karl Fogel <kf...@newton.ch.collab.net> writes:
> Justin Erenkrantz <je...@apache.org> writes:
> > It's a design flaw in how mod_dav walks the tree - basically, it needs
> > to store the response in a separate pool and use an iterative subpool
> > for each file that it looks up. Right now, we can't use an iterative
> > subpool approach because the response is stored in the same pool as
> > the file lookups. So, the memory growth is linear based on how many
> > files are in a directory.
It's worse than linear.
> >
> > It's somewhere on my todo list, but if someone else wants to tackle
> > it, please be my guest. =) -- justin
>
> Is there an issue for it?
Yes, number 773.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Justin Erenkrantz <je...@apache.org> writes:
> It's a design flaw in how mod_dav walks the tree - basically, it needs
> to store the response in a separate pool and use an iterative subpool
> for each file that it looks up. Right now, we can't use an iterative
> subpool approach because the response is stored in the same pool as
> the file lookups. So, the memory growth is linear based on how many
> files are in a directory.
>
> It's somewhere on my todo list, but if someone else wants to tackle
> it, please be my guest. =) -- justin
Is there an issue for it?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Justin Erenkrantz <je...@apache.org>.
It's a design flaw in how mod_dav walks the tree - basically, it needs to
store the response in a separate pool and use an iterative subpool for each
file that it looks up. Right now, we can't use an iterative subpool
approach because the response is stored in the same pool as the file
lookups. So, the memory growth is linear based on how many files are in a
directory.
It's somewhere on my todo list, but if someone else wants to tackle it,
please be my guest. =) -- justin
--On Thursday, October 10, 2002 12:20:29 -0400 Jeff Stuart
<js...@computer-city.net> wrote:
> OUCH!!!! NO WONDER my one check out failed! Does anyone know why that
> much memory is getting used? IE I assume that it's a bug yes? (I hope.
> :))
>
> On Thu, 2002-10-10 at 11:17, Philip Martin wrote:
>> The last time I looked, a PROPFIND on a directory with 800 hundred
>> elements used over 200MB of RAM, and the memory use scaled worse than
>> linearly with the number of elements :-(
>
> --
> Jeff Stuart <js...@computer-city.net>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Jeff Stuart <js...@computer-city.net>.
OUCH!!!! NO WONDER my one check out failed! Does anyone know why that
much memory is getting used? IE I assume that it's a bug yes? (I hope.
:))
On Thu, 2002-10-10 at 11:17, Philip Martin wrote:
> The last time I looked, a PROPFIND on a directory with 800 hundred
> elements used over 200MB of RAM, and the memory use scaled worse than
> linearly with the number of elements :-(
--
Jeff Stuart <js...@computer-city.net>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Philip Martin <ph...@codematters.co.uk> wrote:
> The last time I looked, a PROPFIND on a directory
> with 800 hundred
> elements used over 200MB of RAM, and the memory use
> scaled worse than
> linearly with the number of elements :-(
>
> Is your server exceeding a process limit? Are the
> servers ulimit
> settings sensible?
I'm not sure. In anycase, even if I were to find out,
I wouldn't be able to do anything about it since I'm
trying to keep my use of Subversion under the radar
for now.
I guess I'll use the local interface whenever I run
into this problem. Hopefully, it'll get resolved soon
(or at least before 1.0).
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Philip Martin <ph...@codematters.co.uk>.
Noel Yap <ya...@yahoo.com> writes:
> Yes, there're lots of PROPFIND's on the directory
> containing lots of items. I did notice that the
> problem seems to occur around the time when the
> directory changes.
The last time I looked, a PROPFIND on a directory with 800 hundred
elements used over 200MB of RAM, and the memory use scaled worse than
linearly with the number of elements :-(
Is your server exceeding a process limit? Are the servers ulimit
settings sensible?
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Philip Martin <ph...@codematters.co.uk> wrote:
> Noel Yap <ya...@yahoo.com> writes:
>
> > --- Ben Collins <bc...@debian.org> wrote:
> > > What's the error on the server?
> >
> > The apache error log isn't showing anything. Are
> > there any other logs I should look at?
>
> The access log. What is happening there? Are there
> any PROPFINDS on
> directories containing lots of files? A PROPFIND on
> a directory
> containing lots of items is a known problem, see
> issue 773.
Yes, there're lots of PROPFIND's on the directory
containing lots of items. I did notice that the
problem seems to occur around the time when the
directory changes.
OTOH, I'm running into this problem upon import and
checkin. I haven't yet tried to do a checkout so I'll
have to see if I run into the problem there, too.
> > I didn't mention that the error I'm getting occurs
> > during the "Adding" phase of the commit, not the
> > "Transmitting". Do you think this is a
> client-side
> > thing? Or is it still possible that it's a
> > server-side thing?
>
> Is your server running on Linux? Is it being killed
> by the the OOM
> killer? Take a look in the syslog.
I'm running the server on a Solaris 2.8 machine. The
only stuff I see within syslog aren't related to
subversion or apache.
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Philip Martin <ph...@codematters.co.uk>.
Noel Yap <ya...@yahoo.com> writes:
> --- Ben Collins <bc...@debian.org> wrote:
> > What's the error on the server?
>
> The apache error log isn't showing anything. Are
> there any other logs I should look at?
The access log. What is happening there? Are there any PROPFINDS on
directories containing lots of files? A PROPFIND on a directory
containing lots of items is a known problem, see issue 773.
>
> I didn't mention that the error I'm getting occurs
> during the "Adding" phase of the commit, not the
> "Transmitting". Do you think this is a client-side
> thing? Or is it still possible that it's a
> server-side thing?
Is your server running on Linux? Is it being killed by the the OOM
killer? Take a look in the syslog.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Ben Collins <bc...@debian.org> wrote:
> What's the error on the server?
The apache error log isn't showing anything. Are
there any other logs I should look at?
I didn't mention that the error I'm getting occurs
during the "Adding" phase of the commit, not the
"Transmitting". Do you think this is a client-side
thing? Or is it still possible that it's a
server-side thing?
> Try this over
> ra_local so you can see
> what's actually going on. Sounds like you are
> running out of memory and
> the apache proc is being killed.
I'll let you know what happens when I try this.
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Ben Collins <bc...@debian.org>.
On Thu, Oct 10, 2002 at 05:44:44AM -0700, Noel Yap wrote:
> --- Ben Collins <bc...@debian.org> wrote:
> > Look at db/DB_CONFIG in the repo. Increase the locks
> > available (for the
> > kernel imports I tested, I used 10000 for each).
> > After changing them
> > you'll need to do:
> >
> > cd db
> > db_recover -ve -h .
>
> I now have the following in my DB_CONFIG file:
> set_lk_max_locks 65536
> set_lk_max_lockers 4096
> set_lk_max_objects 4096
> set_lg_bsize 1048576
> set_lg_max 4194304
>
> but I'm still getting the error:
> svn: RA layer request failed
> svn: Commit failed (details follow):
> svn: MKCOL of
> /svn-repo/!svn/wrk/c23a74d8-a5ac-0310-a8f3-a252d8919f93/olf/main/LATEST/sql/complete:
> Could not read status line: Connection aborted
>
> during a large commit. It's not even that large; less
> than 2000 files in a couple of directories. Could the
> problem be the number of files within one directory?
What's the error on the server? Try this over ra_local so you can see
what's actually going on. Sounds like you are running out of memory and
the apache proc is being killed.
--
Debian - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo - http://www.deqo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Noel Yap <ya...@yahoo.com>.
--- Ben Collins <bc...@debian.org> wrote:
> Look at db/DB_CONFIG in the repo. Increase the locks
> available (for the
> kernel imports I tested, I used 10000 for each).
> After changing them
> you'll need to do:
>
> cd db
> db_recover -ve -h .
I now have the following in my DB_CONFIG file:
set_lk_max_locks 65536
set_lk_max_lockers 4096
set_lk_max_objects 4096
set_lg_bsize 1048576
set_lg_max 4194304
but I'm still getting the error:
svn: RA layer request failed
svn: Commit failed (details follow):
svn: MKCOL of
/svn-repo/!svn/wrk/c23a74d8-a5ac-0310-a8f3-a252d8919f93/olf/main/LATEST/sql/complete:
Could not read status line: Connection aborted
during a large commit. It's not even that large; less
than 2000 files in a couple of directories. Could the
problem be the number of files within one directory?
Thanks,
Noel
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: problems with large commits
Posted by Ben Collins <bc...@debian.org>.
On Wed, Oct 09, 2002 at 02:31:45PM -0700, Noel Yap wrote:
> I'm not sure exactly what resource is used up or
> whether it's on the client or the server, but large
> imports or large commits fail (I forgot to copy the
> exact error).
Look at db/DB_CONFIG in the repo. Increase the locks available (for the
kernel imports I tested, I used 10000 for each). After changing them
you'll need to do:
cd db
db_recover -ve -h .
...for the change to take affect.
--
Debian - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo - http://www.deqo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org