You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Jon Bendtsen <jb...@laerdal.dk> on 2008/03/12 13:45:32 UTC
Why does SVN + apache try to close non existing file descriptors?
Hi
I have a problem with SVN and/or apache which does not apply to
svnserve.
I use svn log to test this, but my users complain that any SVN command
is slow.
Using svnserve directly it takes less than a second to get an output
of svn log.
Using SVN through apache it takes about 28 minutes to get a log.
So i used strace on apache and it turns out it tries to close
1024*1024 file descriptors
more than once. Every time takes almost a minute. It starts 41 times,
but for some reason
it does not count to 1024*1024 every time. But it does count to
1024*1024 22 times,
and some times only to some hundred thousand times. The error message
is like this
13:47:17 close(227260) = -1 EBADF (Bad file descriptor)
Most of the times the processes only have a few files open, so
anything after 2 or 3
gets an error message. This means i get a strace file with 1024 * 1024
error lines.
Q) why brute force close all files?
a) ask the kernel to close all my open files
b) ask the kernel which files do I have open and then close only those
c) remember what files i have open and then close only those
I "solved" the slowness problem by setting the hardlimit to 1024 pr.
apache
process, but using svn+apache is still slower than svnserve, and svn
+apache
still tries to close about 1020 non existing files, which i find is a
waste of resources.
This is ls of all strace files from this command:
for pid in $(ps aux | grep apache | cut -b 10-15); do strace -t -p
$pid -o slow.$pid -ff & done
As you can see, some files get pretty big, so svn+apache must waste
alot of resources trying
to close 1024*1024 nonexisting files.
-rw-r--r-- 1 root root 0 Mar 12 13:21 slow.5928
-rw-r--r-- 1 root root 76556314 Mar 12 13:22 slow.5318.6075
-rw-r--r-- 1 root root 76556314 Mar 12 13:23 slow.5318.6337
-rw-r--r-- 1 root root 76556314 Mar 12 13:24 slow.5318.6394
-rw-r--r-- 1 root root 71695658 Mar 12 13:25 slow.5307.6371
-rw-r--r-- 1 root root 76556379 Mar 12 13:26 slow.5321.6765
-rw-r--r-- 1 root root 76556314 Mar 12 13:26 slow.5318.7157
-rw-r--r-- 1 root root 72944829 Mar 12 13:26 slow.5307.6450
-rw-r--r-- 1 root root 76556314 Mar 12 13:28 slow.5321.7903
-rw-r--r-- 1 root root 76556314 Mar 12 13:29 slow.5318.8002
-rw-r--r-- 1 root root 76556379 Mar 12 13:30 slow.5320.8124
-rw-r--r-- 1 root root 138916 Mar 12 13:31 slow.5307.9705
-rw-r--r-- 1 root root 136108 Mar 12 13:31 slow.5307.9704
-rw-r--r-- 1 root root 137098 Mar 12 13:31 slow.5307.9733
-rw-r--r-- 1 root root 76556379 Mar 12 13:32 slow.5322.8779
-rw-r--r-- 1 root root 76556314 Mar 12 13:32 slow.5321.8815
-rw-r--r-- 1 root root 76556379 Mar 12 13:32 slow.5319.8855
-rw-r--r-- 1 root root 638243 Mar 12 13:32 slow.5307.9707
-rw-r--r-- 1 root root 1206081 Mar 12 13:33 slow.5307.9708
-rw-r--r-- 1 root root 76556314 Mar 12 13:33 slow.5318.9306
-rw-r--r-- 1 root root 126265 Mar 12 13:33 slow.5307.9048
-rw-r--r-- 1 root root 76556314 Mar 12 13:35 slow.5319.10001
-rw-r--r-- 1 root root 76556314 Mar 12 13:35 slow.5322.9850
-rw-r--r-- 1 root root 474476 Mar 12 13:35 slow.5307.6778
-rw-r--r-- 1 root root 76556314 Mar 12 13:35 slow.5318.10705
-rw-r--r-- 1 root root 76556314 Mar 12 13:37 slow.5319.11034
-rw-r--r-- 1 root root 76556314 Mar 12 13:37 slow.5318.11059
-rw-r--r-- 1 root root 67824798 Mar 12 13:38 slow.5307.7655
-rw-r--r-- 1 root root 76556314 Mar 12 13:39 slow.5319.12559
-rw-r--r-- 1 root root 76556314 Mar 12 13:39 slow.5318.12629
-rw-r--r-- 1 root root 65815368 Mar 12 13:41 slow.5307.8012
-rw-r--r-- 1 root root 76556314 Mar 12 13:42 slow.5319.12825
-rw-r--r-- 1 root root 76556314 Mar 12 13:43 slow.5318.12836
-rw-r--r-- 1 root root 76556379 Mar 12 13:43 slow.5322.13169
-rw-r--r-- 1 root root 66366551 Mar 12 13:44 slow.5307.8129
-rw-r--r-- 1 root root 76556314 Mar 12 13:45 slow.5319.14382
-rw-r--r-- 1 root root 76556314 Mar 12 13:46 slow.5318.14452
-rw-r--r-- 1 root root 76556314 Mar 12 13:46 slow.5322.14461
-rw-r--r-- 1 root root 48226304 Mar 12 13:47 slow.5319.15656
-rw-r--r-- 1 root root 35885056 Mar 12 13:47 slow.5318.15907
-rw-r--r-- 1 root root 61104128 Mar 12 13:47 slow.5307.9356
-rw-r--r-- 1 root root 63528960 Mar 12 13:47 slow.5307.9044
-rw-r--r-- 1 root root 28590080 Mar 12 13:47 slow.5307.12631
-rw-r--r-- 1 root root 30232576 Mar 12 13:47 slow.5307.12623
-rw-r--r-- 1 root root 831488 Mar 12 13:47 slow.5307.7902
-rw-r--r-- 1 root root 13615104 Mar 12 13:47 slow.5307.14380
-rw-r--r-- 1 root root 2695168 Mar 12 13:47 slow.5318.16154
-rw-r--r-- 1 root root 225280 Mar 12 13:47 slow.5318
-rw-r--r-- 1 root root 49152 Mar 12 13:47 slow.5307.9722
-rw-r--r-- 1 root root 1417216 Mar 12 13:47 slow.5307.16148
-rw-r--r-- 1 root root 0 Mar 12 13:47 slow.5307.16163
-rw-r--r-- 1 root root 64528384 Mar 12 13:47 slow.5307.9215
-rw-r--r-- 1 root root 20271104 Mar 12 13:47 slow.5307.13209
-rw-r--r-- 1 root root 389120 Mar 12 13:47 slow.5307.9358
-rw-r--r-- 1 root root 503808 Mar 12 13:47 slow.5307.8866
-rw-r--r-- 1 root root 270336 Mar 12 13:47 slow.5307.8164
-rw-r--r-- 1 root root 716800 Mar 12 13:47 slow.5307.8163
-rw-r--r-- 1 root root 1011712 Mar 12 13:47 slow.5307.8162
-rw-r--r-- 1 root root 501678 Mar 12 13:47 slow.5307.6376
-rw-r--r-- 1 root root 666947 Mar 12 13:47 slow.5321
-rw-r--r-- 1 root root 438096 Mar 12 13:47 slow.5320
-rw-r--r-- 1 root root 536576 Mar 12 13:47 slow.5322
-rw-r--r-- 1 root root 188416 Mar 12 13:47 slow.5307
-rw-r--r-- 1 root root 69632 Mar 12 13:47 slow.5307.6369
-rw-r--r-- 1 root root 536576 Mar 12 13:47 slow.5319
-rw-r--r-- 1 root root 786432 Mar 12 13:47 slow.5307.8132
-rw-r--r-- 1 root root 159744 Mar 12 13:47 slow.5307.8130
-rw-r--r-- 1 root root 27796 Mar 12 13:47 slow.5307.8884
-rw-r--r-- 1 root root 60106 Mar 12 13:51 slow.5307.6076
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file
descriptors?
Posted by Phil Endecott <sp...@chezphil.org>.
Jon Bendtsen wrote:
> So i used strace on apache and it turns out it tries to close
> 1024*1024 file descriptors more than once.
> Q) why brute force close all files?
> b) ask the kernel which files do I have open and then close only those
Here is some code to do this. I hereby place this code in the public domain.
#include <sys/types.h>
#include <dirent.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
void close_all_open_file_descriptors(int min_fd) {
// Find all open file descriptors >= min_fd using
// /proc/self/fd and close them.
// Use min_fd=3 to leave stdin/out/err open.
// Probably won't work if there are other threads
// opening or closing files where we're running.
DIR* proc_self_fd = opendir("/proc/self/fd");
if (!proc_self_fd) {
report_error("opendir(/proc/self/fd) failed");
}
// We must take care not to close the file descriptor
// underlying the DIR* that we're using:
int dir_fd = dirfd(proc_self_fd);
while (1) {
struct dirent* d = readdir(proc_self_fd);
if (!d) {
// Should mean end-of-directory, but could also be
// an error; how to distinguish?
break;
}
// We need to ignore the . and .. entries.
// The portable way to do this is to stat() the
// pathname. An alternative is to use the non-portable
// d_type field in struct_dirent. This is less portable
// - but this code is not very portable anyway, since it
// uses /proc.
#ifndef _DIRENT_HAVE_D_TYPE
#error This code needs a struct direct with d_type; see 'man readdir'
#endif
if (d->d_type == DT_DIR) {
continue;
}
int fd = strtol(d->d_name,NULL,10);
if (fd >= min_fd && fd != dir_fd) {
close(fd);
}
}
closedir(proc_self_fd);
}
I have no idea where this should live; perhaps somewhere in APR? It
obviously needs to be inside a #if linux block.
Phil.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file descriptors?
Posted by Marc Haisenko <ha...@comdasys.com>.
On Wednesday 12 March 2008, Phil Endecott wrote:
> Marc Haisenko wrote:
> > It's highly unlikely that SubVersion would have more
> > than say 128 file handles open
>
> It's quite possible that _apache_ has a very large number of files (and
> sockets) open.
>
> Phil.
Oh, yes, I haven't about that...
Marc
--
Marc Haisenko
Comdasys AG
Rüdesheimer Str. 7
80686 München
Germany
Tel.: +49 (0)89 548 433 321
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Re: Why does SVN + apache try to close non existing file
descriptors?
Posted by Phil Endecott <sp...@chezphil.org>.
Marc Haisenko wrote:
> It's highly unlikely that SubVersion would have more
> than say 128 file handles open
It's quite possible that _apache_ has a very large number of files (and
sockets) open.
Phil.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file descriptors?
Posted by Marc Haisenko <ha...@comdasys.com>.
On Wednesday 12 March 2008, Jon Bendtsen wrote:
> Q) why brute force close all files?
> a) ask the kernel to close all my open files
> b) ask the kernel which files do I have open and then close only those
> c) remember what files i have open and then close only those
This problem may sound easy but is quite tricky to solve: there's no portable
solution for a) and b), and c) is not reliable. For example, on Linux I think
there's no way for a), and b) would most likely have to rely on
analyzing /proc/*/fd. But this won't work on systems like FreeBSD or Mac OS X
which don't have /proc.
This is why several UNIX applications do the brute force way. But it may be a
good idea to limit it. It's highly unlikely that SubVersion would have more
than say 128 file handles open, with a safety factor let's make that 1024. If
by chance there really ARE more than 1024 file descriptors open you propably
have bigger problems than not having all file descriptors closed ;-)
Marc
--
Marc Haisenko
Comdasys AG
Rüdesheimer Str. 7
80686 München
Germany
Tel.: +49 (0)89 548 433 321
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file descriptors?
Posted by Erik Huelsmann <eh...@gmail.com>.
On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
> Hi
>
> I have a problem with SVN and/or apache which does not apply to
> svnserve.
> I use svn log to test this, but my users complain that any SVN command
> is slow.
>
> Using svnserve directly it takes less than a second to get an output
> of svn log.
> Using SVN through apache it takes about 28 minutes to get a log.
>
> So i used strace on apache and it turns out it tries to close
> 1024*1024 file descriptors
> more than once. Every time takes almost a minute. It starts 41 times,
> but for some reason
> it does not count to 1024*1024 every time. But it does count to
> 1024*1024 22 times,
> and some times only to some hundred thousand times. The error message
> is like this
> 13:47:17 close(227260) = -1 EBADF (Bad file descriptor)
> Most of the times the processes only have a few files open, so
> anything after 2 or 3
> gets an error message. This means i get a strace file with 1024 * 1024
> error lines.
>
> Q) why brute force close all files?
I think this is an apache or APR question: I can't think of any reason
why Subversion would be closing file handles. I *can* see reasons why
Apache would do that when starting a client to handle a request. Which
MPM did you configure for your Apache? mpm_worker? mpm_prefork?
> a) ask the kernel to close all my open files
> b) ask the kernel which files do I have open and then close only those
> c) remember what files i have open and then close only those
I think this would have been implemented by the Apache folks if it
could be done reliably: they're very keen on performance. Maybe you
could try the same question with them?
bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file descriptors?
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 12/03/2008, at 15.50, Marc Haisenko wrote:
> On Wednesday 12 March 2008, Jon Bendtsen wrote:
>> Q) why brute force close all files?
>> a) ask the kernel to close all my open files
>> b) ask the kernel which files do I have open and then close only
>> those
>> c) remember what files i have open and then close only those
>
> This problem may sound easy but is quite tricky to solve: there's no
> portable
> solution for a) and b), and c) is not reliable. For example, on
> Linux I think
> there's no way for a), and b) would most likely have to rely on
> analyzing /proc/*/fd. But this won't work on systems like FreeBSD or
> Mac OS X
> which don't have /proc.
Then you use an ifdef linux? just like Phil suggested.
> This is why several UNIX applications do the brute force way. But it
> may be a
> good idea to limit it. It's highly unlikely that SubVersion would
> have more
> than say 128 file handles open, with a safety factor let's make that
> 1024. If
> by chance there really ARE more than 1024 file descriptors open you
> propably
> have bigger problems than not having all file descriptors closed ;-)
I did limit it to 1024, and it appears to be much faster now, so users
doesnt complain.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Why does SVN + apache try to close non existing file descriptors?
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 12/03/2008, at 17.44, Erik Huelsmann wrote:
> On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
>> Hi
>>
>> I have a problem with SVN and/or apache which does not apply to
>> svnserve.
>> I use svn log to test this, but my users complain that any SVN
>> command
>> is slow.
>>
>> Using svnserve directly it takes less than a second to get an output
>> of svn log.
>> Using SVN through apache it takes about 28 minutes to get a log.
>>
>> So i used strace on apache and it turns out it tries to close
>> 1024*1024 file descriptors
>> more than once. Every time takes almost a minute. It starts 41 times,
>> but for some reason
>> it does not count to 1024*1024 every time. But it does count to
>> 1024*1024 22 times,
>> and some times only to some hundred thousand times. The error message
>> is like this
>> 13:47:17 close(227260) = -1 EBADF (Bad file descriptor)
>> Most of the times the processes only have a few files open, so
>> anything after 2 or 3
>> gets an error message. This means i get a strace file with 1024 *
>> 1024
>> error lines.
>>
>> Q) why brute force close all files?
>
> I think this is an apache or APR question: I can't think of any reason
> why Subversion would be closing file handles. I *can* see reasons why
> Apache would do that when starting a client to handle a request. Which
> MPM did you configure for your Apache? mpm_worker? mpm_prefork?
ii apache2-mpm-prefork 2.0.54-5sarge2
traditional model for Apache2
Will mpm_worker be faster?
>
>> a) ask the kernel to close all my open files
>> b) ask the kernel which files do I have open and then close
>> only those
>> c) remember what files i have open and then close only those
>
> I think this would have been implemented by the Apache folks if it
> could be done reliably: they're very keen on performance. Maybe you
> could try the same question with them?
Yes, i can try that.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org