You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Jon Bendtsen <jb...@laerdal.dk> on 2008/03/10 16:14:06 UTC
svn log using apache is SLOW
Hi
My apache 2.0 based subversion 1.4 is quite slow at doing svn log
commands.
Using strace i find stuff like this
close(0) = 0
close(1) = 0
close(14) = 0
dup2(13, 0) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
close(2) = 0
close(3) = 0
close(4) = 0
close(5) = 0
close(6) = 0
close(7) = 0
close(8) = 0
close(9) = 0
close(10) = 0
close(11) = 0
close(12) = 0
close(14) = -1 EBADF (Bad file descriptor)
close(15) = -1 EBADF (Bad file descriptor)
close(16) = -1 EBADF (Bad file descriptor)
close(17) = -1 EBADF (Bad file descriptor)
close(18) = -1 EBADF (Bad file descriptor)
close(19) = -1 EBADF (Bad file descriptor)
close(20) = -1 EBADF (Bad file descriptor)
close(21) = -1 EBADF (Bad file descriptor)
close(22) = -1 EBADF (Bad file descriptor)
close(23) = -1 EBADF (Bad file descriptor)
close(24) = -1 EBADF (Bad file descriptor)
...
close(1048573) = -1 EBADF (Bad file
descriptor)
close(1048574) = -1 EBADF (Bad file
descriptor)
close(1048575) = -1 EBADF (Bad file
descriptor)
close(1048576) = -1 EBADF (Bad file
descriptor)
And that takes alot of time.
But why are it trying to close 1024 * 1024 file descriptors?
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Erik Huelsmann <eh...@gmail.com>.
On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk> wrote:
> Hi
>
> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
> commands.
The default answer here is: yes, if you enable path based
authorization, svn log can be very slow over HTTP. There's information
about disabling path based authz here:
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.httpd.authz.pathauthzoff.
HTH,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Phil Endecott <sp...@chezphil.org>.
Jon Bendtsen wrote:
> Hi
>
> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
> commands.
Mine too, though probably not as slow as yours (see below).
> Using strace i find stuff like this
Is this strace from the client or the server? I guess the server, right?
And what OS is it? I'm guessing Linux.
> close(0) = 0
> close(1) = 0
> close(14) = 0
> dup2(13, 0) = 0
> getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
> close(2) = 0
[snip]
> close(1048576) = -1 EBADF (Bad file
> descriptor)
>
> And that takes alot of time.
>
> But why are it trying to close 1024 * 1024 file descriptors?
It's trying to close all open file descriptors, and it has chosen to do
this by attempting to close every possible file descriptor.
Some OSes have specific functions to do this, e.g. fcntl(F_CLOSEM).
Linux doesn't have a feature like that as far as I'm aware. Closing
all possible file descriptors (i.e. up to RLIMIT_NOFILE) is a common
way to do it. I think that the most effective method on Linux is to
examine /proc/self/fd.
But the interesting thing is that on your system, the maximum number of
file descriptors is 1 million. On mine it's much lower, so trying to
close all files in this way will is "fast enough that I don't
complain", if not actually "fast". Maybe you have deliberately
increased the maximum number of open files as some sort of Apache
tuning, or something?
As I understand it - and I'm not an expert - the maximum number of open
files is set by /proc/sys/fs/file-max (aka sysctl fs.file-max); on my
systems I see values of 48 000 and 100 000. So you could try reducing
that. I'm not sure where the default comes from, but it may be a
heuristic based on the amount of RAM that you have.
Disclaimer: I know nothing about the internals of Subversion. I'm
replying only because I've had to solve the problem of closing all
files in my own code. If someone else answers differently, they're
probably right.
Phil.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Erik Huelsmann <eh...@gmail.com>.
On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
> On 10/03/2008, at 19.29, Erik Huelsmann wrote:
>
> > On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk>
> > wrote:
> >> Hi
> >>
> >> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
> >> commands.
> >
> > The default answer here is: yes, if you enable path based
> > authorization, svn log can be very slow over HTTP. There's information
> > about disabling path based authz here:
> > http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.httpd.authz.pathauthzoff
> > .
>
> I know it is the standard answer, but i do not think it applies in my
> case.
>
> I also use svnserve, and svnserver is MANY times faster, and svnserve
> also uses
> authz-db = dav_svn.authz
But the mechanism used by mod_dav_svn to do its access authorization
differs from svnserve, so until you have actually tested this, I'm
unable to eliminated this reason.
> I believe the problem is that svn or apache wants to close
> 34897573403487348
> file descriptors.
Well, I can hardly imagine this would affect log but no other
commands. That means that if *only* 'log' is slow, then I don't think
it's this problem.
bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Re: svn log using apache is SLOW
Posted by Andy Levy <an...@gmail.com>.
On Wed, Mar 12, 2008 at 9:39 AM, Gleason, Todd <tg...@impac.com> wrote:
> Is there a way to enable path-based authorization only when committing?
> I can see allowing any user with access to a repository to be able to
> see the entire thing, but wanting to keep most users from writing to
> tags folders and to the main branch areas of previous releases that have
> been locked down. Even if this was to be enforced using a commit hook,
> it would probably be acceptable if it boosted performance significantly.
I think svnperms.py (implemented as a pre-commit hook) is what you're
looking for.
> -----Original Message-----
> From: Jon Bendtsen [mailto:jbendtsen@laerdal.dk]
> Sent: Wednesday, March 12, 2008 4:58 AM
> To: Erik Huelsmann
> Cc: users@subversion.tigris.org
> Subject: Re: svn log using apache is SLOW
>
> On 12/03/2008, at 11.48, Erik Huelsmann wrote:
>
> > On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
> >> On 10/03/2008, at 19.29, Erik Huelsmann wrote:
> >>
> >>> On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk>
> >>> wrote:
> >>>> Hi
> >>>>
> >>>> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
> >>>> commands.
> >>>
> >>> The default answer here is: yes, if you enable path based
> >>> authorization, svn log can be very slow over HTTP. There's
> >>> information
> >>> about disabling path based authz here:
> >>>
> http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.ht
> tpd.authz.pathauthzoff
> >>> .
> >>
> >> I know it is the standard answer, but i do not think it applies in my
> >> case.
> >>
> >> I also use svnserve, and svnserver is MANY times faster, and svnserve
> >> also uses
> >> authz-db = dav_svn.authz
> >
> > But the mechanism used by mod_dav_svn to do its access authorization
> > differs from svnserve, so until you have actually tested this, I'm
> > unable to eliminated this reason.
>
> I have tried both using apache and svnserve.
>
>
> >
> >> I believe the problem is that svn or apache wants to close
> >> 34897573403487348
> >> file descriptors.
> >
> > Well, I can hardly imagine this would affect log but no other
> > commands. That means that if *only* 'log' is slow, then I don't think
> > it's this problem.
>
> I am sorry i was saying only log was slow, because every svn command
> using
> apache is much slower than using svnserve. I do my testing using svn
> log because
> that does not change the repository.
>
>
>
> JonB
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Re: svn log using apache is SLOW
Posted by "Gleason, Todd" <tg...@impac.com>.
Is there a way to enable path-based authorization only when committing?
I can see allowing any user with access to a repository to be able to
see the entire thing, but wanting to keep most users from writing to
tags folders and to the main branch areas of previous releases that have
been locked down. Even if this was to be enforced using a commit hook,
it would probably be acceptable if it boosted performance significantly.
-----Original Message-----
From: Jon Bendtsen [mailto:jbendtsen@laerdal.dk]
Sent: Wednesday, March 12, 2008 4:58 AM
To: Erik Huelsmann
Cc: users@subversion.tigris.org
Subject: Re: svn log using apache is SLOW
On 12/03/2008, at 11.48, Erik Huelsmann wrote:
> On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
>> On 10/03/2008, at 19.29, Erik Huelsmann wrote:
>>
>>> On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk>
>>> wrote:
>>>> Hi
>>>>
>>>> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
>>>> commands.
>>>
>>> The default answer here is: yes, if you enable path based
>>> authorization, svn log can be very slow over HTTP. There's
>>> information
>>> about disabling path based authz here:
>>>
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.ht
tpd.authz.pathauthzoff
>>> .
>>
>> I know it is the standard answer, but i do not think it applies in my
>> case.
>>
>> I also use svnserve, and svnserver is MANY times faster, and svnserve
>> also uses
>> authz-db = dav_svn.authz
>
> But the mechanism used by mod_dav_svn to do its access authorization
> differs from svnserve, so until you have actually tested this, I'm
> unable to eliminated this reason.
I have tried both using apache and svnserve.
>
>> I believe the problem is that svn or apache wants to close
>> 34897573403487348
>> file descriptors.
>
> Well, I can hardly imagine this would affect log but no other
> commands. That means that if *only* 'log' is slow, then I don't think
> it's this problem.
I am sorry i was saying only log was slow, because every svn command
using
apache is much slower than using svnserve. I do my testing using svn
log because
that does not change the repository.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svn log using apache is SLOW
Posted by "Gleason, Todd" <tg...@impac.com>.
Sorry, I meant using Apache. We'd want standard authorized users to be
able to read the whole repository (it isn't public but if you can get
in, you can read it all), but only a select few to be able to write to
tags and certain branches.
-----Original Message-----
From: Jon Bendtsen [mailto:jbendtsen@laerdal.dk]
Sent: Wednesday, March 12, 2008 7:49 AM
To: Gleason, Todd
Cc: List Subversion
Subject: Re: svn log using apache is SLOW
On 12/03/2008, at 14.39, Gleason, Todd wrote:
> Is there a way to enable path-based authorization only when
> committing?
> I can see allowing any user with access to a repository to be able to
> see the entire thing, but wanting to keep most users from writing to
> tags folders and to the main branch areas of previous releases that
> have
> been locked down. Even if this was to be enforced using a commit
> hook,
> it would probably be acceptable if it boosted performance
> significantly.
Using svnserve there might. My svnserve.conf includes these lines:
anon-access = none
auth-access = write
So i think you can set anon access to read only, and then allow people
to read
or do log? commands or??
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 12/03/2008, at 14.51, Gleason, Todd wrote:
> Sorry, I meant using Apache. We'd want standard authorized users to
> be
> able to read the whole repository (it isn't public but if you can get
> in, you can read it all), but only a select few to be able to write to
> tags and certain branches.
You might be able to do that as well.
This is from my
/etc/apache2/mods-available/dav_svn.conf
# The following three lines allow anonymous read, but make
# committers authenticate themselves.
# <LimitExcept GET PROPFIND OPTIONS REPORT>
# Require valid-user
# </LimitExcept>
So i think you can do what you want.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 12/03/2008, at 14.39, Gleason, Todd wrote:
> Is there a way to enable path-based authorization only when
> committing?
> I can see allowing any user with access to a repository to be able to
> see the entire thing, but wanting to keep most users from writing to
> tags folders and to the main branch areas of previous releases that
> have
> been locked down. Even if this was to be enforced using a commit
> hook,
> it would probably be acceptable if it boosted performance
> significantly.
Using svnserve there might. My svnserve.conf includes these lines:
anon-access = none
auth-access = write
So i think you can set anon access to read only, and then allow people
to read
or do log? commands or??
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 12/03/2008, at 11.48, Erik Huelsmann wrote:
> On 3/12/08, Jon Bendtsen <jb...@laerdal.dk> wrote:
>> On 10/03/2008, at 19.29, Erik Huelsmann wrote:
>>
>>> On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk>
>>> wrote:
>>>> Hi
>>>>
>>>> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
>>>> commands.
>>>
>>> The default answer here is: yes, if you enable path based
>>> authorization, svn log can be very slow over HTTP. There's
>>> information
>>> about disabling path based authz here:
>>> http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.httpd.authz.pathauthzoff
>>> .
>>
>> I know it is the standard answer, but i do not think it applies in my
>> case.
>>
>> I also use svnserve, and svnserver is MANY times faster, and svnserve
>> also uses
>> authz-db = dav_svn.authz
>
> But the mechanism used by mod_dav_svn to do its access authorization
> differs from svnserve, so until you have actually tested this, I'm
> unable to eliminated this reason.
I have tried both using apache and svnserve.
>
>> I believe the problem is that svn or apache wants to close
>> 34897573403487348
>> file descriptors.
>
> Well, I can hardly imagine this would affect log but no other
> commands. That means that if *only* 'log' is slow, then I don't think
> it's this problem.
I am sorry i was saying only log was slow, because every svn command
using
apache is much slower than using svnserve. I do my testing using svn
log because
that does not change the repository.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 10/03/2008, at 19.29, Erik Huelsmann wrote:
> On Mon, Mar 10, 2008 at 5:14 PM, Jon Bendtsen <jb...@laerdal.dk>
> wrote:
>> Hi
>>
>> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
>> commands.
>
> The default answer here is: yes, if you enable path based
> authorization, svn log can be very slow over HTTP. There's information
> about disabling path based authz here:
> http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.serverconfig.httpd.authz.pathauthzoff
> .
I know it is the standard answer, but i do not think it applies in my
case.
I also use svnserve, and svnserver is MANY times faster, and svnserve
also uses
authz-db = dav_svn.authz
I believe the problem is that svn or apache wants to close
34897573403487348
file descriptors.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Erik Huelsmann <eh...@gmail.com>.
On Wed, Mar 12, 2008 at 11:48 AM, Jon Bendtsen <jb...@laerdal.dk> wrote:
> On 10/03/2008, at 18.37, Phil Endecott wrote:
>
>
> > Jon Bendtsen wrote:
> >> Hi
> >>
> >> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
> >> commands.
> >
>
> > Mine too, though probably not as slow as yours (see below).
>
> svnserve is fast, very fast.
Yes. We know svnserve is much faster than Apache. Much of that has to
do with the protocol they use: svnserve uses a statefull protocol with
a single connection, whereas Apache uses a stateless protocol which
allows only one request per connection.
In the past there have been speed comparisons between the two. Maybe
you can search the archives to give you some idea of what is expected
to be normal?
bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn log using apache is SLOW
Posted by Jon Bendtsen <jb...@laerdal.dk>.
On 10/03/2008, at 18.37, Phil Endecott wrote:
> Jon Bendtsen wrote:
>> Hi
>>
>> My apache 2.0 based subversion 1.4 is quite slow at doing svn log
>> commands.
>
> Mine too, though probably not as slow as yours (see below).
svnserve is fast, very fast.
>> Using strace i find stuff like this
>
> Is this strace from the client or the server? I guess the server,
> right?
it is from the server
> And what OS is it? I'm guessing Linux.
it is linux.
>> close(0) = 0
>> close(1) = 0
>> close(14) = 0
>> dup2(13, 0) = 0
>> getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024})
>> = 0
>> close(2) = 0
> [snip]
>> close(1048576) = -1 EBADF (Bad file
>> descriptor)
>>
>> And that takes alot of time.
>>
>> But why are it trying to close 1024 * 1024 file descriptors?
>
> It's trying to close all open file descriptors, and it has chosen to
> do this by attempting to close every possible file descriptor.
>
> Some OSes have specific functions to do this, e.g. fcntl(F_CLOSEM).
> Linux doesn't have a feature like that as far as I'm aware. Closing
> all possible file descriptors (i.e. up to RLIMIT_NOFILE) is a common
> way to do it. I think that the most effective method on Linux is to
> examine /proc/self/fd.
>
> But the interesting thing is that on your system, the maximum number
> of file descriptors is 1 million. On mine it's much lower, so
> trying to close all files in this way will is "fast enough that I
> don't complain", if not actually "fast". Maybe you have
> deliberately increased the maximum number of open files as some sort
> of Apache tuning, or something?
I have not done so deliberately. ulimit (now) says unlimited. I have
tried setting it to 1024 when starting apache.
ulimit -S -n 1024
but it doesnt appear to be working. Maybe i should also set it
somewhere else? But i dont know where, and i've
only started looking. Hints would be appriciated.
> As I understand it - and I'm not an expert - the maximum number of
> open files is set by /proc/sys/fs/file-max (aka sysctl fs.file-max);
> on my systems I see values of 48 000 and 100 000. So you could try
> reducing that. I'm not sure where the default comes from, but it
> may be a heuristic based on the amount of RAM that you have.
mine is 408964
> Disclaimer: I know nothing about the internals of Subversion. I'm
> replying only because I've had to solve the problem of closing all
> files in my own code. If someone else answers differently, they're
> probably right.
Thanks for replying, i believe your answer more than the other answer.
JonB
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org