You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by David Weintraub <qa...@gmail.com> on 2009/04/01 16:35:29 UTC
KeepAlive on Apache and Subversion
What is the official Subversion setting for KeepAlive? Should this be
on or off? How does it affect Subversion. We use Apache's httpd as a
server for our Subversion repository and this HTTP server is not used
for anything but Subversion.
--
David Weintraub
qazwart@gmail.com
All attachments have been scanned for viruses and have been found to
be virus free. So, just go ahead and open them. Go ahead... Just
click on it... Trust me.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1508726
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: 1.6.2 poor commit performance with large data
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 08, 2009 at 01:57:32PM -0500, Justin Johnson wrote:
> For anyone who is interested, I tried adding the source code for the linux
> 2.6.29.4 kernel to a working copy and committing. From Windows (1.6.2
> command line) the commit slowed down rather quickly and consumed all my
> memory. It took so long (5 hours +) I eventually just killed the process.
>
> I then tried the same thing from a Solaris (1.6.2) working copy. It went
> much faster, but still slowed down after a while and eventually failed
> with the following error.
>
> Out of memory - terminating application.
> Abort
>
> So the problem is on Windows and UNIX, but I'm still waiting to hear if
> the leak affects the 1.6.2 server at all, and also when the 1.6.3 release
> will be out.
Thanks!
1.6.3 should be out soon.
If you can't wait, grab http://svn.collab.net/repos/svn/branches/1.6.x
which already contains a fix which should lessen the impact of the
overall memory problem during commit. But note that adding, rather than
importing, the Linux source tree might not work until issue #1964 has
really been fixed.
Stefan
RE: 1.6.2 poor commit performance with large data
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> -----Original Message-----
> From: B Smith-Mannschott [mailto:bsmith.occs@gmail.com]
> Sent: Friday, May 29, 2009 11:27 AM
> To: Bolstridge, Andrew
> Cc: users@subversion.tigris.org
> Subject: Re: 1.6.2 poor commit performance with large data
>
[snip]
> This sounds like the same problem as described in [this thread][1]
> from a few days ago. Said thread links to [Issue 1964][2] which looks
> like the same problem.
>
> [1]:
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessage
> Id=2352720
> [2]: http://subversion.tigris.org/issues/show_bug.cgi?id=1964
>
> So far, evidence suggests that this issue crops up only under Windows.
> I haven't seen anything suggesting *why* this is the case, so any
> information you can provide is likely to be helpful.
>
I've had a look at this running again today. It seems to be throttled in the System process (not any of the sub-processes, so it must be a driver). On my quad-core box System.exe hits 25% CPU and stays there.
There is a lot of page faults (after transferring 73Mb, I've got 74 million page faults with a delta of roughly 1000 per second).
Memory usage steadily (but very slowly) increases once the transfer has started to stall, currently showing 169Mb/164Mb in taskmanager.
and then.... I had a brainwave. I turned off the virus checker and suddenly things went back to normal. (I use Trend which isn't quite so bad). The memory usage is still increasing at a rate of 1Mb per second but the page fault delta for System process has dropped to 0.
In the end, transferring 9500 mostly cpp source-code files (187Mb) used peak memory of 1.5Gb RAM.
If anyone wants to check the 'leak', I made a heap comparison showing the allocations. I did this using TortoiseSVN (as that has some symbols), unfortunately it doesn't have enough symbols for the svn libraries or it'd be more useful. If anyone has built SVN for Windows, and will create some .pdb symbols, then I'd be happy to re-run my commits and see if I can find out more about this issue.
It appears that the call to libapr_tsvn's apr_allocator_max_free_set is doing the damage, after transferring 130Mb (roughly 7000 files), it has 'leaked' 749Mb. I can’t see what has been calling it without some effort (and symbols) but maybe someone more knowledgeable of SVN source can offer some suggestions. It appears to mostly allocate 8k at a time, but occasionally allocates 16k (if that helps any).
Be aware I've cross-posted this to the TortoiseSVN list too.
Andy
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2358836
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: 1.6.2 poor commit performance with large data
Posted by "B. Smith-Mannschott" <bs...@gmail.com>.
On Fri, May 29, 2009 at 11:57, Bolstridge, Andrew
<an...@intergraph.com> wrote:
> I'm trying to commit a large directory (about 1Gb of data in 20,000
> files in 6000 directories)(though those figures also include ignored and
> not-added files, so I'd guess knock 10-20% off that).
>
> This is a new set of source code from a vendor, roughly identical to one
> I did a few months ago using svn 1.5.5.
>
> Previously, the directory was added and committed without too much of a
> problem, it took a little while, but I consistently got 2kps on my
> checkins all through the process (more when I turned the virus checker
> off on the server!).
>
> Now I'm trying to add the same using svn 1.6.2, and it's not too happy.
> Using TortoiseSVN I see it is very quick at first, but rapidly slows
> down after 30 MB has been transferred. After 40Mb, the transfer rate has
> slowed considerably. The CPU usage is right down (from 17% to 0.5%), and
> the memory usage increases steadily. When I killed it (using Tortoise)
> is had transferred 43Mb, but had used 183Mb RAM and had also used 183MB
> pagefile.
>
> I've tried it using the svn command line, but that shows the same
> behaviour (except I can't see how much transferred, just a command
> window full of dots).
>
>
> Short of committing each subdir individually, can anyone say why the
> performance has dropped so dramatically for large datasets? This was
> fine with 1.5.5, otherwise I'd have just assumed I was trying to push it
> too hard.
>
>
> I'm happy to test with debug version or provide a crashdump if anyone
> wants to provide one, but bear in mind, I'm running on Windows (XPsp3,
> dual CPU box w 3Gb RAM)
>
This sounds like the same problem as described in [this thread][1]
from a few days ago. Said thread links to [Issue 1964][2] which looks
like the same problem.
[1]: http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2352720
[2]: http://subversion.tigris.org/issues/show_bug.cgi?id=1964
So far, evidence suggests that this issue crops up only under Windows.
I haven't seen anything suggesting *why* this is the case, so any
information you can provide is likely to be helpful.
// Ben
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2356710
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
1.6.2 poor commit performance with large data
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
I'm trying to commit a large directory (about 1Gb of data in 20,000
files in 6000 directories)(though those figures also include ignored and
not-added files, so I'd guess knock 10-20% off that).
This is a new set of source code from a vendor, roughly identical to one
I did a few months ago using svn 1.5.5.
Previously, the directory was added and committed without too much of a
problem, it took a little while, but I consistently got 2kps on my
checkins all through the process (more when I turned the virus checker
off on the server!).
Now I'm trying to add the same using svn 1.6.2, and it's not too happy.
Using TortoiseSVN I see it is very quick at first, but rapidly slows
down after 30 MB has been transferred. After 40Mb, the transfer rate has
slowed considerably. The CPU usage is right down (from 17% to 0.5%), and
the memory usage increases steadily. When I killed it (using Tortoise)
is had transferred 43Mb, but had used 183Mb RAM and had also used 183MB
pagefile.
I've tried it using the svn command line, but that shows the same
behaviour (except I can't see how much transferred, just a command
window full of dots).
Short of committing each subdir individually, can anyone say why the
performance has dropped so dramatically for large datasets? This was
fine with 1.5.5, otherwise I'd have just assumed I was trying to push it
too hard.
I'm happy to test with debug version or provide a crashdump if anyone
wants to provide one, but bear in mind, I'm running on Windows (XPsp3,
dual CPU box w 3Gb RAM)
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2356708
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: KeepAlive on Apache and Subversion
Posted by Jeremy Whitlock <jc...@gmail.com>.
> What is the official Subversion setting for KeepAlive? Should this be
> on or off? How does it affect Subversion. We use Apache's httpd as a
> server for our Subversion repository and this HTTP server is not used
> for anything but Subversion.
The "KeepAlive" in Apache does have an impact on Subversion. If you
have KeepAlive off, you end up with multiple Apache connections being
used to serve your single Subversion request. That means that for
each connection created, you reauthenticate. If you have a custom
authentication/authorization mechanism in place, like a custom Apache
module or using mod_[perl|python|etc.] to handle your Subversion
authn/authz, you end up reauthenticating unnecessarily because your
cache scope in Apache for modules is usually in the connection pool.
So...you should see a jump in performance with KeepAlive on. This
does not mean the user is prompted to reauthenticate but the logic
used to authenticate a user can be reran if your Subversion operation
uses multiple connections. In a situation where you use the network
to authenticate, like ldap/database/web services/etc., this can be a
performance hit.
--
Take care,
Jeremy Whitlock
http://www.thoughtspark.org
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1509570
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].