You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ron Bieber <ro...@bieberlabs.com> on 2004/03/18 12:33:48 UTC
Drastic performance difference with subversion client on Solaris 9
Good morning. I've been debugging a performance "problem" that we have
been having for about a week now and am running out of options. I
would like to throw the scenario we are experiencing up here and see if
we can get some answers.
We are currently evaluating Subversion to replace our CVS installation.
Currently we have a CVS repository with about 6500 files in it that we
have exported from CVS and imported into Subversion (cvs2svn took on the
order of 30 days to convert the repository, so we're not doing that).
The server is a 1.0.1 server running Solaris 9. It is a 2 CPU / 4 gig
box. Subversion is running under Apache 2.0.48 with mod_deflate
enabled.
We have run checkouts on a number of clients and get differing results,
the most drastic on Solaris 9. Checking out one of the modules that
consists of 93 .java and .c source files, we get the following results:
Redhat 9 box as client running 1.0.1: Checkout time: 6 seconds
Windows box as client running 1.0.0: Checkout time: 14 seconds
Solaris Box 1 (Solaris 2.6) running 1.0.1: Checkout time: 57 seconds
Solaris Box 2 (Solaris 9) running 1.0.1: Checkout time: 50 seconds
Solaris Box 3 (same as repository - Solaris 9) Checkout time: 50 seconds
Solaris Box 4 (Solaris 8 - Subversion 1.0.0) Checkout time: 53 seconds
My question is this: Is anyone else having these drastic differences
when running the Subversion client on Solaris 9? We've had our admins
check the networking between the boxes and I'm told this is not the
problem. This ties off with the fact that checking out the same 93
file module on the repository box itself takes the same amount of time
(using localhost) as the other Solaris boxes on the network.
Here's a summary of the research that I've done so far:
* network between boxes seems to be fine.
* same server is used in all cases, so it isn't a server bottleneck.
* while checkout on the Solaris 9 box is 50 seconds, export of head for
the same module is about 6 (comparable with Redhat 9 checkout time)
* Given that the module is about 1% of the total repository size, the
current Solaris 9 timings are unacceptable for checking out the whole
repository for actual work.
* On the Solaris 8 client, process seems to be sleeping in iowait > 86%
* On Solaris 9 boxes, client process seems to sleep while iowait time
hovers around 23%
* On all solaris cases, system and user time are all one second or
under, which indicates a lot of sleeping on the part of the process.
I would appreciate any light that any Solaris Guru can shed on this.
I'm at my wits end at this point.
Thanks in advance,
-- Ron
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Thursday, March 18, 2004 5:00 PM -0600 Ron Bieber <ro...@bieberlabs.com>
wrote:
> here are the timings for cvs2svn checkout:
>
> Checked out revision 838.
>
> real 2m7.190s
> user 0m1.590s
> sys 0m3.360s
At this point, you need to give detailed description of your environment
because your results aren't close to anything else.
What OS patches do you have applied - you do have all of the recommended
Solaris patches applied, right? Are you sure you aren't network-bound - could
you have full/half-duplex problems?
What box did you do compilation on: your Solaris 2.6 box? What happens if you
compile SVN on your Solaris 9 box? While the binaries won't be backwards
compatible, APR does gets to take advantage of some features if it is compiled
on newer Solaris versions.
If you are current wrt patches and your network isn't the bottleneck and
changing where it is compiled, then let's go a level deeper: what speed is
your processor? What compiler: version, optimizations, etc? Kernel revision,
file system mount options, local/networked drives, etc, etc.
On an old Ultra-5 on Solaris 8, I get:
Checkout to NFS'd home dir:
svn co http://svn.collab.net/repos/cvs2svn/trunk cvs2svn 2.53s user 4.77s
system 18% cpu 38.998 total
Checkout to local /tmp:
svn co http://svn.collab.net/repos/cvs2svn/trunk cvs2svn 2.37s user 0.97s
system 22% cpu 14.872 total
% uname -a
SunOS ... 5.8 Generic_108528-27 sun4u sparc SUNW,Ultra-5_10
% psrinfo -v
Status of processor 0 as of: 03/18/04 20:35:15
Processor has been on-line since 02/23/04 08:03:35.
The sparcv9 processor operates at 400 MHz,
and has a sparcv9 floating point processor.
% prtconf | grep Memory
Memory size: 128 Megabytes
% cc -V
cc: Sun WorkShop 6 update 2 C 5.3 2001/05/15
(I use debugging builds with no optimizations.)
% ping -s svn.collab.net
PING svn.collab.net: 56 data bytes
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=0. time=69. ms
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=1. time=121. ms
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=2. time=71. ms
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=3. time=70. ms
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=4. time=69. ms
64 bytes from morbius.ch.collab.net (216.127.237.133): icmp_seq=5. time=67. ms
This box is not idle as it has some nice'd jobs running. So, I'd expect this
to be about the worse case performance-wise *if* the network isn't your
bottleneck as these boxes are on a pretty fat pipe. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris
Posted by Ron Bieber <ro...@bieberlabs.com>.
here are the timings for cvs2svn checkout:
Checked out revision 838.
real 2m7.190s
user 0m1.590s
sys 0m3.360s
On Thu, 18 Mar 2004, Justin Erenkrantz wrote:
> --On Thursday, March 18, 2004 6:33 AM -0600 Ron Bieber <ro...@bieberlabs.com>
> wrote:
>
> > We have run checkouts on a number of clients and get differing results,
> > the most drastic on Solaris 9. Checking out one of the modules that
> > consists of 93 .java and .c source files, we get the following results:
>
> I certainly don't see such a drastic performance difference. I just did a
> checkout of cvs2svn's trunk on Solaris 9 and Darwin.
>
> Solaris 9:
> svn co http://svn.collab.net/repos/cvs2svn/trunk 0.65s user 0.93s system 6%
> cpu 25.002 total
>
> Mac OS X 10.3.3:
> svn co http://svn.collab.net/repos/cvs2svn/trunk 2.07s user 3.48s system 27%
> cpu 20.222 total
>
> That's within margin of error for me and hinting that it's probably a network
> bottleneck on my end.
>
> How long does it take to check out cvs2svn for you? It has a relatively small
> working copy, so the WC size should be in the ballpark of yours. -- justin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris 9
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Thursday, March 18, 2004 6:33 AM -0600 Ron Bieber <ro...@bieberlabs.com>
wrote:
> We have run checkouts on a number of clients and get differing results,
> the most drastic on Solaris 9. Checking out one of the modules that
> consists of 93 .java and .c source files, we get the following results:
I certainly don't see such a drastic performance difference. I just did a
checkout of cvs2svn's trunk on Solaris 9 and Darwin.
Solaris 9:
svn co http://svn.collab.net/repos/cvs2svn/trunk 0.65s user 0.93s system 6%
cpu 25.002 total
Mac OS X 10.3.3:
svn co http://svn.collab.net/repos/cvs2svn/trunk 2.07s user 3.48s system 27%
cpu 20.222 total
That's within margin of error for me and hinting that it's probably a network
bottleneck on my end.
How long does it take to check out cvs2svn for you? It has a relatively small
working copy, so the WC size should be in the ballpark of yours. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris 9
Posted by es...@logic.net.
On Tue, Mar 30, 2004 at 06:04:18AM -0600, Ron Bieber wrote:
> This was proven out further by turning on delayed I/O using the fastfs
> tool. When we do this, times on the client go back down to the times
> we see on other operating systems.
I'd like to add that you may want to add "logging" to the mount options
on any filesystems you've done this to. While it can't completely save
your hide in the event of failure, you'll at least (hopefully) minimize
the damage by only hosing up a journal (and thus losing some data), rather
than getting back a damaged or unrepairable filesystem (thus losing ALL
data ;-).
--
Edward S. Marshall <es...@logic.net>
http://esm.logic.net/
Felix qui potuit rerum cognoscere causas.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Greg Hudson <gh...@MIT.EDU>.
On Tue, 2004-03-30 at 07:04, Ron Bieber wrote:
> As Linux file systems default to nosynch and I've never had a problem
> with boxes losing data in power failures here at home, I don't see a big
> problem with this as long as the box isn't running an oracle production
> database or something like that. Given the fact that the computer room
> at work has interruptible power and backup generators anyway, I see even
> less of a problem.
The situation is somewhat more complicated than you give it credit for
in your first sentence.
The Linux ext2 filesystem behaves as you say, with completely asynch
writes and no particular attention to ensuring failure atomicity of any
kind. But it does have a pretty sophisticated fsck designed to get a
filesystem back into working order in the face of failures. (It can't
protect against certain kinds of problems, like data blocks containing
data from the wrong files, but at least it won't throw up its hands in
disgust and tell you to newfs.)
Modern Linux filesystems (reiser, xfs, ext3) have asynchronous file
creation but use techniques (careful ordering or journals) to make sure
that the on-disk filesystem is consistent at all times. As a result,
they tend to be a little slower than ext2, but most safer. Most people
who use Linux these days use one of these filesystems.
Solaris with asynch writes has neither the advantage of a sophisticated
fsck (as far as I know) nor the advantage of ordered writes or
journaling. Although you stand a pretty good chance of being okay in
the face of an unclean shutdown, it's not very safe at all.
So, feel free to rely on backups and uninterruptable power; those are
fine things. But I don't think you can rely on the experience of Linux
users.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Greg Hudson <gh...@MIT.EDU>.
On Tue, 2004-03-30 at 07:04, Ron Bieber wrote:
> As Linux file systems default to nosynch and I've never had a problem
> with boxes losing data in power failures here at home, I don't see a big
> problem with this as long as the box isn't running an oracle production
> database or something like that. Given the fact that the computer room
> at work has interruptible power and backup generators anyway, I see even
> less of a problem.
The situation is somewhat more complicated than you give it credit for
in your first sentence.
The Linux ext2 filesystem behaves as you say, with completely asynch
writes and no particular attention to ensuring failure atomicity of any
kind. But it does have a pretty sophisticated fsck designed to get a
filesystem back into working order in the face of failures. (It can't
protect against certain kinds of problems, like data blocks containing
data from the wrong files, but at least it won't throw up its hands in
disgust and tell you to newfs.)
Modern Linux filesystems (reiser, xfs, ext3) have asynchronous file
creation but use techniques (careful ordering or journals) to make sure
that the on-disk filesystem is consistent at all times. As a result,
they tend to be a little slower than ext2, but most safer. Most people
who use Linux these days use one of these filesystems.
Solaris with asynch writes has neither the advantage of a sophisticated
fsck (as far as I know) nor the advantage of ordered writes or
journaling. Although you stand a pretty good chance of being okay in
the face of an unclean shutdown, it's not very safe at all.
So, feel free to rely on backups and uninterruptable power; those are
fine things. But I don't think you can rely on the experience of Linux
users.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris 9
Posted by es...@logic.net.
On Tue, Mar 30, 2004 at 06:04:18AM -0600, Ron Bieber wrote:
> This was proven out further by turning on delayed I/O using the fastfs
> tool. When we do this, times on the client go back down to the times
> we see on other operating systems.
I'd like to add that you may want to add "logging" to the mount options
on any filesystems you've done this to. While it can't completely save
your hide in the event of failure, you'll at least (hopefully) minimize
the damage by only hosing up a journal (and thus losing some data), rather
than getting back a damaged or unrepairable filesystem (thus losing ALL
data ;-).
--
Edward S. Marshall <es...@logic.net>
http://esm.logic.net/
Felix qui potuit rerum cognoscere causas.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Ron Bieber <ro...@bieberlabs.com>.
Folks,
I wanted to send a follow up message to my original email on using
Solaris 9 with Solaris and the drastic performance penalty I was seeing
on the Solaris/SPARC platform.
The performance problem only occurred when using the client. After
performing many tests, including turning logging on, mounting with
noatime, and others we were unable to get our subversion client running
at an acceptable level to use it in a production environment without
causing a lot of wait time for our developers. We consistently saw an
76-85% decrease in performance when running the client on Solaris vs.
Windows or Linux boxes.
The performance hit we were experiencing was due to the UFS file system
requiring synchronous disk access each time a file is created, as was
pointed out in an email by Greg Hudson last week.
This was proven out further by turning on delayed I/O using the fastfs
tool. When we do this, times on the client go back down to the times
we see on other operating systems.
While I know that using fastfs in a "production" environment is looked
down upon because you can't count on the disk being synchronized in the
event of a power failure, I was wondering if anyone on the list actually
uses this delayed I/O feature of the operating system for things like
users home directories that are backed up nightly.
As Linux file systems default to nosynch and I've never had a problem
with boxes losing data in power failures here at home, I don't see a big
problem with this as long as the box isn't running an oracle production
database or something like that. Given the fact that the computer room
at work has interruptible power and backup generators anyway, I see even
less of a problem.
Also, I wanted to make clear that the server hums along nicely. There
is no performance impact running SVN under Apache on Solaris 9. The
big hit comes from the client creating the work area meta data during
the checkout operation.
Thanks to everyone on the list who responded as I was researching this
problem. Your input was very valuable in pointing me in the right
direction.
-- Ron
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris 9
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Friday, March 19, 2004 7:16 AM -0800 Jay 'Whip' Grizzard
<el...@lupine.org> wrote:
> In the meantime, though, it's an easy thing to test -- If you still have
> /tmp mounted off of swap (which is the default, and makes it a ram-based
> filesystem), try checking out your repository to a directory in /tmp. If
> it's still slow, your problem is something else. If it's much faster,
> your problem is almost certainly all the non-async stuff happening at
> file creation time...
Ron, I definitely would recommend testing this next. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Ron Bieber <ro...@bieberlabs.com>.
Folks,
I wanted to send a follow up message to my original email on using
Solaris 9 with Solaris and the drastic performance penalty I was seeing
on the Solaris/SPARC platform.
The performance problem only occurred when using the client. After
performing many tests, including turning logging on, mounting with
noatime, and others we were unable to get our subversion client running
at an acceptable level to use it in a production environment without
causing a lot of wait time for our developers. We consistently saw an
76-85% decrease in performance when running the client on Solaris vs.
Windows or Linux boxes.
The performance hit we were experiencing was due to the UFS file system
requiring synchronous disk access each time a file is created, as was
pointed out in an email by Greg Hudson last week.
This was proven out further by turning on delayed I/O using the fastfs
tool. When we do this, times on the client go back down to the times
we see on other operating systems.
While I know that using fastfs in a "production" environment is looked
down upon because you can't count on the disk being synchronized in the
event of a power failure, I was wondering if anyone on the list actually
uses this delayed I/O feature of the operating system for things like
users home directories that are backed up nightly.
As Linux file systems default to nosynch and I've never had a problem
with boxes losing data in power failures here at home, I don't see a big
problem with this as long as the box isn't running an oracle production
database or something like that. Given the fact that the computer room
at work has interruptible power and backup generators anyway, I see even
less of a problem.
Also, I wanted to make clear that the server hums along nicely. There
is no performance impact running SVN under Apache on Solaris 9. The
big hit comes from the client creating the work area meta data during
the checkout operation.
Thanks to everyone on the list who responded as I was researching this
problem. Your input was very valuable in pointing me in the right
direction.
-- Ron
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on Solaris 9
Posted by Jay 'Whip' Grizzard <el...@lupine.org>.
> > b) Configure this apache to serve some static files and try massively
> > downloading them to different hosts (wget -a and maybe most important ab
> > - apache benchmark).
> I'll try this and post back.
[snip]
I may have missed it (though a search of my mailbox doesn't seem to say so),
but did you follow up on the comments reguarding filesystem journaling and
the like on Solaris? Operations that create a lot of files /are/ somewhat
slow on Solaris (and, really, any OS with sane filesystem semantics (e.g.
not linux)), and could explain your problems, since subversion creates so
many files when creating a repository.
If this is the case, turning on journaling on your filesystems will surely
help.
In the meantime, though, it's an easy thing to test -- If you still have
/tmp mounted off of swap (which is the default, and makes it a ram-based
filesystem), try checking out your repository to a directory in /tmp. If
it's still slow, your problem is something else. If it's much faster,
your problem is almost certainly all the non-async stuff happening at
file creation time...
-jay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Ron Bieber <ro...@bieberlabs.com>.
On Fri, 2004-03-19 at 04:52, Marcin Kasperski wrote:
> > The server is a 1.0.1 server running Solaris 9. It is a 2 CPU / 4 gig
> > box. Subversion is running under Apache 2.0.48 with mod_deflate
> > enabled.
>
> I do not use Solaris so I can not comment on the possible reasons but in
> your position I would try the following two things which can give some
> directions to the further search:
>
> a) Configure svnserve and check whether the numbers change anyhow when
> using svnserve instead of apache+mod_svn
Already tried svnserve and the results are comparable.
>
>
> b) Configure this apache to serve some static files and try massively
> downloading them to different hosts (wget -a and maybe most important ab
> - apache benchmark).
I'll try this and post back.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Marcin Kasperski <Ma...@softax.com.pl>.
> The server is a 1.0.1 server running Solaris 9. It is a 2 CPU / 4 gig
> box. Subversion is running under Apache 2.0.48 with mod_deflate
> enabled.
I do not use Solaris so I can not comment on the possible reasons but in
your position I would try the following two things which can give some
directions to the further search:
a) Configure svnserve and check whether the numbers change anyhow when
using svnserve instead of apache+mod_svn
b) Configure this apache to serve some static files and try massively
downloading them to different hosts (wget -a and maybe most important ab
- apache benchmark).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Drastic performance difference with subversion client on
Solaris 9
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2004-03-18 at 07:33, Ron Bieber wrote:
> My question is this: Is anyone else having these drastic differences
> when running the Subversion client on Solaris 9? We've had our admins
> check the networking between the boxes and I'm told this is not the
> problem. This ties off with the fact that checking out the same 93
> file module on the repository box itself takes the same amount of time
> (using localhost) as the other Solaris boxes on the network.
We have a limited number of Solaris users, I think. My first guess
would be that the problem is in raw file creation speed. Some
filesystems require a synchronous disk access each time a file is
created.
To test this theory, see how long it takes to "cp -r" a working
directory under Solaris. Subversion can't get any faster than that
without changing its working directory structure (which is more verbose
than it needs to be, admittedly).
Turning on logging might help the problem (in /etc/vfstab, replace "-"
with "logging" in the last column for the appropriate filesystem), but
it might not; I don't know if the Solaris UFS logging support removes
the requirement for a synchronous disk access at file creation. My
reading of
http://www.solarisinternals.com/si/reading/sunworldonline/swol-05-1999/swol-05-filesystem.html is that it may just show things down.
You could also mount the filesystem async, but that's really dangerous.
(People lived with that danger in Linux for years with ext2fs, but the
ext2 fsck was perhaps more sophisticated at cleaning up failures.
Again, not sure.) The web page I cited above suggests there are some
commercially available filesystems which might help (but you'd also be
marginalizing yourself, which is never good). Purchasing faster disks
might also be an option.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org