You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Liu, Xiaodu \"Louie\"" <Xi...@westernasset.com> on 2010/05/26 19:20:30 UTC
SVN merge issue in Solaris
We have the svn merge performance issue on Solaris. For a given repo, if we do svn merge on windows box, it takes about 35 seconds, but same merge will take about 6 minutes.
Both svn version 1.6.6 and 1.6.11 clients are tested with the same issues.
Please help!
Thanks
Louie
**********************************************************************
E-mail sent through the Internet is not secure. Western Asset
therefore recommends that you do not send any confidential or
sensitive information to us via electronic mail, including social
security numbers, account numbers, or personal identification
numbers. Delivery, and or timely delivery of Internet mail is not
guaranteed. Western Asset therefore recommends that you do not send
time sensitive or action-oriented messages to us via electronic
mail.
**********************************************************************
Re: SVN merge issue in Solaris
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, May 26, 2010 at 11:01 PM, Kadam, Shailesh
<Sh...@westernasset.com> wrote:
> Hi Johan,
> We are running on solaris zone and able to run truss on the process, we see
> majority of time svn spends is in reading the .svn/entries file again and again.
>
> Why is that.
>
> Thanks
> Sal
>
> getcwd("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST", 1024) = 0
> open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
> fcntl(6, F_GETFD, 0x00000000) = 0
> fcntl(6, F_SETFD, 0x00000001) = 0
> read(6, " 1 0\n\n d i r\n 4 5 3\n".., 80) = 80
> close(6) = 0
> open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
> fcntl(6, F_GETFD, 0x00000000) = 0
> fcntl(6, F_SETFD, 0x00000001) = 0
> read(6, " 1 0\n\n d i r\n 4 5 3\n".., 4096) = 4096
> read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
> read(6, " 2 0 c f 9 d d 6 a\n 2 0".., 4096) = 4096
> read(6, " 4 1 8 8 8 8 e b 3 0 d a".., 4096) = 4096
> read(6, "\n\n 8 9 7\n\f\n A S I A".., 4096) = 4096
> read(6, " 1 1 . 9 1 8 7 9 9 Z\n e".., 4096) = 4096
> read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
> read(6, "\n\n\n 2 0 1 0 - 0 5 - 2".., 4096) = 4096
> read(6, " M _ P D S _ S E N D _ P".., 4096) = 4096
> read(6, "\n 6\n c o n t r o l d e".., 4096) = 4096
> read(6, " 1 0 - 0 4 - 2 2 T 0 3 :".., 4096) = 4096
> read(6, "\n 2 0 1 0 - 0 5 - 2 6 T".., 4096) = 4096
> read(6, " 5 : 5 0 . 0 1 2 8 9 2 Z".., 4096) = 4096
> read(6, " 2 0 1 0 - 0 5 - 2 6 T 1".., 4096) = 4096
> read(6, " a c 5 f c 8 0 6 2 1 7 7".., 4096) = 4096
> read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
Well, I'm not an SVN developer, just a user who happens to have digged
into some of this stuff. So if you're asking why the .svn/entries are
read over and over again (if that's actually the case), I really
couldn't say. There is probably some amount of optimization/caching in
the current working copy code, but maybe not enough to avoid repeated
reads for all the possible use cases.
However, if you're asking why these files are read at all, I already
gave the answer: these files are part of the "working copy metadata",
bookkeeping info that svn uses to keep track of the working copy. Up
until now (until 1.6.x that is) this metadata is spread in all those
little files (entries, all-wcprops, text-base, props, ...) inside the
.svn directories (one per directory of your working copy). This is
quite inefficient (on some OS's and filesystems it's worse than on
others).
One of the main features of the next SVN release (1.7), which is
currently in development and expected for release late this summer, is
a complete rewrite of the working copy system, with centralized
metadata. This should make these metadata lookups much more
efficient...
If you have a major problem right now with the 1.6 working copy
system, you'll need to have pretty strong arguments to convince anyone
to spend time and energy to fix it in 1.6. Your best bet (apart from
trying some workarounds to alleviate the problem) is probably to wait
for 1.7.
And please do test any beta's or release candidates of 1.7 when they
come out (there are none yet, but there probably will be), to see if
they help, if they solve your problems or cause other issues, ... and
give your feedback to the developers as soon as possible. It will
certainly help to raise the quality of the final product.
P.S.: about the truss data: I see a lot of reads in 4096 byte chunks.
That happens to be the buffer size with which svn (APR actually)
typically reads these things. The fact that you have lots of them
probably means it's a very big entries file. Maybe that's because it's
a directory with a lot of children?
Cheers,
--
Johan
RE: SVN merge issue in Solaris
Posted by "Kadam, Shailesh" <Sh...@westernasset.com>.
Hi Johan,
We are running on solaris zone and able to run truss on the process, we see
majority of time svn spends is in reading the .svn/entries file again and again.
Why is that.
Thanks
Sal
getcwd("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST", 1024) = 0
open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
fcntl(6, F_GETFD, 0x00000000) = 0
fcntl(6, F_SETFD, 0x00000001) = 0
read(6, " 1 0\n\n d i r\n 4 5 3\n".., 80) = 80
close(6) = 0
open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
fcntl(6, F_GETFD, 0x00000000) = 0
fcntl(6, F_SETFD, 0x00000001) = 0
read(6, " 1 0\n\n d i r\n 4 5 3\n".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
read(6, " 2 0 c f 9 d d 6 a\n 2 0".., 4096) = 4096
read(6, " 4 1 8 8 8 8 e b 3 0 d a".., 4096) = 4096
read(6, "\n\n 8 9 7\n\f\n A S I A".., 4096) = 4096
read(6, " 1 1 . 9 1 8 7 9 9 Z\n e".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
read(6, "\n\n\n 2 0 1 0 - 0 5 - 2".., 4096) = 4096
read(6, " M _ P D S _ S E N D _ P".., 4096) = 4096
read(6, "\n 6\n c o n t r o l d e".., 4096) = 4096
read(6, " 1 0 - 0 4 - 2 2 T 0 3 :".., 4096) = 4096
read(6, "\n 2 0 1 0 - 0 5 - 2 6 T".., 4096) = 4096
read(6, " 5 : 5 0 . 0 1 2 8 9 2 Z".., 4096) = 4096
read(6, " 2 0 1 0 - 0 5 - 2 6 T 1".., 4096) = 4096
read(6, " a c 5 f c 8 0 6 2 1 7 7".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
-----Original Message-----
From: Johan Corveleyn [mailto:jcorvel@gmail.com]
Sent: Wednesday, May 26, 2010 1:57 PM
To: Liu, Xiaodu "Louie"
Cc: users@subversion.apache.org; Kadam, Shailesh
Subject: Re: SVN merge issue in Solaris
On Wed, May 26, 2010 at 9:20 PM, Liu, Xiaodu "Louie"
<Xi...@westernasset.com> wrote:
> We have the svn merge performance issue on Solaris. For a given repo,
> if we do svn merge on windows box, it takes about 35 seconds, but same
> merge will take about 6 minutes.
>
> Both svn version 1.6.6 and 1.6.11 clients are tested with the same issues.
There could be lots of differences between both (client) systems. If network connectivity to the repository is the same, I would take a look at the filesystem where the working copy is stored (where you are merging into). For instance, if your working copy on the Solaris box is on a remote filesystem mounted via NFS, or Samba, compared to the Windows box having its working copy on a fast local drive, that could explain the difference.
Keep in mind that Subversion stores its working-copy metadata currently in lots of little files in lots of .svn subdirectories in every directory. Some svn operations are therefor very heavy on I/O.
E.g. for most operations (merge, update, commit, ...) the working copy needs to be locked, and after the operation unlocked. This is done by putting (and removing) little lock files in all those .svn directories.
SVN's next release (1.7, slated for release late summer) will be much better in this area, because its metadata will be centralized in one single database at the root of the working copy.
Cheers,
--
Johan
**********************************************************************
E-mail sent through the Internet is not secure. Western Asset
therefore recommends that you do not send any confidential or
sensitive information to us via electronic mail, including social
security numbers, account numbers, or personal identification
numbers. Delivery, and or timely delivery of Internet mail is not
guaranteed. Western Asset therefore recommends that you do not send
time sensitive or action-oriented messages to us via electronic
mail.
**********************************************************************
RE: SVN merge issue in Solaris
Posted by "Kadam, Shailesh" <Sh...@westernasset.com>.
Hi Johan,
We are running on solaris zone and able to run truss on the process, we see
majority of time svn spends is in reading the .svn/entries file again and again.
Why is that.
Thanks
Sal
getcwd("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST", 1024) = 0
open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
fcntl(6, F_GETFD, 0x00000000) = 0
fcntl(6, F_SETFD, 0x00000001) = 0
read(6, " 1 0\n\n d i r\n 4 5 3\n".., 80) = 80
close(6) = 0
open("/opt/projects/common_services/rjm/autsys-workspace/workspace/merge_test/MERGE_TEST/.svn/entries", O_RDONLY|O_LARGEFILE) = 6
fcntl(6, F_GETFD, 0x00000000) = 0
fcntl(6, F_SETFD, 0x00000001) = 0
read(6, " 1 0\n\n d i r\n 4 5 3\n".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
read(6, " 2 0 c f 9 d d 6 a\n 2 0".., 4096) = 4096
read(6, " 4 1 8 8 8 8 e b 3 0 d a".., 4096) = 4096
read(6, "\n\n 8 9 7\n\f\n A S I A".., 4096) = 4096
read(6, " 1 1 . 9 1 8 7 9 9 Z\n e".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
read(6, "\n\n\n 2 0 1 0 - 0 5 - 2".., 4096) = 4096
read(6, " M _ P D S _ S E N D _ P".., 4096) = 4096
read(6, "\n 6\n c o n t r o l d e".., 4096) = 4096
read(6, " 1 0 - 0 4 - 2 2 T 0 3 :".., 4096) = 4096
read(6, "\n 2 0 1 0 - 0 5 - 2 6 T".., 4096) = 4096
read(6, " 5 : 5 0 . 0 1 2 8 9 2 Z".., 4096) = 4096
read(6, " 2 0 1 0 - 0 5 - 2 6 T 1".., 4096) = 4096
read(6, " a c 5 f c 8 0 6 2 1 7 7".., 4096) = 4096
read(6, "\n\n\n\n\n\n\n\n\n\n\n\n".., 4096) = 4096
-----Original Message-----
From: Johan Corveleyn [mailto:jcorvel@gmail.com]
Sent: Wednesday, May 26, 2010 1:57 PM
To: Liu, Xiaodu "Louie"
Cc: users@subversion.apache.org; Kadam, Shailesh
Subject: Re: SVN merge issue in Solaris
On Wed, May 26, 2010 at 9:20 PM, Liu, Xiaodu "Louie"
<Xi...@westernasset.com> wrote:
> We have the svn merge performance issue on Solaris. For a given repo,
> if we do svn merge on windows box, it takes about 35 seconds, but same
> merge will take about 6 minutes.
>
> Both svn version 1.6.6 and 1.6.11 clients are tested with the same issues.
There could be lots of differences between both (client) systems. If network connectivity to the repository is the same, I would take a look at the filesystem where the working copy is stored (where you are merging into). For instance, if your working copy on the Solaris box is on a remote filesystem mounted via NFS, or Samba, compared to the Windows box having its working copy on a fast local drive, that could explain the difference.
Keep in mind that Subversion stores its working-copy metadata currently in lots of little files in lots of .svn subdirectories in every directory. Some svn operations are therefor very heavy on I/O.
E.g. for most operations (merge, update, commit, ...) the working copy needs to be locked, and after the operation unlocked. This is done by putting (and removing) little lock files in all those .svn directories.
SVN's next release (1.7, slated for release late summer) will be much better in this area, because its metadata will be centralized in one single database at the root of the working copy.
Cheers,
--
Johan
**********************************************************************
E-mail sent through the Internet is not secure. Western Asset
therefore recommends that you do not send any confidential or
sensitive information to us via electronic mail, including social
security numbers, account numbers, or personal identification
numbers. Delivery, and or timely delivery of Internet mail is not
guaranteed. Western Asset therefore recommends that you do not send
time sensitive or action-oriented messages to us via electronic
mail.
**********************************************************************
Re: SVN merge issue in Solaris
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, May 26, 2010 at 9:20 PM, Liu, Xiaodu "Louie"
<Xi...@westernasset.com> wrote:
> We have the svn merge performance issue on Solaris. For a given repo, if
> we do svn merge on windows box, it takes about 35 seconds, but same merge
> will take about 6 minutes.
>
> Both svn version 1.6.6 and 1.6.11 clients are tested with the same issues.
There could be lots of differences between both (client) systems. If
network connectivity to the repository is the same, I would take a
look at the filesystem where the working copy is stored (where you are
merging into). For instance, if your working copy on the Solaris box
is on a remote filesystem mounted via NFS, or Samba, compared to the
Windows box having its working copy on a fast local drive, that could
explain the difference.
Keep in mind that Subversion stores its working-copy metadata
currently in lots of little files in lots of .svn subdirectories in
every directory. Some svn operations are therefor very heavy on I/O.
E.g. for most operations (merge, update, commit, ...) the working copy
needs to be locked, and after the operation unlocked. This is done by
putting (and removing) little lock files in all those .svn
directories.
SVN's next release (1.7, slated for release late summer) will be much
better in this area, because its metadata will be centralized in one
single database at the root of the working copy.
Cheers,
--
Johan