You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Ketting, Michael" <mi...@rubicon.eu> on 2011/08/09 10:13:49 UTC
Significant checkout performance degradation between 1.6.1 and 1.7b2
Hello!
I've recently picked up the subversion 1.7 beta 2 build (included in the latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files, ~2,000 folders, ~180MB).
With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10 minutes.
Is this performance degradation inherent with the use of the centralized SVN information, and thus an intentional tradeoff for the blazing fast Commits/Updates?
Naively, I'd hoped that the checkout speed would get closer to the export-speed with Subversion 1.7, since the Updates are faster, too.
Interestingly, it looks like the export speed also degraded. With Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110 seconds.
Regards, Michael
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Thanks for the quick response, Stefan!
No, the setup is quite ordinary:
The client is using Windows Server 2008 (64bit), using svn 32bit or 64bit makes no difference.
The working copy gets written to a regular disk.
I'm using HTTPS.
The server uses the CollabNet SVN Server 1.6.4
Our repository is publicly available:
https://svn.re-motion.org/svn/Remotion/trunk
The performance numbers (and the significant difference) I got were from the intranet, we're taking Gigabit Ethernet and Switches.
I've now also tried the same from my home computer via the Internet, it was not really that conclusive, with the 1.6 client actually taking a little bit longer than the 1.7 client. But that could also be due to a slower harddisk, network latency, etc. The numbers for the intranet access have been consistent with the checkout taking twice as long and the actual timing.
Regards, Michael
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: Dienstag, 09. August 2011 12:16
> To: Ketting, Michael
> Cc: dev@subversion.apache.org
> Subject: Re: Significant checkout performance degradation between 1.6.1
> and 1.7b2
>
> On Tue, Aug 09, 2011 at 08:13:49AM +0000, Ketting, Michael wrote:
> > Hello!
> >
> > I've recently picked up the subversion 1.7 beta 2 build (included in the
> latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files,
> ~2,000 folders, ~180MB).
> > With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta
> 2, it takes about 10 minutes.
> >
> > Is this performance degradation inherent with the use of the centralized
> SVN information, and thus an intentional tradeoff for the blazing fast
> Commits/Updates?
> > Naively, I'd hoped that the checkout speed would get closer to the export-
> speed with Subversion 1.7, since the Updates are faster, too.
> >
> > Interestingly, it looks like the export speed also degraded. With
> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes
> 110 seconds.
> >
> > Regards, Michael
>
> Is there anything unusual about your setup? E.g. is your working copy
> on a network filesystem or on a local drive?
>
> Generally speaking, checkout speed depends on the files/directories ratio.
>
> In our own measurements with contrived use cases[1] 1.7 checkout is
> faster than 1.6 if few files are spread across many directories.
> But if there are many files and few directories the numbers work
> against 1.7 (checkout is about 1.5 times slower for 100 files in a
> single directory). Your numbers show that the difference can be quite
> large in real-world scenarios.
>
> It is quite possible that there is room for more optimisation and that
> performance fixes will be made even after the 1.7.0 release in patch
> releases.
>
> Thanks for testing the beta! Reports like yours are very valuable.
>
> [1] http://mail-archives.apache.org/mod_mbox/subversion-
> dev/201107.mbox/%3C20110725021010.D5DDF2019C@svn-
> qavm.apache.org%3E
Re: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Aug 09, 2011 at 08:13:49AM +0000, Ketting, Michael wrote:
> Hello!
>
> I've recently picked up the subversion 1.7 beta 2 build (included in the latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files, ~2,000 folders, ~180MB).
> With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10 minutes.
>
> Is this performance degradation inherent with the use of the centralized SVN information, and thus an intentional tradeoff for the blazing fast Commits/Updates?
> Naively, I'd hoped that the checkout speed would get closer to the export-speed with Subversion 1.7, since the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110 seconds.
>
> Regards, Michael
Is there anything unusual about your setup? E.g. is your working copy
on a network filesystem or on a local drive?
Generally speaking, checkout speed depends on the files/directories ratio.
In our own measurements with contrived use cases[1] 1.7 checkout is
faster than 1.6 if few files are spread across many directories.
But if there are many files and few directories the numbers work
against 1.7 (checkout is about 1.5 times slower for 100 files in a
single directory). Your numbers show that the difference can be quite
large in real-world scenarios.
It is quite possible that there is room for more optimisation and that
performance fixes will be made even after the 1.7.0 release in patch releases.
Thanks for testing the beta! Reports like yours are very valuable.
[1] http://mail-archives.apache.org/mod_mbox/subversion-dev/201107.mbox/%3C20110725021010.D5DDF2019C@svn-qavm.apache.org%3E
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hi Mark!
Thanks, I tried this. And you were right, it looks like it's already been using neon as a default. I tried switching it to serve, and it promptly took 10-15% longer than with neon.
Will now check your other suggestions.
Michael
> -----Original Message-----
> From: Mark Phippard [mailto:markphip@gmail.com]
> Sent: Dienstag, 09. August 2011 17:52
> To: Ketting, Michael
> Cc: dev@subversion.apache.org
> Subject: Re: Significant checkout performance degradation between 1.6.1
> and 1.7b2
>
> On Tue, Aug 9, 2011 at 8:49 AM, Ketting, Michael
> <mi...@rubicon.eu> wrote:
>
>
> Hi Mark!
>
> Yes, I'm using HTTP. Well, HTTPS, but from the rest of your response,
> that looks like a minor detail.
>
> This tidbit about the library is interesting news. I just did a bit of
> mailing-list research on this subject.
> How do I switch the library? Is there a command line option or a
> config-setting for this?
> I'm using the precompiled version from collabnet, and have never
> actually compiled subversion myself.
>
>
>
> In Windows Explorer type %APPDATA%\Subversion in the address bar. This
> will take you to the folder where the configuration is stored.
>
> Open the file named "servers" in a text editor like Notepad.
>
> Scroll to the bottom section and add:
>
> [global]
> http-library = neon
>
> This explicitly sets it to use Neon. Keep in mind, that it looks like
> TortoiseSVN is already compiled with Neon as the default, just as SVN 1.6 is.
> So most likely this will not change anything. I do not know when
> TortoiseSVN changed the default, so it is possible you have a client
> compiled with Serf as default. It is worth a shot anyway.
>
> --
> Thanks
>
> Mark Phippard
> http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Aug 9, 2011 at 8:49 AM, Ketting, Michael <michael.ketting@rubicon.eu
> wrote:
> Hi Mark!
>
> Yes, I'm using HTTP. Well, HTTPS, but from the rest of your response, that
> looks like a minor detail.
>
> This tidbit about the library is interesting news. I just did a bit of
> mailing-list research on this subject.
> How do I switch the library? Is there a command line option or a
> config-setting for this?
> I'm using the precompiled version from collabnet, and have never actually
> compiled subversion myself.
>
In Windows Explorer type %APPDATA%\Subversion in the address bar. This will
take you to the folder where the configuration is stored.
Open the file named "servers" in a text editor like Notepad.
Scroll to the bottom section and add:
[global]
http-library = neon
This explicitly sets it to use Neon. Keep in mind, that it looks like
TortoiseSVN is already compiled with Neon as the default, just as SVN 1.6
is. So most likely this will not change anything. I do not know when
TortoiseSVN changed the default, so it is possible you have a client
compiled with Serf as default. It is worth a shot anyway.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hi Mark!
Yes, I'm using HTTP. Well, HTTPS, but from the rest of your response, that looks like a minor detail.
This tidbit about the library is interesting news. I just did a bit of mailing-list research on this subject.
How do I switch the library? Is there a command line option or a config-setting for this?
I'm using the precompiled version from collabnet, and have never actually compiled subversion myself.
Regards, Michael
> -----Original Message-----
> From: Mark Phippard [mailto:markphip@gmail.com]
> Sent: Dienstag, 09. August 2011 14:08
> To: Ketting, Michael
> Cc: dev@subversion.apache.org
> Subject: Re: Significant checkout performance degradation between 1.6.1
> and 1.7b2
>
> Is this via http? Given that export is slower I'd be willing to bet the
> performance difference is from the new http client library - serf. It is
> typically slower than Neon. Try switching to neon and run it again.
>
> Sent from my iPhone
>
> On Aug 9, 2011, at 4:13 AM, "Ketting, Michael"
> <mi...@rubicon.eu> wrote:
>
> > Hello!
> >
> > I've recently picked up the subversion 1.7 beta 2 build (included in the
> latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files,
> ~2,000 folders, ~180MB).
> > With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta
> 2, it takes about 10 minutes.
> >
> > Is this performance degradation inherent with the use of the centralized
> SVN information, and thus an intentional tradeoff for the blazing fast
> Commits/Updates?
> > Naively, I'd hoped that the checkout speed would get closer to the export-
> speed with Subversion 1.7, since the Updates are faster, too.
> >
> > Interestingly, it looks like the export speed also degraded. With
> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes
> 110 seconds.
> >
> > Regards, Michael
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hi Bert!
No, that wasn't running. And just the make sure, I redid my checkout tests, closed all explorer windows and checked with TaskManager. Got the same timing results as before.
Regards, Michael
________________________________
From: Bert Huijben [bert@qqmail.nl]
Sent: Thursday, August 11, 2011 12:52
To: Ketting, Michael; dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
A completely different question:
Do you have a recent TortoiseSVN (TSvnCache.exe) running while checking out for those tests?
I just ruined a testrun by accidentally enabling TSvnCache on it (by accessing a parent directory with the Windows explorer), and for the rest of the checkout TSvnCache took consistently 3 times more CPU than the checkout process by continuously calling ‘svn status’ on the same working copy.
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
Sent: donderdag 11 augustus 2011 12:13
To: dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Bert, you can access the repository here, in case you want to take a closer look: https://svn.re-motion.org/svn/Remotion/trunk/
> What about svn:needs-lock?
No, don't have any locked files.
> svn:eol-style
Never heard of that one before. No, I've never set it. And as far as I know, it's not set in the other repositories, either.
> svn:keyword
Mainly, it's a couple of mime-types for binary files. Maybe a 100? 200? Plus a few ignores. And probably some merge-info sprinkled across the trunk.
Michael
________________________________
From: Bert Huijben [bert@qqmail.nl]
Sent: Thursday, August 11, 2011 12:01
To: Ketting, Michael; dev@subversion.apache.org<ma...@subversion.apache.org>
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Can you tell a bit more about this ‘worst case’ working copy?
Does it use svn:keywords in many places?
What about svn:needs-lock?
More svn:eol-style keywords than the other working copies?
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]<mailto:[mailto:michael.ketting@rubicon.eu]>
Sent: donderdag 11 augustus 2011 10:54
To: dev@subversion.apache.org<ma...@subversion.apache.org>
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Just a bit more information:
I've now also tried the chekcout tests with other other big trunks in our company:
One took 7min (svn 1.6) vs 9min (svn 1.7), the other 4min (svn 1.6) vs 6min (svn 1.7), so, both are slower but in the range also measured with the benchmarks.
Looks like my own project really is the worst case scenario :)
Regards, Michael
________________________________
From: Mark Phippard [markphip@gmail.com]
Sent: Tuesday, August 09, 2011 17:05
To: Ketting, Michael
Cc: dev@subversion.apache.org<ma...@subversion.apache.org>
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com>> wrote:
Is this via http? Given that export is slower I'd be willing to bet the performance difference is from the new http client library - serf. It is typically slower than Neon. Try switching to neon and run it again.
I updated to the latest Beta of TortoiseSVN and it looks to me like they have changed the default HTTP client to Neon already. So unless you have specifically made serf the default client in your servers file it is not likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the benchmarks using 1.6 and 1.7 to see what kind of results you see? When I run the tests I see considerable performance gain with 1.7. The "FolderTests" are probably the closes tests to your scenario. It will be easier to focus on any remaining performance issues if we can identify and measure them in an open and consistent manner so we can see progress and the impact of different changes.
If these benchmarks do not show the same problems you see on your real code, then we need to add more benchmarks so that we can capture whatever the problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Bert Huijben <be...@qqmail.nl>.
A completely different question:
Do you have a recent TortoiseSVN (TSvnCache.exe) running while checking out
for those tests?
I just ruined a testrun by accidentally enabling TSvnCache on it (by
accessing a parent directory with the Windows explorer), and for the rest of
the checkout TSvnCache took consistently 3 times more CPU than the checkout
process by continuously calling 'svn status' on the same working copy.
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
Sent: donderdag 11 augustus 2011 12:13
To: dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Bert, you can access the repository here, in case you want to take a closer
look: https://svn.re-motion.org/svn/Remotion/trunk/
> What about svn:needs-lock?
No, don't have any locked files.
> svn:eol-style
Never heard of that one before. No, I've never set it. And as far as I know,
it's not set in the other repositories, either.
> svn:keyword
Mainly, it's a couple of mime-types for binary files. Maybe a 100? 200? Plus
a few ignores. And probably some merge-info sprinkled across the trunk.
Michael
_____
From: Bert Huijben [bert@qqmail.nl]
Sent: Thursday, August 11, 2011 12:01
To: Ketting, Michael; dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Can you tell a bit more about this 'worst case' working copy?
Does it use svn:keywords in many places?
What about svn:needs-lock?
More svn:eol-style keywords than the other working copies?
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
Sent: donderdag 11 augustus 2011 10:54
To: dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Just a bit more information:
I've now also tried the chekcout tests with other other big trunks in our
company:
One took 7min (svn 1.6) vs 9min (svn 1.7), the other 4min (svn 1.6) vs 6min
(svn 1.7), so, both are slower but in the range also measured with the
benchmarks.
Looks like my own project really is the worst case scenario :)
Regards, Michael
_____
From: Mark Phippard [markphip@gmail.com]
Sent: Tuesday, August 09, 2011 17:05
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and
1.7b2
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com> wrote:
Is this via http? Given that export is slower I'd be willing to bet the
performance difference is from the new http client library - serf. It is
typically slower than Neon. Try switching to neon and run it again.
I updated to the latest Beta of TortoiseSVN and it looks to me like they
have changed the default HTTP client to Neon already. So unless you have
specifically made serf the default client in your servers file it is not
likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion
performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the
benchmarks using 1.6 and 1.7 to see what kind of results you see? When I
run the tests I see considerable performance gain with 1.7. The
"FolderTests" are probably the closes tests to your scenario. It will be
easier to focus on any remaining performance issues if we can identify and
measure them in an open and consistent manner so we can see progress and the
impact of different changes.
If these benchmarks do not show the same problems you see on your real code,
then we need to add more benchmarks so that we can capture whatever the
problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Bert, you can access the repository here, in case you want to take a closer look: https://svn.re-motion.org/svn/Remotion/trunk/
> What about svn:needs-lock?
No, don't have any locked files.
> svn:eol-style
Never heard of that one before. No, I've never set it. And as far as I know, it's not set in the other repositories, either.
> svn:keyword
Mainly, it's a couple of mime-types for binary files. Maybe a 100? 200? Plus a few ignores. And probably some merge-info sprinkled across the trunk.
Michael
________________________________
From: Bert Huijben [bert@qqmail.nl]
Sent: Thursday, August 11, 2011 12:01
To: Ketting, Michael; dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Can you tell a bit more about this ‘worst case’ working copy?
Does it use svn:keywords in many places?
What about svn:needs-lock?
More svn:eol-style keywords than the other working copies?
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
Sent: donderdag 11 augustus 2011 10:54
To: dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Just a bit more information:
I've now also tried the chekcout tests with other other big trunks in our company:
One took 7min (svn 1.6) vs 9min (svn 1.7), the other 4min (svn 1.6) vs 6min (svn 1.7), so, both are slower but in the range also measured with the benchmarks.
Looks like my own project really is the worst case scenario :)
Regards, Michael
________________________________
From: Mark Phippard [markphip@gmail.com]
Sent: Tuesday, August 09, 2011 17:05
To: Ketting, Michael
Cc: dev@subversion.apache.org<ma...@subversion.apache.org>
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com>> wrote:
Is this via http? Given that export is slower I'd be willing to bet the performance difference is from the new http client library - serf. It is typically slower than Neon. Try switching to neon and run it again.
I updated to the latest Beta of TortoiseSVN and it looks to me like they have changed the default HTTP client to Neon already. So unless you have specifically made serf the default client in your servers file it is not likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the benchmarks using 1.6 and 1.7 to see what kind of results you see? When I run the tests I see considerable performance gain with 1.7. The "FolderTests" are probably the closes tests to your scenario. It will be easier to focus on any remaining performance issues if we can identify and measure them in an open and consistent manner so we can see progress and the impact of different changes.
If these benchmarks do not show the same problems you see on your real code, then we need to add more benchmarks so that we can capture whatever the problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Bert Huijben <be...@qqmail.nl>.
Can you tell a bit more about this 'worst case' working copy?
Does it use svn:keywords in many places?
What about svn:needs-lock?
More svn:eol-style keywords than the other working copies?
Bert
From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
Sent: donderdag 11 augustus 2011 10:54
To: dev@subversion.apache.org
Subject: RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Just a bit more information:
I've now also tried the chekcout tests with other other big trunks in our
company:
One took 7min (svn 1.6) vs 9min (svn 1.7), the other 4min (svn 1.6) vs 6min
(svn 1.7), so, both are slower but in the range also measured with the
benchmarks.
Looks like my own project really is the worst case scenario :)
Regards, Michael
_____
From: Mark Phippard [markphip@gmail.com]
Sent: Tuesday, August 09, 2011 17:05
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and
1.7b2
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com> wrote:
Is this via http? Given that export is slower I'd be willing to bet the
performance difference is from the new http client library - serf. It is
typically slower than Neon. Try switching to neon and run it again.
I updated to the latest Beta of TortoiseSVN and it looks to me like they
have changed the default HTTP client to Neon already. So unless you have
specifically made serf the default client in your servers file it is not
likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion
performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the
benchmarks using 1.6 and 1.7 to see what kind of results you see? When I
run the tests I see considerable performance gain with 1.7. The
"FolderTests" are probably the closes tests to your scenario. It will be
easier to focus on any remaining performance issues if we can identify and
measure them in an open and consistent manner so we can see progress and the
impact of different changes.
If these benchmarks do not show the same problems you see on your real code,
then we need to add more benchmarks so that we can capture whatever the
problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Just a bit more information:
I've now also tried the chekcout tests with other other big trunks in our company:
One took 7min (svn 1.6) vs 9min (svn 1.7), the other 4min (svn 1.6) vs 6min (svn 1.7), so, both are slower but in the range also measured with the benchmarks.
Looks like my own project really is the worst case scenario :)
Regards, Michael
________________________________
From: Mark Phippard [markphip@gmail.com]
Sent: Tuesday, August 09, 2011 17:05
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com>> wrote:
Is this via http? Given that export is slower I'd be willing to bet the performance difference is from the new http client library - serf. It is typically slower than Neon. Try switching to neon and run it again.
I updated to the latest Beta of TortoiseSVN and it looks to me like they have changed the default HTTP client to Neon already. So unless you have specifically made serf the default client in your servers file it is not likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the benchmarks using 1.6 and 1.7 to see what kind of results you see? When I run the tests I see considerable performance gain with 1.7. The "FolderTests" are probably the closes tests to your scenario. It will be easier to focus on any remaining performance issues if we can identify and measure them in an open and consistent manner so we can see progress and the impact of different changes.
If these benchmarks do not show the same problems you see on your real code, then we need to add more benchmarks so that we can capture whatever the problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Aug 10, 2011 at 3:31 PM, Johan Corveleyn <jc...@gmail.com> wrote:
Just a thought: do you have any anti-virus or other background
> scanning software active on the client during these tests? If so,
> could you try to rerun the tests without it? It's conceivable that
> some types of anti-virus have more impact on 1.7 than they do on 1.6.
>
> And just to reassure: the tests are run on a local hard disk right?
>
That is a good point. I did configure my A/V to ignore the folder where I
run the benchmarks a long time ago. This was back in March, but back then I
did see a big speedup in 1.7 when I did this. So it is possible A/V impacts
1.7 checkout more than it does 1.6.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hello Johen!
Yes, I'm running them on a local disk.
Good point about the AV software. Had MS Forefront running. Redid my own checkout tests with the directory excluded but got the same results.
Then I tried to rerun the svn benchmark, but I can't get them to run, now:
I set my path to the svn-binaries folder and started the tests the same as I did before (or apparently, not) but I only get the following stacktrace (plus the same for the other commands in the test)
$ svn --version -q
Error running SVN command
java.lang.NullPointerException
at com.devdaily.system.SystemCommandExecutor.getStandardOutputFromCommand(SystemCommandExecutor.java:134)
at com.collabnet.subversion.benchmark.command.SVNCommand.run(SVNCommand.java:69)
at com.collabnet.subversion.benchmark.command.SVNCommand.runAndReturnOutput(SVNCommand.java:52)
at com.collabnet.subversion.benchmark.test.AbstractTest.setup(AbstractTest.java:74)
at com.collabnet.subversion.benchmark.test.AbstractTest.run(AbstractTest.java:128)
at com.collabnet.subversion.benchmark.RunTests.run(RunTests.java:73)
at com.collabnet.subversion.benchmark.RunTests.main(RunTests.java:154)
And yes, actually running the svn version command on the command line works, so the PATH is set correctly. Any idea/suggestions what I might be doing wrong? Must be something obvious but since the stacktrace isn't helping ...
Regards, Michael
________________________________________
From: Johan Corveleyn [jcorvel@gmail.com]
Sent: Wednesday, August 10, 2011 21:31
To: Ketting, Michael
Cc: Mark Phippard; dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Wed, Aug 10, 2011 at 9:07 PM, Mark Phippard <ma...@gmail.com> wrote:
> On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael
> <mi...@rubicon.eu> wrote:
>>
>> Here're the test results for the basic merge repository:
>>
>> Subversion 1.7.0 Beta 3, binaries from collabnet
>> Basic Tests:
>> | 1.7.0-beta3 | rNNNNNNNN | 0:54.539 | 0:47.774 | 0:00.214 | 0:00.075 |
>> 0:00.101 | 0:01.187 | 0:01.365
>>
>> Subversion 1.6.17, binaries from VisualSVN
>>
>> Basic Tests:
>> | 1.6.17 | rNNNNNNNN | 0:42.548 | 0:39.758 | 0:16.504 | 0:00.257 |
>> 0:00.105 | 0:00.637 | 0:02.445
>
> We get pretty different results on these tests (which are just using the
> Subversion code base for an example in the tests). I used a 1.7 server so
> my 1.7 client was able to benefit from the HTTPv2 protocol. I am re-running
> my 1.7 tests with a 1.6 server and will update you when they are done.
Michael,
Just a thought: do you have any anti-virus or other background
scanning software active on the client during these tests? If so,
could you try to rerun the tests without it? It's conceivable that
some types of anti-virus have more impact on 1.7 than they do on 1.6.
And just to reassure: the tests are run on a local hard disk right?
--
Johan
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Aug 10, 2011 at 9:07 PM, Mark Phippard <ma...@gmail.com> wrote:
> On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael
> <mi...@rubicon.eu> wrote:
>>
>> Here're the test results for the basic merge repository:
>>
>> Subversion 1.7.0 Beta 3, binaries from collabnet
>> Basic Tests:
>> | 1.7.0-beta3 | rNNNNNNNN | 0:54.539 | 0:47.774 | 0:00.214 | 0:00.075 |
>> 0:00.101 | 0:01.187 | 0:01.365
>>
>> Subversion 1.6.17, binaries from VisualSVN
>>
>> Basic Tests:
>> | 1.6.17 | rNNNNNNNN | 0:42.548 | 0:39.758 | 0:16.504 | 0:00.257 |
>> 0:00.105 | 0:00.637 | 0:02.445
>
> We get pretty different results on these tests (which are just using the
> Subversion code base for an example in the tests). I used a 1.7 server so
> my 1.7 client was able to benefit from the HTTPv2 protocol. I am re-running
> my 1.7 tests with a 1.6 server and will update you when they are done.
Michael,
Just a thought: do you have any anti-virus or other background
scanning software active on the client during these tests? If so,
could you try to rerun the tests without it? It's conceivable that
some types of anti-virus have more impact on 1.7 than they do on 1.6.
And just to reassure: the tests are run on a local hard disk right?
--
Johan
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael <
michael.ketting@rubicon.eu> wrote:
> Here're the test results for the basic merge repository:
>
> Subversion 1.7.0 Beta 3, binaries from collabnet
> Basic Tests:
> | 1.7.0-beta3 | rNNNNNNNN | 0:54.539 | 0:47.774 | 0:00.214 | 0:00.075 |
> 0:00.101 | 0:01.187 | 0:01.365
>
> Subversion 1.6.17, binaries from VisualSVN
>
> Basic Tests:
> | 1.6.17 | rNNNNNNNN | 0:42.548 | 0:39.758 | 0:16.504 | 0:00.257 | 0:00.105
> | 0:00.637 | 0:02.445
>
We get pretty different results on these tests (which are just using the
Subversion code base for an example in the tests). I used a 1.7 server so
my 1.7 client was able to benefit from the HTTPv2 protocol. I am re-running
my 1.7 tests with a 1.6 server and will update you when they are done.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hi Mark!
Yeah, I was just going to suggest the max path length issue but you beat me to it :)
The runtime really does baffle me, though. The 1.6.17 sounds about right. OK, a minute faster than mine, but...
I reran the test with a relative path for svn 1.7, just to make sure, but still takes as long as before, so I've got no idea what's causing it.
I think it's also of note that Philip got the same factor on his Linux setup, although the total was an order of magnitude smaller.
No idea what to make of that...
Regards, Michael
________________________________
From: Mark Phippard [markphip@gmail.com]
Sent: Thursday, August 11, 2011 18:13
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Thu, Aug 11, 2011 at 11:45 AM, Mark Phippard <ma...@gmail.com>> wrote:
On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael <mi...@rubicon.eu>> wrote:
Our repository is open source, so, in case you believe it helps with benchmarking/finding the bottleneck, you're welcome to exporting the trunk (https://svn.re-motion.org/svn/Remotion/trunk/) and creating a new benchmark repository. Am I correct in my assumption that building a new repository based only on the latest revision of the trunk would result in similar performance figures given it appears to be a client related issue?
I am doing some testing with your repository. I get similar times as you do when using your repository from a Windows client.
However, every time I checkout your repository using 1.6.17 it runs for about 5 minutes but always ends in failure:
svn: Your .svn/tmp directory may be missing or corrupt; run 'svn cleanup' and try again
svn: Can't open file 'test1\Remotion\Data\UnitTests\DomainObjects\Core\Mapping\TestDomain\Validation\Integration\NotSupportedRelations\BidirectionalRelation_RelatedObjectTypeDoesNotMatchOppositeProperty_BelowInheritanceRoot\.svn\tmp\text-base\InvalidRelationClass1.cs.svn-base': The system cannot find the path specified.
I thought this might be an A/V problem, so I moved to a Win2k8 server that does not have any A/V installed. I got the same problem which made me realize it was the problem with max path length that has been addressed in 1.7. I re-ran the checkout using an absolute path so that 1.6 would run to completion. I got some odd results in that it all ran really fast:
SVN 1.6.17 = 3:03
SVN 1.7-b3 = 2:40
So using 1.7 it was faster for me. I do not understand how these checkout times got so fast though. I checked the size of the working copy and they were each about 290MB, so I assume they are complete.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Aug 11, 2011 at 11:45 AM, Mark Phippard <ma...@gmail.com> wrote:
> On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael <
> michael.ketting@rubicon.eu> wrote:
>
>
>> Our repository is open source, so, in case you believe it helps with
>> benchmarking/finding the bottleneck, you're welcome to exporting the trunk (
>> https://svn.re-motion.org/svn/Remotion/trunk/) and creating a new
>> benchmark repository. Am I correct in my assumption that building a new
>> repository based only on the latest revision of the trunk would result in
>> similar performance figures given it appears to be a client related issue?
>>
>>
> I am doing some testing with your repository. I get similar times as you
> do when using your repository from a Windows client.
>
> However, every time I checkout your repository using 1.6.17 it runs for
> about 5 minutes but always ends in failure:
>
> svn: Your .svn/tmp directory may be missing or corrupt; run 'svn cleanup'
> and try again
> svn: Can't open file
> 'test1\Remotion\Data\UnitTests\DomainObjects\Core\Mapping\TestDomain\Validation\Integration\NotSupportedRelations\BidirectionalRelation_RelatedObjectTypeDoesNotMatchOppositeProperty_BelowInheritanceRoot\.svn\tmp\text-base\InvalidRelationClass1.cs.svn-base':
> The system cannot find the path specified.
>
>
I thought this might be an A/V problem, so I moved to a Win2k8 server that
does not have any A/V installed. I got the same problem which made me
realize it was the problem with max path length that has been addressed in
1.7. I re-ran the checkout using an absolute path so that 1.6 would run to
completion. I got some odd results in that it all ran really fast:
SVN 1.6.17 = 3:03
SVN 1.7-b3 = 2:40
So using 1.7 it was faster for me. I do not understand how these checkout
times got so fast though. I checked the size of the working copy and they
were each about 290MB, so I assume they are complete.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Aug 10, 2011 at 2:02 PM, Ketting, Michael <
michael.ketting@rubicon.eu> wrote:
> Our repository is open source, so, in case you believe it helps with
> benchmarking/finding the bottleneck, you're welcome to exporting the trunk (
> https://svn.re-motion.org/svn/Remotion/trunk/) and creating a new
> benchmark repository. Am I correct in my assumption that building a new
> repository based only on the latest revision of the trunk would result in
> similar performance figures given it appears to be a client related issue?
>
>
I am doing some testing with your repository. I get similar times as you do
when using your repository from a Windows client.
However, every time I checkout your repository using 1.6.17 it runs for
about 5 minutes but always ends in failure:
svn: Your .svn/tmp directory may be missing or corrupt; run 'svn cleanup'
and try again
svn: Can't open file
'test1\Remotion\Data\UnitTests\DomainObjects\Core\Mapping\TestDomain\Validation\Integration\NotSupportedRelations\BidirectionalRelation_RelatedObjectTypeDoesNotMatchOppositeProperty_BelowInheritanceRoot\.svn\tmp\text-base\InvalidRelationClass1.cs.svn-base':
The system cannot find the path specified.
So I have no idea if the checkout is only half complete. SVN 1.7 runs to
completion but takes about 11 minutes.
I am not sure why you do not get this same error, but I assume you are
looking for it.
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Here're the test results for the basic merge repository:
Subversion 1.7.0 Beta 3, binaries from collabnet
Basic Tests:
| 1.7.0-beta3 | rNNNNNNNN | 0:54.539 | 0:47.774 | 0:00.214 | 0:00.075 | 0:00.101 | 0:01.187 | 0:01.365
Merge Tests:
| 1.7.0-beta3 | rNNNNNNNN | 0:08.154 | 0:08.984 | 0:08.402 | 0:13.511
Folder Tests:
| 1.7.0-beta3 | rNNNNNNNN | 8:37.779 | 0:01.282 | 0:14.041 | 0:31.032 | 0:08.727 | 0:38.879 | 0:34.304 | 0:08.458 | 0:40.519
Binaries Tests:
| 1.7.0-beta3 | rNNNNNNNN | 0:00.046 | 0:00.030 | 0:00.029 | 0:00.027 | 0:00.025 | 0:00.029 | 0:00.029 | 0:00.028 | 0:00.030 | 0:00.031 | 0:00.023 | 0:00.031 | 0:00.030 | 0:00.031 | 0:00.030
**********
Subversion 1.6.17, binaries from VisualSVN
Basic Tests:
| 1.6.17 | rNNNNNNNN | 0:42.548 | 0:39.758 | 0:16.504 | 0:00.257 | 0:00.105 | 0:00.637 | 0:02.445
Merge Tests:
| 1.6.17 | rNNNNNNNN | 0:11.664 | 0:02.343 | 0:14.951 | 0:24.941
Folder Tests:
| 1.6.17 | rNNNNNNNN | 11:58.850 | 0:04.826 | 1:53.336 | 3:34.584 | 0:02.681 | 5:30.473 | 1:41.592 | 1:50.339 | 1:49.615
Binaries Tests:
| 1.6.17 | rNNNNNNNN | 0:00.062 | 0:00.035 | 0:00.030 | 0:00.031 | 0:00.030 | 0:00.037 | 0:00.029 | 0:00.030 | 0:00.029 | 0:00.027 | 0:00.026 | 0:00.030 | 0:00.031 | 0:00.028 | 0:00.030
And just to make the comparison easier, I've rerun the test with our repository on the same machine as the benchmark:
svn 1.6.17 takes 5 minutes, svn 1.7.0 b3 takes 8 minutes for a complete checkout.
The server's a Windows 2008R2 with collabnet's svn server 1.6.4. The client's also a Windows Server 2008R2. They're connected in a switched network, with a latency < 1ms.
Our repository is open source, so, in case you believe it helps with benchmarking/finding the bottleneck, you're welcome to exporting the trunk (https://svn.re-motion.org/svn/Remotion/trunk/) and creating a new benchmark repository. Am I correct in my assumption that building a new repository based only on the latest revision of the trunk would result in similar performance figures given it appears to be a client related issue?
Is there any additional information I can provide to help figuring this out?
Btw: I noticed while copying the performance numbers that the switch performance is also greatly improved. That's awesome news! We have some projects that do lot's of branching but aren't pursuing git/hg yet.
Regards, Michael
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Aug 9, 2011 at 8:07 AM, Mark Phippard <ma...@gmail.com> wrote:
> Is this via http? Given that export is slower I'd be willing to bet the
> performance difference is from the new http client library - serf. It is
> typically slower than Neon. Try switching to neon and run it again.
>
I updated to the latest Beta of TortoiseSVN and it looks to me like they
have changed the default HTTP client to Neon already. So unless you have
specifically made serf the default client in your servers file it is not
likely that this is your problem.
I developed a set of open-source benchmarks to measure Subversion
performance that you can get here:
https://ctf.open.collab.net/sf/sfmain/do/viewProject/projects.csvn
Perhaps you could set up the repository on your server and run the
benchmarks using 1.6 and 1.7 to see what kind of results you see? When I
run the tests I see considerable performance gain with 1.7. The
"FolderTests" are probably the closes tests to your scenario. It will be
easier to focus on any remaining performance issues if we can identify and
measure them in an open and consistent manner so we can see progress and the
impact of different changes.
If these benchmarks do not show the same problems you see on your real code,
then we need to add more benchmarks so that we can capture whatever the
problem is.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
Is this via http? Given that export is slower I'd be willing to bet the performance difference is from the new http client library - serf. It is typically slower than Neon. Try switching to neon and run it again.
Sent from my iPhone
On Aug 9, 2011, at 4:13 AM, "Ketting, Michael" <mi...@rubicon.eu> wrote:
> Hello!
>
> I've recently picked up the subversion 1.7 beta 2 build (included in the latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files, ~2,000 folders, ~180MB).
> With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10 minutes.
>
> Is this performance degradation inherent with the use of the centralized SVN information, and thus an intentional tradeoff for the blazing fast Commits/Updates?
> Naively, I'd hoped that the checkout speed would get closer to the export-speed with Subversion 1.7, since the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110 seconds.
>
> Regards, Michael
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Thanks, Philip. I've tried to build a repository with just the latest trunk or my repository, to make sure we're as much in sync as possible, but it's still taking just about as long as before to do a checkout. Seems to be a Windows/Linux issue. I googled a bit and from what I found, it's either an inherent problem or some config issue that's going to take ages to diagnose. Oh well, something for another day.
Regards, Michael
________________________________________
From: Philip Martin [philip.martin@wandisco.com]
Sent: Thursday, August 11, 2011 00:10
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
"Ketting, Michael" <mi...@rubicon.eu> writes:
> Those are interesting numbers, Philip. When you say local repository,
> you're talking about localhost, but still using HTTP? I'm curious
> about the actual timing since you got the same ratio as long as you
> used a physical disk but had an overall speed improvement by a factor
> of eight. Is that Linux or localhost?
Using HTTP with both Apache and the client running on my desktop machine
(which is not powerful by todays standards). I get much the same speed
checking out on to my Linux laptop over my LAN from Apache running on my
desktop.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Philip Martin <ph...@wandisco.com>.
"Ketting, Michael" <mi...@rubicon.eu> writes:
> Those are interesting numbers, Philip. When you say local repository,
> you're talking about localhost, but still using HTTP? I'm curious
> about the actual timing since you got the same ratio as long as you
> used a physical disk but had an overall speed improvement by a factor
> of eight. Is that Linux or localhost?
Using HTTP with both Apache and the client running on my desktop machine
(which is not powerful by todays standards). I get much the same speed
checking out on to my Linux laptop over my LAN from Apache running on my
desktop.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Those are interesting numbers, Philip. When you say local repository, you're talking about localhost, but still using HTTP?
I'm curious about the actual timing since you got the same ratio as long as you used a physical disk but had an overall speed improvement by a factor of eight. Is that Linux or localhost?
The other tidbit is the working copy on the ram-disk. While it's not an option for daily business, it certainly might be worth considering for a build agent. Also, I'm wondering if it might be not be a good idea to use SqlLite in an in-memory-only setup during checkout and only persist the data to disk once the checkout has completed. After all, no data would be lost if the process crashed. It would just require a fresh check out instead of an update.
Regards, Michael
________________________________________
From: Philip Martin [philip.martin@wandisco.com]
Sent: Wednesday, August 10, 2011 22:01
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
"Ketting, Michael" <mi...@rubicon.eu> writes:
> I've recently picked up the subversion 1.7 beta 2 build (included in
> the latest TortoiseSVN beta) and did a checkout of our solution
> (~10,000 files, ~2,000 folders, ~180MB). With Subversion 1.6.1, it
> takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10
> minutes.
>
> Is this performance degradation inherent with the use of the
> centralized SVN information, and thus an intentional tradeoff for the
> blazing fast Commits/Updates? Naively, I'd hoped that the checkout
> speed would get closer to the export-speed with Subversion 1.7, since
> the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With
> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now
> takes 110 seconds.
I've put a copy of your trunk in a local repository. On my very
ordinary desktop machine running Linux I can checkout in about 30s with
1.6 and about 50s with 1.7. That's consistent with what I have been
seeing for other datasets recently: checkout with 1.7 is slower than 1.6
when using an ordinary disk. Switching to a RAM disk for the working
copy reduces the 1.7 time to 19s making it slightly faster than 1.6
which takes 22s.
Exporting takes about 13s with both 1.6 and 1.7 and appears to be
limited by the speed of the server.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Aug 10, 2011 at 16:51, Branko Čibej <br...@xbc.nu> wrote:
>...
> Likely to be working copy performance, then. I'll bet sqlite commits
> dominate the client end of the checkout time in 1.7.
I wouldn't be surprised. I seem to recall a basic rule of something
like "50 commits per second". And we make at least one commit per
file. My hope was to batch-add files in each directory. Just collect
up the data (name, checksums, props) and batch-add in the
close_directory() in update_editor.c.
But... before trying that, I'd want to run some perf tests to verify
if the svn_wc___db_base_add_file() is truly where we're spending time.
Cheers,
-g
Re: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by Branko Čibej <br...@xbc.nu>.
On 10.08.2011 22:01, Philip Martin wrote:
> "Ketting, Michael" <mi...@rubicon.eu> writes:
>
>> I've recently picked up the subversion 1.7 beta 2 build (included in
>> the latest TortoiseSVN beta) and did a checkout of our solution
>> (~10,000 files, ~2,000 folders, ~180MB). With Subversion 1.6.1, it
>> takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10
>> minutes.
>>
>> Is this performance degradation inherent with the use of the
>> centralized SVN information, and thus an intentional tradeoff for the
>> blazing fast Commits/Updates? Naively, I'd hoped that the checkout
>> speed would get closer to the export-speed with Subversion 1.7, since
>> the Updates are faster, too.
>>
>> Interestingly, it looks like the export speed also degraded. With
>> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now
>> takes 110 seconds.
> I've put a copy of your trunk in a local repository. On my very
> ordinary desktop machine running Linux I can checkout in about 30s with
> 1.6 and about 50s with 1.7. That's consistent with what I have been
> seeing for other datasets recently: checkout with 1.7 is slower than 1.6
> when using an ordinary disk. Switching to a RAM disk for the working
> copy reduces the 1.7 time to 19s making it slightly faster than 1.6
> which takes 22s.
>
> Exporting takes about 13s with both 1.6 and 1.7 and appears to be
> limited by the speed of the server.
Likely to be working copy performance, then. I'll bet sqlite commits
dominate the client end of the checkout time in 1.7.
-- Brane
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Philip Martin <ph...@wandisco.com>.
"Ketting, Michael" <mi...@rubicon.eu> writes:
> I've recently picked up the subversion 1.7 beta 2 build (included in
> the latest TortoiseSVN beta) and did a checkout of our solution
> (~10,000 files, ~2,000 folders, ~180MB). With Subversion 1.6.1, it
> takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10
> minutes.
>
> Is this performance degradation inherent with the use of the
> centralized SVN information, and thus an intentional tradeoff for the
> blazing fast Commits/Updates? Naively, I'd hoped that the checkout
> speed would get closer to the export-speed with Subversion 1.7, since
> the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With
> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now
> takes 110 seconds.
I've put a copy of your trunk in a local repository. On my very
ordinary desktop machine running Linux I can checkout in about 30s with
1.6 and about 50s with 1.7. That's consistent with what I have been
seeing for other datasets recently: checkout with 1.7 is slower than 1.6
when using an ordinary disk. Switching to a RAM disk for the working
copy reduces the 1.7 time to 19s making it slightly faster than 1.6
which takes 22s.
Exporting takes about 13s with both 1.6 and 1.7 and appears to be
limited by the speed of the server.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hi Mark!
That's interesting news on 1.6.17. I upgraded to the latest stable and rerun my checkout tests. At least the checkout's fast now. Am afraid I don't have any real-world numbers for the other stuff. I'm using an SSD for my daily work, so the local file access during Update/Commit isn't an issue any longer.
Please see my other reply for my benchmark results.
Regards, Michael
From: Mark Phippard [mailto:markphip@gmail.com]
Sent: Dienstag, 09. August 2011 20:20
To: Ketting, Michael
Cc: dev@subversion.apache.org
Subject: Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
On Tue, Aug 9, 2011 at 4:13 AM, Ketting, Michael <mi...@rubicon.eu>> wrote:
I've recently picked up the subversion 1.7 beta 2 build (included in the latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files, ~2,000 folders, ~180MB).
With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta 2, it takes about 10 minutes.
Is this performance degradation inherent with the use of the centralized SVN information, and thus an intentional tradeoff for the blazing fast Commits/Updates?
Naively, I'd hoped that the checkout speed would get closer to the export-speed with Subversion 1.7, since the Updates are faster, too.
Interestingly, it looks like the export speed also degraded. With Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110 seconds.
I re-ran the performance benchmarks using 1.6.17 and 1.7.0-beta3.
Server is running 1.7.0-beta3 and running on Solaris x86 VM with 1 GB RAM
Accessed server via https://
Server authentication via LDAP (Active Directory)
Clients configured to use Neon.
Client was Windows 7 laptop accessing server via VPN. Latency to server is about 120ms
You can see the results in this spreadsheet:
https://spreadsheets.google.com/spreadsheet/ccc?key=0AqWkwCpe4YoidHVpMlJhR0V3QmdWSThsb2c5d1FVV3c&hl=en_US
I highlighted in green the areas where SVN 1.7 is significantly faster (most everywhere).
There are a couple areas in red where 1.6 is significantly faster. This is on the test where there are about 4-5K icons in a single folder. Checkout/commit with 1.7 are slower but other areas are faster. FWIW, until 1.6.17 this was about 5x slower in 1.6. We backported a bugfix from 1.7 to 1.6.17 that apparently has even more impact in 1.6 than it did in 1.7.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Aug 9, 2011 at 4:13 AM, Ketting, Michael <michael.ketting@rubicon.eu
> wrote:
> I've recently picked up the subversion 1.7 beta 2 build (included in the
> latest TortoiseSVN beta) and did a checkout of our solution (~10,000 files,
> ~2,000 folders, ~180MB).
> With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7 beta
> 2, it takes about 10 minutes.
>
> Is this performance degradation inherent with the use of the centralized
> SVN information, and thus an intentional tradeoff for the blazing fast
> Commits/Updates?
> Naively, I'd hoped that the checkout speed would get closer to the
> export-speed with Subversion 1.7, since the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With
> Subversion 1.6 it took about 90 seconds, with Subversion 1.7, it now takes
> 110 seconds.
>
>
I re-ran the performance benchmarks using 1.6.17 and 1.7.0-beta3.
Server is running 1.7.0-beta3 and running on Solaris x86 VM with 1 GB RAM
Accessed server via https://
Server authentication via LDAP (Active Directory)
Clients configured to use Neon.
Client was Windows 7 laptop accessing server via VPN. Latency to server is
about 120ms
You can see the results in this spreadsheet:
https://spreadsheets.google.com/spreadsheet/ccc?key=0AqWkwCpe4YoidHVpMlJhR0V3QmdWSThsb2c5d1FVV3c&hl=en_US
I highlighted in green the areas where SVN 1.7 is significantly faster (most
everywhere).
There are a couple areas in red where 1.6 is significantly faster. This is
on the test where there are about 4-5K icons in a single folder.
Checkout/commit with 1.7 are slower but other areas are faster. FWIW,
until 1.6.17 this was about 5x slower in 1.6. We backported a bugfix from
1.7 to 1.6.17 that apparently has even more impact in 1.6 than it did in
1.7.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: Significant checkout performance degradation between 1.6.1 and
1.7b2
Posted by "Ketting, Michael" <mi...@rubicon.eu>.
Hello Bert!
I'm using HTTPS. Should have thought to mention that.
I've added more information in my response to Stefan's mail.
Regards, Michael
> -----Original Message-----
> From: Bert Huijben [mailto:bert@qqmail.nl]
> Sent: Dienstag, 09. August 2011 12:27
> To: Ketting, Michael; dev@subversion.apache.org
> Subject: RE: Significant checkout performance degradation between 1.6.1
> and 1.7b2
>
>
>
> > -----Original Message-----
> > From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
> > Sent: dinsdag 9 augustus 2011 10:14
> > To: dev@subversion.apache.org
> > Subject: Significant checkout performance degradation between 1.6.1 and
> > 1.7b2
> >
> > Hello!
> >
> > I've recently picked up the subversion 1.7 beta 2 build (included in the
> latest
> > TortoiseSVN beta) and did a checkout of our solution (~10,000 files,
> ~2,000
> > folders, ~180MB).
> > With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7
> beta 2,
> > it takes about 10 minutes.
> >
> > Is this performance degradation inherent with the use of the centralized
> SVN
> > information, and thus an intentional tradeoff for the blazing fast
> > Commits/Updates?
> > Naively, I'd hoped that the checkout speed would get closer to the export-
> > speed with Subversion 1.7, since the Updates are faster, too.
> >
> > Interestingly, it looks like the export speed also degraded. With
> Subversion
> > 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110
> seconds.
>
> How do you contact your repository?
>
> http://, svn://, file://, svn+ssh://
>
> Bert
RE: Significant checkout performance degradation between 1.6.1 and 1.7b2
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Ketting, Michael [mailto:michael.ketting@rubicon.eu]
> Sent: dinsdag 9 augustus 2011 10:14
> To: dev@subversion.apache.org
> Subject: Significant checkout performance degradation between 1.6.1 and
> 1.7b2
>
> Hello!
>
> I've recently picked up the subversion 1.7 beta 2 build (included in the
latest
> TortoiseSVN beta) and did a checkout of our solution (~10,000 files,
~2,000
> folders, ~180MB).
> With Subversion 1.6.1, it takes roughly 5 minutes, with Subversion 1.7
beta 2,
> it takes about 10 minutes.
>
> Is this performance degradation inherent with the use of the centralized
SVN
> information, and thus an intentional tradeoff for the blazing fast
> Commits/Updates?
> Naively, I'd hoped that the checkout speed would get closer to the export-
> speed with Subversion 1.7, since the Updates are faster, too.
>
> Interestingly, it looks like the export speed also degraded. With
Subversion
> 1.6 it took about 90 seconds, with Subversion 1.7, it now takes 110
seconds.
How do you contact your repository?
http://, svn://, file://, svn+ssh://
Bert