You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Philip Martin <ph...@wandisco.com> on 2011/06/27 12:34:13 UTC
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
stsp@apache.org writes:
> Author: stsp
> Date: Sun Jun 26 12:28:28 2011
> New Revision: 1139766
>
> URL: http://svn.apache.org/viewvc?rev=1139766&view=rev
> Log:
> * subversion/svn/main.c
> (svn_cl__cmd_table): In the help text for 'svn upgrade', explain that
> the upgrade preserves local modifications, and that an upgrade from
> 1.6 to 1.7 takes more time than a new checkout.
>
> Modified:
> subversion/trunk/subversion/svn/main.c
>
> Modified: subversion/trunk/subversion/svn/main.c
> URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/svn/main.c?rev=1139766&r1=1139765&r2=1139766&view=diff
> ==============================================================================
> --- subversion/trunk/subversion/svn/main.c (original)
> +++ subversion/trunk/subversion/svn/main.c Sun Jun 26 12:28:28 2011
> @@ -1380,7 +1380,13 @@ const svn_opt_subcommand_desc2_t svn_cl_
>
> { "upgrade", svn_cl__upgrade, {0}, N_
> ("Upgrade the metadata storage format for a working copy.\n"
> - "usage: upgrade WCPATH...\n"),
> + "usage: upgrade WCPATH...\n"
> + "\n"
> + " Local modifications are preserved.\n"
> + "\n"
> + " Note: Upgrading a working copy from the format used in Subversion 1.6\n"
> + " to the format used in Subversion 1.7 takes much more time than checking\n"
> + " out a new working copy with the 1.7 client.\n"),
I disagree.
Upgrade requires more disk IO but requires significantly less network IO
and uses marginally less CPU. Checkout is likely to be faster when disk
IO is the limiting factor, but if network IO is the limiting factor then
checkout could be significantly slower.
--
Philip
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Jun 28, 2011 at 02:23:35PM +0200, Stefan Sperling wrote:
> On Mon, Jun 27, 2011 at 02:03:18PM +0100, Philip Martin wrote:
> > Stefan Sperling <st...@elego.de> writes:
> >
> > > Maybe we should tell people that they should only use update if
> > > they have local mods, and perform a checkout otherwise.
> > > This rule of thumb is probably going to yield the best overall results.
> >
> > I think we should recomend checkout simply because the code is better
> > tested.
>
> And how would you phrase the corresponding note in the upgrade help text?
>
> Or do you simply not want to mention this at all? I think we should
> mention it somehow.
I've removed the note in r1145612.
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 27, 2011 at 02:03:18PM +0100, Philip Martin wrote:
> Stefan Sperling <st...@elego.de> writes:
>
> > Maybe we should tell people that they should only use update if
> > they have local mods, and perform a checkout otherwise.
> > This rule of thumb is probably going to yield the best overall results.
>
> I think we should recomend checkout simply because the code is better
> tested.
And how would you phrase the corresponding note in the upgrade help text?
Or do you simply not want to mention this at all? I think we should
mention it somehow.
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:
> Maybe we should tell people that they should only use update if
> they have local mods, and perform a checkout otherwise.
> This rule of thumb is probably going to yield the best overall results.
I think we should recomend checkout simply because the code is better
tested.
--
Philip
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 27, 2011 at 01:42:43PM +0100, Philip Martin wrote:
> It's difficult to get the wording right because we don't know enough
> about the relative performance. Should we be indicating any preference
> for one over the other? Using "might" can be interpreted as upgrade
> being preferred and that there is a small chance that it will be slow.
> For all we know upgrade is usually slower and there is only a small
> chance that it is faster.
>
> If we know nothing about a working copy, apart from it being unmodified,
> and nothing about the local machine, the server or the network, do we
> recommend upgrade or checkout? I don't know.
>
Maybe we should tell people that they should only use update if
they have local mods, and perform a checkout otherwise.
This rule of thumb is probably going to yield the best overall results.
Working copy size definitely is a factor here.
I remember earlier measurements that suggested upgrade performance
was linear to WC size. But I don't think that matches what I've observed.
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:
> OK.
>
> What about a note that says that upgrade *might* be slower than checkout?
>
> Index: subversion/svn/main.c
> ===================================================================
> --- subversion/svn/main.c (revision 1140083)
> +++ subversion/svn/main.c (working copy)
> @@ -1385,7 +1385,7 @@ const svn_opt_subcommand_desc2_t svn_cl__cmd_table
> " Local modifications are preserved.\n"
> "\n"
> " Note: Upgrading a working copy from the format used in Subversion 1.6\n"
> - " to the format used in Subversion 1.7 takes much more time than checking\n"
> + " to the format used in Subversion 1.7 might take more time than checking\n"
> " out a new working copy with the 1.7 client.\n"),
> { 'q' } },
It's difficult to get the wording right because we don't know enough
about the relative performance. Should we be indicating any preference
for one over the other? Using "might" can be interpreted as upgrade
being preferred and that there is a small chance that it will be slow.
For all we know upgrade is usually slower and there is only a small
chance that it is faster.
If we know nothing about a working copy, apart from it being unmodified,
and nothing about the local machine, the server or the network, do we
recommend upgrade or checkout? I don't know.
--
Philip
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 27, 2011 at 12:52:37PM +0100, Philip Martin wrote:
> Stefan Sperling <st...@elego.de> writes:
>
> >> I disagree.
> >>
> >> Upgrade requires more disk IO but requires significantly less network IO
> >> and uses marginally less CPU. Checkout is likely to be faster when disk
> >> IO is the limiting factor, but if network IO is the limiting factor then
> >> checkout could be significantly slower.
> >
> > When I upgraded a 1.6.x working copy of 2GB size, it took more than one hour.
> > I eventually just aborted the upgrade. A checkout completes within
> > a couple of minutes.
> >
> > The repository is remote and the bottleneck should be my 16MB/s downstream
> > link (the server is connected to my ISP via the berlin internet exchange
> > at 100Mbit/s). So based on what you're saying I should be seeing a much
> > better upgrade vs. checkout ratio.
> >
> > Link to the repository: http://svn.dslinux.org/svn/trunk
> >
> > svn also seemed to spend more and more time hogging the CPU as the upgrade
> > progressed. Note: I have not profiled this, it is all based on what I
> > perceived while waiting for the upgrade to complete, and I kept box getting
> > more and more impatiant. I'd be willing to repeat this experiment with a
> > profiled build.
>
> I don't know why that upgrade is slow, but perhaps you are limited by
> disk IO. Maybe upgrade doesn't scale properly.
>
> It's wrong to claim that checkout is faster than upgrade
> unconditionally. A checkout of Subversion trunk from Apache takes about
> 10s on my machine, an upgrade takes 5s. That's with a hot cache, with a
> cold cache upgrade takes about 15s. The checkout speed varies,
> sometimes it takes ove 20s. Which one is faster?
OK.
What about a note that says that upgrade *might* be slower than checkout?
Index: subversion/svn/main.c
===================================================================
--- subversion/svn/main.c (revision 1140083)
+++ subversion/svn/main.c (working copy)
@@ -1385,7 +1385,7 @@ const svn_opt_subcommand_desc2_t svn_cl__cmd_table
" Local modifications are preserved.\n"
"\n"
" Note: Upgrading a working copy from the format used in Subversion 1.6\n"
- " to the format used in Subversion 1.7 takes much more time than checking\n"
+ " to the format used in Subversion 1.7 might take more time than checking\n"
" out a new working copy with the 1.7 client.\n"),
{ 'q' } },
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:
>> I disagree.
>>
>> Upgrade requires more disk IO but requires significantly less network IO
>> and uses marginally less CPU. Checkout is likely to be faster when disk
>> IO is the limiting factor, but if network IO is the limiting factor then
>> checkout could be significantly slower.
>
> When I upgraded a 1.6.x working copy of 2GB size, it took more than one hour.
> I eventually just aborted the upgrade. A checkout completes within
> a couple of minutes.
>
> The repository is remote and the bottleneck should be my 16MB/s downstream
> link (the server is connected to my ISP via the berlin internet exchange
> at 100Mbit/s). So based on what you're saying I should be seeing a much
> better upgrade vs. checkout ratio.
>
> Link to the repository: http://svn.dslinux.org/svn/trunk
>
> svn also seemed to spend more and more time hogging the CPU as the upgrade
> progressed. Note: I have not profiled this, it is all based on what I
> perceived while waiting for the upgrade to complete, and I kept box getting
> more and more impatiant. I'd be willing to repeat this experiment with a
> profiled build.
I don't know why that upgrade is slow, but perhaps you are limited by
disk IO. Maybe upgrade doesn't scale properly.
It's wrong to claim that checkout is faster than upgrade
unconditionally. A checkout of Subversion trunk from Apache takes about
10s on my machine, an upgrade takes 5s. That's with a hot cache, with a
cold cache upgrade takes about 15s. The checkout speed varies,
sometimes it takes ove 20s. Which one is faster?
--
Philip
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:
> When I upgraded a 1.6.x working copy of 2GB size, it took more than one hour.
> I eventually just aborted the upgrade. A checkout completes within
> a couple of minutes.
>
> The repository is remote and the bottleneck should be my 16MB/s downstream
> link (the server is connected to my ISP via the berlin internet exchange
> at 100Mbit/s). So based on what you're saying I should be seeing a much
> better upgrade vs. checkout ratio.
>
> Link to the repository: http://svn.dslinux.org/svn/trunk
>
> svn also seemed to spend more and more time hogging the CPU as the upgrade
> progressed. Note: I have not profiled this, it is all based on what I
> perceived while waiting for the upgrade to complete, and I kept box getting
> more and more impatiant. I'd be willing to repeat this experiment with a
> pr filed build.
Moving from IRC to dev.
We fixed some SQL queries that were not using indices, so upgrade now
scales much better but it is still not as fast as checkout. On my
machine:
1.6 checkout
real 4m49.977s
user 1m41.086s
sys 0m59.368s
1.6-1.7 upgrade
real 16m12.824s
user 6m17.728s
sys 2m4.960s
1.7 checkout
real 7m47.625s
user 1m17.125s
sys 1m15.113s
I can improve the speed further by increasing the SQLite cache size from
2,000 to 200,000 pages:
1.6-1.7 upgrade cache x100
real 9m41.934s
user 5m40.817s
sys 0m34.730s
This cache increase has only a small effect on checkout:
1.7 checkout cache x100
real 7m29.400s
user 1m13.689s
sys 1m17.029s
and that might be down to my connection. The cache's different effect
on upgrade/checkout may be because upgrade is a single, large
transaction while checkout is a series of small transactions.
I suspect that part of the performance problem for upgrade is that it
involves reads and writes to the same disk. Just copying using cp(1)
the pristine directory takes minutes on this machine. By contrast
checkout is mostly just writes. A RAID array to improve upgrade, or a
slower network connection to degrade checkout, would probably alter the
balance.
Running "svn status" on this large (2.4G in 1.6, 2.0G in 1.7) working
copy shows that 1.7 is faster than 1.6 with a cold cache (50s to 80s)
but slower with a hot cache (1.8s to 1.4s).
--
Philip
Re: svn commit: r1139766 - /subversion/trunk/subversion/svn/main.c
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Jun 27, 2011 at 11:34:13AM +0100, Philip Martin wrote:
> stsp@apache.org writes:
>
> > Author: stsp
> > Date: Sun Jun 26 12:28:28 2011
> > New Revision: 1139766
> >
> > URL: http://svn.apache.org/viewvc?rev=1139766&view=rev
> > Log:
> > * subversion/svn/main.c
> > (svn_cl__cmd_table): In the help text for 'svn upgrade', explain that
> > the upgrade preserves local modifications, and that an upgrade from
> > 1.6 to 1.7 takes more time than a new checkout.
> >
> > Modified:
> > subversion/trunk/subversion/svn/main.c
> >
> > Modified: subversion/trunk/subversion/svn/main.c
> > URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/svn/main.c?rev=1139766&r1=1139765&r2=1139766&view=diff
> > ==============================================================================
> > --- subversion/trunk/subversion/svn/main.c (original)
> > +++ subversion/trunk/subversion/svn/main.c Sun Jun 26 12:28:28 2011
> > @@ -1380,7 +1380,13 @@ const svn_opt_subcommand_desc2_t svn_cl_
> >
> > { "upgrade", svn_cl__upgrade, {0}, N_
> > ("Upgrade the metadata storage format for a working copy.\n"
> > - "usage: upgrade WCPATH...\n"),
> > + "usage: upgrade WCPATH...\n"
> > + "\n"
> > + " Local modifications are preserved.\n"
> > + "\n"
> > + " Note: Upgrading a working copy from the format used in Subversion 1.6\n"
> > + " to the format used in Subversion 1.7 takes much more time than checking\n"
> > + " out a new working copy with the 1.7 client.\n"),
>
> I disagree.
>
> Upgrade requires more disk IO but requires significantly less network IO
> and uses marginally less CPU. Checkout is likely to be faster when disk
> IO is the limiting factor, but if network IO is the limiting factor then
> checkout could be significantly slower.
When I upgraded a 1.6.x working copy of 2GB size, it took more than one hour.
I eventually just aborted the upgrade. A checkout completes within
a couple of minutes.
The repository is remote and the bottleneck should be my 16MB/s downstream
link (the server is connected to my ISP via the berlin internet exchange
at 100Mbit/s). So based on what you're saying I should be seeing a much
better upgrade vs. checkout ratio.
Link to the repository: http://svn.dslinux.org/svn/trunk
svn also seemed to spend more and more time hogging the CPU as the upgrade
progressed. Note: I have not profiled this, it is all based on what I
perceived while waiting for the upgrade to complete, and I kept box getting
more and more impatiant. I'd be willing to repeat this experiment with a
profiled build.