You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ross Mark <ro...@controllingedge.com.au> on 2005/01/04 03:07:33 UTC
[PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
I spoke of this patch to speed up the use of svn:externals last year
when 1.1 was being readied for release. Being the slack person I am I
only just got around to porting it from 1.0 to 1.2.
The idea for the patch is very simple. The original svn code would
delete any existing working copy for an external item if its url didn't
match the entry in svn:externals. This meant that updates were very
inefficient for externals if the url changed. ie if you had the
svn:externals entry 'td file:///rep/tags/ver1' and the url was changed
to ...tags/ver2 the entire td directory would have been deleted and
checked out again.
I've also attached a test script that demonstrates the problem, plus I
used it to verify my changes. The script creates a repository and a
working copy. A file is created that is then copied like a tag. The file
is changed and a new copy is made. An svn:externals entry is made that
references the first copy. It is then changed to reference
the second copy. Without the patch the changes between the first copy
and the second copy would be seen as a new checkout. With the patch it
is seen as an update.
Here is the relevant output from the script without the patch applied.
Running an update to checkout externals/ver1 as td
Fetching external item into 'trunk/lib/td'
A trunk/lib/td/dir1
A trunk/lib/td/dir1/file1
Updated external to revision 5.
Updated to revision 5.
Changing svn:externals so td is externals/ver2
property 'svn:externals' set on 'trunk/lib'
Running an update to checkout externals/ver2 as td
Fetching external item into 'trunk/lib/td'
A trunk/lib/td/dir1
A trunk/lib/td/dir1/file1
Updated external to revision 6.
Updated to revision 6.
Here is the same script but with a patched version of svn.
Running an update to checkout externals/ver1 as td
Fetching external item into 'trunk/lib/td'
A trunk/lib/td/dir1
A trunk/lib/td/dir1/file1
Updated external to revision 5.
Updated to revision 5.
Changing svn:externals so td is externals/ver2
property 'svn:externals' set on 'trunk/lib'
Running an update to checkout externals/ver2 as td
Fetching external item into 'trunk/lib/td'
U trunk/lib/td/dir1/file1
Updated external to revision 6.
Updated to revision 6.
The difference you see is that it only required an update to get to
revision 6 rather than a new checkout.
Log Message:
If the url of an svn:externals entry changed attempt to do a switch with
the new url
and only if that fails do we try and delete the existing wc and checkout
the new.
* subversion/libsvn_client/externals.c
(handle_external_item_change) Before calling relegate_external try
and perform a switch using the new url. If that fails then call
relegate_external.
Re: [PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
Posted by Ross Mark <ro...@controllingedge.com.au>.
Ross Mark wrote:
> Philip Martin wrote:
>
>> Ross Mark <ro...@controllingedge.com.au> writes:
>>
>> I like the basic idea. Have you run the regression tests with this
>> patch applied?
>>
>>
>
> I'm running them now. Normally I interrupted the regression test once
> the externals_test had be run.
>
And they all passed with success.
Ross
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal
"svn switch" rather than a straight delete.
Posted by Philip Martin <ph...@codematters.co.uk>.
Ross Mark <ro...@controllingedge.com.au> writes:
> Philip Martin wrote:
>
>>What's the reason for calling svn_client_cleanup? Given that cleanup
>>will steal WC locks it's probably not the right thing to do.
>>
> The problem is, if the new url for the externals entry points to a
> different repository the attempted switch obtains the locks then bails
> out leaving the locks once it realise that the repositories are
> different. This is the one part of the process that I didn't like and
> am happy to take any suggestions.
I see, that happens because a failed switch doesn't explicitly close
the access baton leaving pool cleanup to remove unnecessary locks.
How about something like:
/* Note: using a subpool to cleanup WC locks if the switch fails */
apr_pool_t *subpool = svn_pool_create (pool)
err = svn_client_switch (... subpool);
svn_pool_destroy (subpool);
if (err)
{
// No call to svn_client_cleanup required
}
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
Posted by Ross Mark <ro...@controllingedge.com.au>.
Philip Martin wrote:
>Ross Mark <ro...@controllingedge.com.au> writes:
>
>I like the basic idea. Have you run the regression tests with this
>patch applied?
>
>
I'm running them now. Normally I interrupted the regression test once
the externals_test had be run.
>
>
>>+ /* Try doing a switch first */
>>+ err = svn_client_switch (NULL,
>>+ path,
>>+ new_item->url,
>>+ &(new_item->revision),
>>+ TRUE,
>>+ ib->ctx,
>>+ ib->pool);
>>+ if (err)
>>+ {
>>+ /* The switch failed so do a cleanup and delete the old*/
>>+ svn_error_clear (err);
>>+ err = svn_client_cleanup(path, ib->ctx, ib->pool);
>>
>>
>
>What's the reason for calling svn_client_cleanup? Given that cleanup
>will steal WC locks it's probably not the right thing to do.
>
>
>
The problem is, if the new url for the externals entry points to a
different repository the attempted switch obtains the locks then bails
out leaving the locks once it realise that the repositories are
different. This is the one part of the process that I didn't like and am
happy to take any suggestions.
>>+ if (err)
>>+ svn_error_clear(err);
>>
>>
>
>Hmm, that ignores all errors, is that the right thing to do or should
>it only be certain specific errors? I suppose one could argue that
>the original code would always do the relegate/checkout stuff and so
>all errors is OK. I'm not sure what's best here.
>
>
>
I took the attitude that the original code just deleted everything so
the end result is the same.
Ross
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal
"svn switch" rather than a straight delete.
Posted by Philip Martin <ph...@codematters.co.uk>.
Ross Mark <ro...@controllingedge.com.au> writes:
I like the basic idea. Have you run the regression tests with this
patch applied?
> --- subversion/libsvn_client/externals.c (revision 12580)
> +++ subversion/libsvn_client/externals.c (working copy)
> @@ -285,15 +285,7 @@
> }
> else if (! compare_external_items (new_item, old_item))
> {
> - /* ### Better yet, compare_external_items should report the
> - nature of the difference. That way, when it's just a change
> - in the "-r REV" portion, for example, we could do an update
> - here instead of a relegation followed by full checkout. */
> -
> - SVN_ERR (relegate_external (path,
> - ib->ctx->cancel_func,
> - ib->ctx->cancel_baton,
> - ib->pool));
> + svn_error_t *err;
>
> /* First notify that we're about to handle an external. */
> if (ib->ctx->notify_func)
> @@ -306,12 +298,41 @@
> svn_wc_notify_state_unknown,
> SVN_INVALID_REVNUM);
>
> - SVN_ERR (svn_client__checkout_internal (NULL, new_item->url, path,
> - &(new_item->revision),
> - &(new_item->revision),
> - TRUE, /* recurse */
> - ib->timestamp_sleep,
> - ib->ctx, ib->pool));
> + /* Try doing a switch first */
> + err = svn_client_switch (NULL,
> + path,
> + new_item->url,
> + &(new_item->revision),
> + TRUE,
> + ib->ctx,
> + ib->pool);
> + if (err)
> + {
> + /* The switch failed so do a cleanup and delete the old*/
> + svn_error_clear (err);
> + err = svn_client_cleanup(path, ib->ctx, ib->pool);
What's the reason for calling svn_client_cleanup? Given that cleanup
will steal WC locks it's probably not the right thing to do.
> + if (err)
> + svn_error_clear(err);
Hmm, that ignores all errors, is that the right thing to do or should
it only be certain specific errors? I suppose one could argue that
the original code would always do the relegate/checkout stuff and so
all errors is OK. I'm not sure what's best here.
> +
> + /* ### Better yet, compare_external_items should report the
> + nature of the difference. That way, when it's just a change
> + in the "-r REV" portion, for example, we could do an update
> + here instead of a relegation followed by full checkout. */
> +
> + SVN_ERR (relegate_external (path,
> + ib->ctx->cancel_func,
> + ib->ctx->cancel_baton,
> + ib->pool));
> +
> +
> +
> + SVN_ERR (svn_client__checkout_internal (NULL, new_item->url, path,
> + &(new_item->revision),
> + &(new_item->revision),
> + TRUE, /* recurse */
> + ib->timestamp_sleep,
> + ib->ctx, ib->pool));
> + }
> }
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
Posted by Ross Mark <ro...@controllingedge.com.au>.
Thanks for that Michael.
I haven't forgotten about it either just I ran out of time to work on
it. Work commitments flooded me plus we have just moved into a new house
so spare time is scarce. I hope to resubmit the patch soon with the
switch_internal function.
Ross
Michael W Thelen wrote:
> Philip Martin wrote:
>
>> Ross Mark <ro...@controllingedge.com.au> writes:
>>
>>> Is it worth submitting the patch as is knowing there will be an
>>> extra delay or should I include in the patch your suggested change
>>> for svn_client__switch_internal?
>>
>>
>> I'd be reluctant to apply it without __switch_internal.
>
>
> So it's not forgotten, I've filed the original patch as issue #2209:
> http://subversion.tigris.org/issues/show_bug.cgi?id=2209
>
> --
> Michael W Thelen
> It is a mistake to think you can solve any major problems just with
> potatoes. -- Douglas Adams
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
Posted by Michael W Thelen <mi...@pietdepsi.com>.
Philip Martin wrote:
> Ross Mark <ro...@controllingedge.com.au> writes:
>
>
>>Is it worth submitting the patch as is knowing there will be an
>>extra delay or should I include in the patch your suggested change
>>for svn_client__switch_internal?
>
>
> I'd be reluctant to apply it without __switch_internal.
So it's not forgotten, I've filed the original patch as issue #2209:
http://subversion.tigris.org/issues/show_bug.cgi?id=2209
--
Michael W Thelen
It is a mistake to think you can solve any major problems just with
potatoes. -- Douglas Adams
Re: [PATCH] Speed up svn:externals updates by using an internal
"svn switch" rather than a straight delete.
Posted by Philip Martin <ph...@codematters.co.uk>.
Ross Mark <ro...@controllingedge.com.au> writes:
> Is it worth submitting the patch as is knowing there will be an
> extra delay or should I include in the patch your suggested change
> for svn_client__switch_internal?
I'd be reluctant to apply it without __switch_internal.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal "svn
switch" rather than a straight delete.
Posted by Ross Mark <ro...@controllingedge.com.au>.
Philip Martin wrote:
>Ross Mark <ro...@controllingedge.com.au> writes:
>
>
>
>>- SVN_ERR (svn_client__checkout_internal (NULL, new_item->url, path,
>>- &(new_item->revision),
>>- &(new_item->revision),
>>- TRUE, /* recurse */
>>- ib->timestamp_sleep,
>>- ib->ctx, ib->pool));
>>+ /* Try doing a switch first */
>>+ err = svn_client_switch (NULL,
>>+ path,
>>+ new_item->url,
>>+ &(new_item->revision),
>>+ TRUE,
>>+ ib->ctx,
>>+ ib->pool);
>>
>>
>
>Notice that while __checkout_internal() is used instead of _checkout()
>the new code uses _switch() directly. That means that there will be a
>svn_sleep_for_timestamps() for each external, and one of the reasons
>__checkout_internal() was created was to avoid exactly that problem.
>It looks like a __switch_internal() is needed.
>
>
>
I was just about to submit a new patch that uses the subpool and avoids
the locking problems, good thing those regression tests take awhile to
run. Is it worth submitting the patch as is knowing there will be an
extra delay or should I include in the patch your suggested change for
svn_client__switch_internal?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Speed up svn:externals updates by using an internal
"svn switch" rather than a straight delete.
Posted by Philip Martin <ph...@codematters.co.uk>.
Ross Mark <ro...@controllingedge.com.au> writes:
> - SVN_ERR (svn_client__checkout_internal (NULL, new_item->url, path,
> - &(new_item->revision),
> - &(new_item->revision),
> - TRUE, /* recurse */
> - ib->timestamp_sleep,
> - ib->ctx, ib->pool));
> + /* Try doing a switch first */
> + err = svn_client_switch (NULL,
> + path,
> + new_item->url,
> + &(new_item->revision),
> + TRUE,
> + ib->ctx,
> + ib->pool);
Notice that while __checkout_internal() is used instead of _checkout()
the new code uses _switch() directly. That means that there will be a
svn_sleep_for_timestamps() for each external, and one of the reasons
__checkout_internal() was created was to avoid exactly that problem.
It looks like a __switch_internal() is needed.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org