You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Julian Foad <ju...@apache.org> on 2022/02/01 13:57:42 UTC
Multi-WC-format branch: preparing for merge to trunk
We need multi-wc-format for issue #525 pristines-on-demand, as discussed
in the long thread "A two-part vision for Subversion and large binary objects."
It is designed to be a generic framework, not tied to a specific
feature, so while I will do some tests on a local merge of
pristines-on-demand on top of multi-wc-format, we should merge them to
trunk separately.
This is a heads-up that I'm going to look at getting this ready for
merge to trunk over the next few days.
From the last report I read, and a quick examination recently, it looks
like there is nothing but minor fix-ups needed. That's my hope.
Tests:
One possible substantial thing we might need is regression tests. From
my understanding of what this feature does, however, there is not much
to test. Basically, given:
- a WC where (format >= SVN_WC__SUPPORTED_VERSION
and format < SVN_WC__VERSION)
if:
- Subversion works the same as (format == SVN_WC__VERSION); and
- Subversion neither upgrades the WC nor reports that an upgrade it needed;
that is all we need to know.
There are some special cases such as "svn upgrade" tests.
For all other cases, with "white box" knowledge (knowing how this code
works) we can write a single test for this case and be fairly well
convinced it will work everywhere. It doesn't seem necessary to automate
running over the whole test suite with (format < SVN_WC__VERSION). It
may be useful for me to make a way to run the test suite manually with
that condition, in order to check for special cases.
And I wrote earlier (January 14) "the 'compatibility' we're speaking
of here is just whether the client accepts the format number as one that
it can work with, and if not then it errors out with the "client too
old" error..."
So we may need an additional test for the "WC is too old" case, if
adjusting the existing tests doesn't seem sufficient.
Does that sound about right?
Anyone who has an interest in this, please join in.
- Julian
Issue #4883: Multi-WC-format
Posted by Julian Foad <ju...@foad.me.uk>.
This feature is now filed as
https://subversion.apache.org/issue/4883 "multi-wc-format"
Re. the four enhancement suggestions I listed earlier in this thread:
> - The "svn upgrade" command may grow an option to only show the
> current and target working copy versions.
Added as a comment to #4885 "multi-wc-format: WC upgraded and
not-upgraded notifications".
> - Let the default "compatible version" be specified in user config.
Filed as:
https://subversion.apache.org/issue/4886 "multi-wc-format: config for
default WC version for checkout & upgrade"
> - Clarify/unify the option name (--compatible-version/--wc-format-version/...)
Filed as:
https://subversion.apache.org/issue/4887 "multi-wc-format: clarify/unify
option names for compatible-version"
> - Clarify the supported versions display (in svn --version)
Improved in https://svn.apache.org/r1898378, along with necessary API changes:
Old display in 'svn --version':
| Supported working copy (WC) versions: from 1.8 to 1.15
New display in 'svn --version':
| Supported working copy (WC) formats:
|
| * compatible with Subversion v1.8 to v1.15 (WC format 31)
| * compatible with Subversion v1.15 (WC format 32)
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@foad.me.uk>.
Merged to trunk now. Testers, enhancers welcome. More below.
Please try it out if you haven't.
The basics:
wc format 31:
- compatible with svn 1.8 to 1.14 and now 1.15-dev
wc format 32:
- compatible with svn 1.15-dev only,
- as yet introduces no schema changes (we expect those to come when we
merge 'pristines-on-demand-on-mwf' branch to trunk)
There is no supported downgrade, so be careful with testing this. In
this development state you might not want to upgrade an important wc to
compatible-version=1.15. Check out a disposable WC instead.
(In emergency you might get away with:
sqlite3 .svn/wc.db 'pragma user_version=31;'
)
Commands affected:
svn checkout # WC is compatible with 1.8 to 1.14
svn checkout --compatible-version=1.15 # format bump
svn upgrade # upgrade an old (svn<=1.7) WC to f31 (1.8~1.14)
svn upgrade --compatible-version=1.15 # upgrade to 1.15
Anyone would be very welcome to take a crack at any suggested
enhancements, including but not limited to those I listed earlier:
> - The "svn upgrade" command may grow an option to only show the current
> and target working copy versions. (Perhaps `svn upgrade --dry-run`?)
>
> - Let the default "compatible version" be specified in the user
> config. (That is, the default version for 'checkout' and 'upgrade',
> which are the places where a version can be specified.)
>
> - Clarify/unify the option name. svn commands also have other
> compatibility concerns besides the WC format.
> Current option names:
> --compatible-version # for 'svnadmin': repository version
> --compatible-version # for 'svn': WC version
> --wc-format-version # for the tests: WC version
> Suggestions for WC version everywhere ('svn' and tests):
> --wc-compatible-version
> --wc-compatible-format
>
> - Clarify the supported versions display:
> -Supported working copy (WC) versions: from 1.8 to 1.15
> +Supported working copy (WC) versions:
> + --compatible-version=1.8 (supported by svn 1.8 to 1.15; default)
> + --compatible-version=1.15 (supported by svn 1.15)
>
> We might want some of these, or other changes, before release.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@apache.org>.
TL;DR: OK to merge multi-wc-format to trunk soon?
Having worked with it these last few weeks, I think the essential pieces
are in place for multi-wc-format to be merged to trunk.
I am still working on the last pieces of pristines-on-demand rebased on
top of this branch: fixing tests to work properly with both WC versions.
What I am seeing there is all specific to the pristines-on-demand
changes, and doesn't seem to be requiring anything extra of the basic
multi-wc-format capabilities. That suggests this multi-wc-format branch
provides what it needs to provide.
This is my list of possible enhancements to multi-wc-format:
- The "svn upgrade" command may grow an option to only show the current
and target working copy versions. (Perhaps `svn upgrade --dry-run`?)
- Let the default "compatible version" be specified in the user
config. (That is, the default version for 'checkout' and 'upgrade',
which are the places where a version can be specified.)
- Clarify/unify the option name. svn commands also have other
compatibility concerns besides the WC format.
Current option names:
--compatible-version # for 'svnadmin': repository version
--compatible-version # for 'svn': WC version
--wc-format-version # for the tests: WC version
Suggestions for WC version everywhere ('svn' and tests):
--wc-compatible-version
--wc-compatible-format
- Clarify the supported versions display:
-Supported working copy (WC) versions: from 1.8 to 1.15
+Supported working copy (WC) versions:
+ --compatible-version=1.8 (supported by svn 1.8 to 1.15; default)
+ --compatible-version=1.15 (supported by svn 1.15)
We might want some of these, or other changes, before release. I don't
see any of them being needed before merge to trunk. The criterion is
basically, as much as possible should be done on trunk, as I understand
it. Please speak up about any changes needed, and if anything is needed
before merge.
Does anyone want extra time to review the branch, or have any other
concerns prior to merging it? If not I will do some further review of my
own (double-checking API changes and test changes) and then I should
think I could go ahead and merge it in a few days. I'm not meaning to be
pushy, so do let me know if anything doesn't seem right.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@foad.me.uk>.
r1897943 makes the default for "--compatible-version" (and tests) be "1.8".
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Feb 10, 2022 at 7:25 AM Julian Foad <ju...@foad.me.uk> wrote:
>
> It strikes me now, given Subversion's "mature, stable" phase of life,
> that we should default to backward compatibility.
>
> Change the default to --compatible-version=1.8 (same as =1.14), instead
> of 1.15 (format 32). Both for "checkout" and "upgrade". The user will
> need to specify "--compatible-version=1.15" when they want to use
> new-in-1.15-format features (i.e. pristines-on-demand).
>
> We should let this option be specified in the user config
> (file/registry) too.
>
> Makes sense?
Yes, +1
Thanks for considering it.
Mark
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@foad.me.uk>.
It strikes me now, given Subversion's "mature, stable" phase of life,
that we should default to backward compatibility.
Change the default to --compatible-version=1.8 (same as =1.14), instead
of 1.15 (format 32). Both for "checkout" and "upgrade". The user will
need to specify "--compatible-version=1.15" when they want to use
new-in-1.15-format features (i.e. pristines-on-demand).
We should let this option be specified in the user config
(file/registry) too.
Makes sense?
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@apache.org>.
> I think this branch is now ready to merge to trunk.
I am now asking for review.
To help, here is an overview of the branch changes, compiled from its
log messages:
------------------------------------------------------------------------
Instead of supporting just one format, introduce a current format
(the default for new working copies) and a lowest supported format.
Change the way new working copies are created: instead of the base
schema defining the current format, it defines the lowest supporting
format and a series of format updates are performed to bring it to the
current shape.
Add the --compatible-version option to the checkout and upgrade
commands.
Guess the format of dependent working copies (e.g., for externals) from
the current working copy context.
Add a line to the 'svn --version' output that shows supported WC versions.
Include a place-holder format bump. This makes it easier to merge with a
branch that includes a format bump (such as the 'pristines-on-demand'
branch).
Add test suite option '--wc-format-version'. This accepts a version
string like '1.14.0' and is used for all WC creation, e.g. 'svn
checkout/upgrade --compatible-version=X'.
------------------------------------------------------------------------
Highlights of API-like additions and changes:
* subversion/include/svn_client.h
(svn_client_checkout4,
svn_client_upgrade2): New versions.
(svn_client_supported_wc_version): New.
* subversion/include/svn_wc.h
(svn_wc_info_t): Add new fields: wc_format, wc_format_min, wc_format_max.
* subversion/include/private/svn_wc_private.h
(svn_wc__format_from_version
svn_wc__is_supported_format
svn_wc__max_supported_format
svn_wc__min_supported_format
svn_wc__is_supported_format_version
svn_wc__max_supported_format_version
svn_wc__min_supported_format_version
svn_wc__format_from_context): New.
(svn_wc__ensure_adm): New, replaces svn_wc_ensure_adm4.
(svn_wc__upgrade): New.
* subversion/libsvn_wc/wc-metadata.sql
(STMT_CREATE_SCHEMA): Set 'user_version' to the minimum supported
version (SVN_WC__SUPPORTED_VERSION), not the latest current version.
------------------------------------------------------------------------
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@apache.org>.
I think this branch is now ready to merge to trunk.
In a series of commits I have added '--wc-format-version' argument to
the C and Python test suite framework on this branch, and tested that it
works with both '1.15' (the default, WC format 32) and '1.14' (wc format 31).
I tested that the desired format is actually being used in WCs when this
option is passed in to the test suite, using this temporary code:
```
Index: subversion/libsvn_wc/wc_db.c
===================================================================
--- subversion/libsvn_wc/wc_db.c (revision 1897914)
+++ subversion/libsvn_wc/wc_db.c (working copy)
@@ -1508,2 +1508,9 @@ svn_wc__db_init(svn_wc__db_t *db,
+#define FORMAT_EXPECTED 31
+ if (wcroot->format != FORMAT_EXPECTED)
+ {
+ SVN_DBG(("### format is %d; expected %d", wcroot->format, FORMAT_EXPECTED));
+ SVN_ERR_MALFUNCTION_NO_RETURN();
+ }
+
/* Any previously cached children may now have a new WCROOT, most
likely that
@@ -13464,2 +13471,8 @@ svn_wc__db_upgrade_begin(svn_sqlite__db_
+ if (wcroot->format != FORMAT_EXPECTED)
+ {
+ SVN_DBG(("### format is %d; expected %d", wcroot->format, FORMAT_EXPECTED));
+ SVN_ERR_MALFUNCTION_NO_RETURN();
+ }
+
/* The WCROOT is complete. Stash it into DB. */
```
These checks occur after create_db() calls. I think they cover most if
not all places where a WC can be created.
That would seem to be everything we need on this branch.
Any further thoughts about readiness to merge to trunk, Brane or anyone?
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Branko Čibej <br...@apache.org>.
On 03.02.2022 17:30, Julian Foad wrote:
> Julian Foad wrote:
>> Branko Čibej wrote:
>>> way to report the actual format (from the wc-db) through libsvn-wc to
>>> the client?
>>
>> Ah yes, there is no API for that, [...]
> These exist on the branch currently ("*_supported_*" are new on the branch):
>
> - At libsvn_client (public):
> svn_client_version() // e.g. svn_version_t("1.15.0-dev")
> svn_client_supported_wc_version() // e.g. svn_version_t("1.8.0")
>
> - In libsvn_wc (internal declarations):
> SVN_WC__VERSION // e.g. "32" for svn 1.15-dev
> SVN_WC__SUPPORTED_VERSION // e.g. "31" for svn 1.8 through 1.14
>
> - Conversion between WC format number and svn_version_t:
> svn_wc__format_from_version() // exposed in svn_wc_private.h
> svn_wc__version_string_from_format() // internal only
>
> I suggest:
>
> - Keep the SVN_WC__* constants internal, as they are.
>
> - Expose *both* conversion functions equally as private APIs. No reason
> not to, and I think the new --wc-format option in the test suite is
> going to need them. (I could wait and see if it does; I'm writing it
> now.) One already is in svn_wc_private.h. The other says:
>
> * ### It's not ideal to encode this sort of knowledge in this low-level
> * library. On the other hand, it doesn't need to be updated often and
> * should be easily found when it does need to be updated. */
>
> which applies equally to both and is fine for private APIs.
>
> - Expose the min and max WC format numbers privately in
> svn_wc_private.h. It seems perverse not to, when higher level APIs
> exist. Code outside libsvn_wc, such as wc tests, will need or want
> these. (I can wait and see if it does. I'm writing it soon.)
>
> svn_wc__format() // e.g. "32" for svn 1.15-dev
> svn_wc__supported_format() // e.g. "31" for svn 1.8 through 1.14
Good points. I wanted to avoid widening the libsvn_wc API, but, indeed,
that train is long gone.
> There is no libsvn_client API for getting the WC format number directly.
> I suppose it makes sense that client-level code is written with
> reference to client-level version numbers.
Exactly. libsvn_client needs to know, but its users do not.
-- Brane
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@apache.org>.
Julian Foad wrote:
> Branko Čibej wrote:
>> way to report the actual format (from the wc-db) through libsvn-wc to
>> the client?
>
> Ah yes, there is no API for that, [...]
These exist on the branch currently ("*_supported_*" are new on the branch):
- At libsvn_client (public):
svn_client_version() // e.g. svn_version_t("1.15.0-dev")
svn_client_supported_wc_version() // e.g. svn_version_t("1.8.0")
- In libsvn_wc (internal declarations):
SVN_WC__VERSION // e.g. "32" for svn 1.15-dev
SVN_WC__SUPPORTED_VERSION // e.g. "31" for svn 1.8 through 1.14
- Conversion between WC format number and svn_version_t:
svn_wc__format_from_version() // exposed in svn_wc_private.h
svn_wc__version_string_from_format() // internal only
I suggest:
- Keep the SVN_WC__* constants internal, as they are.
- Expose *both* conversion functions equally as private APIs. No reason
not to, and I think the new --wc-format option in the test suite is
going to need them. (I could wait and see if it does; I'm writing it
now.) One already is in svn_wc_private.h. The other says:
* ### It's not ideal to encode this sort of knowledge in this low-level
* library. On the other hand, it doesn't need to be updated often and
* should be easily found when it does need to be updated. */
which applies equally to both and is fine for private APIs.
- Expose the min and max WC format numbers privately in
svn_wc_private.h. It seems perverse not to, when higher level APIs
exist. Code outside libsvn_wc, such as wc tests, will need or want
these. (I can wait and see if it does. I'm writing it soon.)
svn_wc__format() // e.g. "32" for svn 1.15-dev
svn_wc__supported_format() // e.g. "31" for svn 1.8 through 1.14
There is no libsvn_client API for getting the WC format number directly.
I suppose it makes sense that client-level code is written with
reference to client-level version numbers.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@apache.org>.
Branko Čibej wrote:
> I still seem to remember we have a bit missing on the multi-wc-format
> branch, but I can't remember what ... maybe we don't currently have a
> way to report the actual format (from the wc-db) through libsvn-wc to
> the client? Or has that been fixed?
Ah yes, there is no API for that, and any new feature built on this will
need to know, in general. (The current state of pristines-on-demand
doesn't, but maybe it will by the time it's finished.)
I will add it.
- Julian
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Branko Čibej <br...@apache.org>.
On 02.02.2022 20:49, Julian Foad wrote:
>> [...] a single test [...] It doesn't seem necessary to automate
>> running over the whole test suite with (format < SVN_WC__VERSION) [...]
> Meh. That's only true for the place-holder code as present on the
> 'multi-wc-format' branch, where the code reports multiple formats are
> supported but there are no actual differences between the formats. Any
> non-trivial feature (such as pristines-on-demand) that works with
> multiple WC formats will have code paths that access database tables
> differently depending on the format. I ran into this as soon as I tested
> 'pristines-on-demand' merged with 'multi-wc-format'. Those code paths
> need to be tested against both (or multiple) formats.
>
> Therefore it IS necessary at least to add a parameter to the test suite
> framework to specify the WC format that all the tests will use. That
> should be done as part of this branch.
>
> The test suite already has...
>
> --server-minor-version=SERVER_MINOR_VERSION
> --fsfs-version=FSFS_VERSION
>
> I will copy that pattern.
Sounds good.
I still seem to remember we have a bit missing on the multi-wc-format
branch, but I can't remember what ... maybe we don't currently have a
way to report the actual format (from the wc-db) through libsvn-wc to
the client? Or has that been fixed?
-- Brane
Re: Multi-WC-format branch: preparing for merge to trunk
Posted by Julian Foad <ju...@foad.me.uk>.
> [...] a single test [...] It doesn't seem necessary to automate
> running over the whole test suite with (format < SVN_WC__VERSION) [...]
Meh. That's only true for the place-holder code as present on the
'multi-wc-format' branch, where the code reports multiple formats are
supported but there are no actual differences between the formats. Any
non-trivial feature (such as pristines-on-demand) that works with
multiple WC formats will have code paths that access database tables
differently depending on the format. I ran into this as soon as I tested
'pristines-on-demand' merged with 'multi-wc-format'. Those code paths
need to be tested against both (or multiple) formats.
Therefore it IS necessary at least to add a parameter to the test suite
framework to specify the WC format that all the tests will use. That
should be done as part of this branch.
The test suite already has...
--server-minor-version=SERVER_MINOR_VERSION
--fsfs-version=FSFS_VERSION
I will copy that pattern.
- Julian