You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Neels J Hofmeyr <ne...@elego.de> on 2011/10/06 22:28:58 UTC
EXTERNALS table -- good or bad?
Hi all,
I have committed code to trunk that populates the EXTERNALS table during
upgrade from 1.6. With a nudge from Philip, I noticed that the EXTERNALS
table is still a neither-here-nor-there implementation. Now I'm unsure.
This table is / would be very useful if one tries to find all externals
defined or checked out in a given subtree, without having to first find and
then parse all externals skels.
But in fact it is little more than a cache for svn:externals props. It
duplicates information in a way. But it adds the knowledge of exactly which
repository and relpath an external is from (stores URLs in repos-root and
repos-relpath, readily parsed from the various formats found in
svn:externals definitions).
So it is useful for speeding up things, but it also has a danger: heavens
forbid if the EXTERNALS table is ever out of sync with the svn:externals
props. So, we may often be forced to parse the props anyway (cleanup?).
What do you guys think about the EXTERNALS table? Yay or nay? What thought
has gone into this so far?
Your answer is interesting to me right now because it will change the fixes
for file-externals-under-unversioned (WIP) and it will either shorten or
obsolete this nomination:
[[[
* r1173574, r1174342, r1174693, r1174699, r1178910
Fix issue #4016: 'externals after upgrade from 1.6 have wrong URL
information in EXTERNALS rows'
Also: previously, the first failing external upgrade aborted all other
external upgrades, now just prints the error and carries on.
Justification:
Users will start upgrading their WCs soon, and it would be nice to
have proper information in wc.db. I know of no real problems when
EXTERNALS rows are wrong and/or missing info, but we might see them
soon enough, or hit problems if same users upgrade same WCs to 1.8,
much later.
Argument against: "If EXTERNALS is just some sort of cache perhaps
upgrade should just leave the columns null?"
Argument for: "Currently 1.7 populates those columns. So upgrade
should, too. We don't know which way we'll go -- drop EXTERNALS
columns or heavily rely on them. So rather make an upgraded 1.6 WC
look exactly the same as a fresh 1.7 WC would, saving us special
cases in subsequent upgrade code. Meaning: power users with age-old
WCs won't get dropped during upgrade to 1.8, whichever way we
choose."
Notes:
r1173574 is just a cosmetic change that avoids conflicts in r1174693.
r1174699 removes obsoleted stuff, but doesn't change the outcome.
r1178910 fixes upgrade_tests.py 2 on windows.
The entire patch seems quite large, but a large portion of it is just
moving a function from static to private API.
Conflicts:
--accept=tc
(r1174693, indent mismatch in handle_external_item_change() caused by
cosmetic fix from r1169770)
Votes:
+1: neels
]]]
Thanks,
~Neels
(What is it with externals, that they always seem to bring out ambivalent
half-hearted implementations that have everyone stumped?)
Re: EXTERNALS table -- good or bad?
Posted by Neels J Hofmeyr <ne...@elego.de>.
On 10/06/2011 11:39 PM, Alexey Neyman wrote:
> Hi Neels,
>
> Would this help 'svn commit' descend into external directories and check them
> in?
In fact, I have a patch in progress that addresses this. However, the
question I am asking in this thread goes deeper and is independent from how
externals are treated during commit. The fundamental issue, that commit
could not handle multiple working copies in one call, has been fixed, so it
is possible to advance here.
FYI, the current plans are: Subversion by default ignores all externals from
commit recursion (both file- and dir-externals); But, if --include-externals
is passed to 'svn commit', all externals are included in recursion. I.e. all
of file- and dir externals, be they nested below unversioned directories or
other working copies, will be included by a commit with --include-externals.
But I can say with certainty that this feature/change will not be released
before Subversion 1.8.0.
While this does not depend on an EXTERNALS table, it would run faster with
an EXTERNALS table. But maybe not that much faster as would justify a given
amount of caching complexity.
~Neels
Re: EXTERNALS table -- good or bad?
Posted by Alexey Neyman <st...@att.net>.
Hi Neels,
Would this help 'svn commit' descend into external directories and check them
in?
<<<
Perhaps most disappointingly, the working copies created via the externals
definition support are still disconnected from the primary working copy (on
whose versioned directories the svn:externals property was actually set). And
Subversion still truly operates only on nondisjoint working copies. So, for
example, if you want to commit changes that you've made in one or more of
those external working copies, you must run svn commit explicitly on those
working copies—committing on the primary working copy will not recurse into
any external ones.
<<<
This is a long-standing described limitation which becomes more and more of an
issue in our set-up...
Regards,
Alexey.
On Thursday, October 06, 2011 01:28:58 pm Neels J Hofmeyr wrote:
> Hi all,
>
> I have committed code to trunk that populates the EXTERNALS table during
> upgrade from 1.6. With a nudge from Philip, I noticed that the EXTERNALS
> table is still a neither-here-nor-there implementation. Now I'm unsure.
>
> This table is / would be very useful if one tries to find all externals
> defined or checked out in a given subtree, without having to first find and
> then parse all externals skels.
>
> But in fact it is little more than a cache for svn:externals props. It
> duplicates information in a way. But it adds the knowledge of exactly which
> repository and relpath an external is from (stores URLs in repos-root and
> repos-relpath, readily parsed from the various formats found in
> svn:externals definitions).
>
> So it is useful for speeding up things, but it also has a danger: heavens
> forbid if the EXTERNALS table is ever out of sync with the svn:externals
> props. So, we may often be forced to parse the props anyway (cleanup?).
>
> What do you guys think about the EXTERNALS table? Yay or nay? What thought
> has gone into this so far?
>
>
> Your answer is interesting to me right now because it will change the fixes
> for file-externals-under-unversioned (WIP) and it will either shorten or
> obsolete this nomination:
>
> [[[
> * r1173574, r1174342, r1174693, r1174699, r1178910
> Fix issue #4016: 'externals after upgrade from 1.6 have wrong URL
> information in EXTERNALS rows'
> Also: previously, the first failing external upgrade aborted all other
> external upgrades, now just prints the error and carries on.
> Justification:
> Users will start upgrading their WCs soon, and it would be nice to
> have proper information in wc.db. I know of no real problems when
> EXTERNALS rows are wrong and/or missing info, but we might see them
> soon enough, or hit problems if same users upgrade same WCs to 1.8,
> much later.
> Argument against: "If EXTERNALS is just some sort of cache perhaps
> upgrade should just leave the columns null?"
> Argument for: "Currently 1.7 populates those columns. So upgrade
> should, too. We don't know which way we'll go -- drop EXTERNALS
> columns or heavily rely on them. So rather make an upgraded 1.6 WC
> look exactly the same as a fresh 1.7 WC would, saving us special
> cases in subsequent upgrade code. Meaning: power users with age-old
> WCs won't get dropped during upgrade to 1.8, whichever way we
> choose."
> Notes:
> r1173574 is just a cosmetic change that avoids conflicts in r1174693.
> r1174699 removes obsoleted stuff, but doesn't change the outcome.
> r1178910 fixes upgrade_tests.py 2 on windows.
> The entire patch seems quite large, but a large portion of it is just
> moving a function from static to private API.
> Conflicts:
> --accept=tc
> (r1174693, indent mismatch in handle_external_item_change() caused by
> cosmetic fix from r1169770)
> Votes:
> +1: neels
> ]]]
>
> Thanks,
> ~Neels
>
> (What is it with externals, that they always seem to bring out ambivalent
> half-hearted implementations that have everyone stumped?)
Re: EXTERNALS table -- good or bad?
Posted by Branko Čibej <br...@xbc.nu>.
On 11.10.2011 10:01, Daniel Shahaf wrote:
> Branko Čibej wrote on Tue, Oct 11, 2011 at 09:47:31 +0200:
>> On 11.10.2011 05:06, Konstantin Kolinko wrote:
>>> 2011/10/10 Branko Čibej <br...@apache.org>:
>>>>>> Answer's simple: never store the svn:externals property with the other
>>>>>> props, always parse it into and read it from the EXTERNALS table. After
>>>>>> all, it /is/ a magic property. This can be a completely wc-local
>>>>>> implementation-specific thing.
>>>>> ...and change the interface for managing externals, too?
>>>>>
>>>>> Currently we set externals by setting some multiline string (propset). When
>>>>> coming back later (propget/edit), we expect that string to be exactly
>>>>> identical to what was set earlier -- even if it contains my seven favourite
>>>>> quotes of Mark Twain. Parsing that value to a table immediately and
>>>> bringing
>>>>> it back later discards of Twain's quotes. Those aren't a use case, but line
>>>>> comments in svn:externals and various URL notations are ("^/foo" and
>>>>> "svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
>>>>> row, i.e. same repos_id and def_repos_relpath).
>>>>>
>>>>> Ok, we could also store that plain string in the externals table instead of
>>>>> as a prop value. Does it really make a difference?
>>>>>
>>>>> Or we're talking about a complete feature change, not using the
>>>>> propset/propget API. Maybe something like
>>>>>
>>>>> svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
>>>>>
>>>>> Hmm, I think we better leave the prop as is, and have hooks that update the
>>>>> EXTERNALS table on every prop change (i.e. the way it is implemented right
>>>>> now, possible bugs aside).
>>>>> ...and try to find and somehow handle any abort conditions that might leave
>>>>> the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
>>>>> check/update on 'svn cleanup'.
>>>> I don't really see that a propset to svn:externals, followed immediately
>>>> by a propget, should yield exactly the same content, as long as it's
>>>> semantically identical. In other words, it'd be quite OK to treat such a
>>>> propset differently from other propsets, i.e., to parse whatever was set
>>>> into the EXTERNALS table, and have the propget read from there. It's not
>>>> as if we ever promised that magic properties are anything but magic. :)
>>> EXTERNALS table is in wc only, but propget can be called on repository URL.
>>>
>>> I think it would be a mess if results of calling propget on URL and on
>>> WC do not match.
>> The repository would only see pre-parsed svn:externals, so the results
>> would (eventually) match. As to it being a mess, I don't understand why.
>> After all, the idea is for the result to be semantially identical.
>> Besides, nobody complains about, e.g., svn:executable, where we already
>> munge the value.
>>
> svn:executable is binary, and we've munged it since pre-1.0. If we
> start now deleting comments in svn:externals, we'd (a) be implementing
> a data-loss feature ("Subversion silently deleted my Mark Twain quotes
> from my svn:externals property"), (b) removing the option to include
> comments in property values.
>
Hmm, I just caught myself discussing changes to svn:externals that
didn't involve removing the whole horrible feature. Must remember to be
more careful. :)
-- Brane
Re: EXTERNALS table -- good or bad?
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Branko Čibej wrote on Tue, Oct 11, 2011 at 09:47:31 +0200:
> On 11.10.2011 05:06, Konstantin Kolinko wrote:
> > 2011/10/10 Branko Čibej <br...@apache.org>:
> >>>> Answer's simple: never store the svn:externals property with the other
> >>>> props, always parse it into and read it from the EXTERNALS table. After
> >>>> all, it /is/ a magic property. This can be a completely wc-local
> >>>> implementation-specific thing.
> >>> ...and change the interface for managing externals, too?
> >>>
> >>> Currently we set externals by setting some multiline string (propset). When
> >>> coming back later (propget/edit), we expect that string to be exactly
> >>> identical to what was set earlier -- even if it contains my seven favourite
> >>> quotes of Mark Twain. Parsing that value to a table immediately and
> >> bringing
> >>> it back later discards of Twain's quotes. Those aren't a use case, but line
> >>> comments in svn:externals and various URL notations are ("^/foo" and
> >>> "svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
> >>> row, i.e. same repos_id and def_repos_relpath).
> >>>
> >>> Ok, we could also store that plain string in the externals table instead of
> >>> as a prop value. Does it really make a difference?
> >>>
> >>> Or we're talking about a complete feature change, not using the
> >>> propset/propget API. Maybe something like
> >>>
> >>> svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
> >>>
> >>> Hmm, I think we better leave the prop as is, and have hooks that update the
> >>> EXTERNALS table on every prop change (i.e. the way it is implemented right
> >>> now, possible bugs aside).
> >>> ...and try to find and somehow handle any abort conditions that might leave
> >>> the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
> >>> check/update on 'svn cleanup'.
> >> I don't really see that a propset to svn:externals, followed immediately
> >> by a propget, should yield exactly the same content, as long as it's
> >> semantically identical. In other words, it'd be quite OK to treat such a
> >> propset differently from other propsets, i.e., to parse whatever was set
> >> into the EXTERNALS table, and have the propget read from there. It's not
> >> as if we ever promised that magic properties are anything but magic. :)
> > EXTERNALS table is in wc only, but propget can be called on repository URL.
> >
> > I think it would be a mess if results of calling propget on URL and on
> > WC do not match.
>
> The repository would only see pre-parsed svn:externals, so the results
> would (eventually) match. As to it being a mess, I don't understand why.
> After all, the idea is for the result to be semantially identical.
> Besides, nobody complains about, e.g., svn:executable, where we already
> munge the value.
>
svn:executable is binary, and we've munged it since pre-1.0. If we
start now deleting comments in svn:externals, we'd (a) be implementing
a data-loss feature ("Subversion silently deleted my Mark Twain quotes
from my svn:externals property"), (b) removing the option to include
comments in property values.
> -- Brane
>
Re: EXTERNALS table -- good or bad?
Posted by Branko Čibej <br...@apache.org>.
On 11.10.2011 05:06, Konstantin Kolinko wrote:
> 2011/10/10 Branko Čibej <br...@apache.org>:
>>>> Answer's simple: never store the svn:externals property with the other
>>>> props, always parse it into and read it from the EXTERNALS table. After
>>>> all, it /is/ a magic property. This can be a completely wc-local
>>>> implementation-specific thing.
>>> ...and change the interface for managing externals, too?
>>>
>>> Currently we set externals by setting some multiline string (propset). When
>>> coming back later (propget/edit), we expect that string to be exactly
>>> identical to what was set earlier -- even if it contains my seven favourite
>>> quotes of Mark Twain. Parsing that value to a table immediately and
>> bringing
>>> it back later discards of Twain's quotes. Those aren't a use case, but line
>>> comments in svn:externals and various URL notations are ("^/foo" and
>>> "svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
>>> row, i.e. same repos_id and def_repos_relpath).
>>>
>>> Ok, we could also store that plain string in the externals table instead of
>>> as a prop value. Does it really make a difference?
>>>
>>> Or we're talking about a complete feature change, not using the
>>> propset/propget API. Maybe something like
>>>
>>> svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
>>>
>>> Hmm, I think we better leave the prop as is, and have hooks that update the
>>> EXTERNALS table on every prop change (i.e. the way it is implemented right
>>> now, possible bugs aside).
>>> ...and try to find and somehow handle any abort conditions that might leave
>>> the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
>>> check/update on 'svn cleanup'.
>> I don't really see that a propset to svn:externals, followed immediately
>> by a propget, should yield exactly the same content, as long as it's
>> semantically identical. In other words, it'd be quite OK to treat such a
>> propset differently from other propsets, i.e., to parse whatever was set
>> into the EXTERNALS table, and have the propget read from there. It's not
>> as if we ever promised that magic properties are anything but magic. :)
> EXTERNALS table is in wc only, but propget can be called on repository URL.
>
> I think it would be a mess if results of calling propget on URL and on
> WC do not match.
The repository would only see pre-parsed svn:externals, so the results
would (eventually) match. As to it being a mess, I don't understand why.
After all, the idea is for the result to be semantially identical.
Besides, nobody complains about, e.g., svn:executable, where we already
munge the value.
-- Brane
Re: EXTERNALS table -- good or bad?
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/10/10 Branko Čibej <br...@apache.org>:
>>>
>>> Answer's simple: never store the svn:externals property with the other
>>> props, always parse it into and read it from the EXTERNALS table. After
>>> all, it /is/ a magic property. This can be a completely wc-local
>>> implementation-specific thing.
>>
>> ...and change the interface for managing externals, too?
>>
>> Currently we set externals by setting some multiline string (propset). When
>> coming back later (propget/edit), we expect that string to be exactly
>> identical to what was set earlier -- even if it contains my seven favourite
>> quotes of Mark Twain. Parsing that value to a table immediately and
> bringing
>> it back later discards of Twain's quotes. Those aren't a use case, but line
>> comments in svn:externals and various URL notations are ("^/foo" and
>> "svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
>> row, i.e. same repos_id and def_repos_relpath).
>>
>> Ok, we could also store that plain string in the externals table instead of
>> as a prop value. Does it really make a difference?
>>
>> Or we're talking about a complete feature change, not using the
>> propset/propget API. Maybe something like
>>
>> svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
>>
>> Hmm, I think we better leave the prop as is, and have hooks that update the
>> EXTERNALS table on every prop change (i.e. the way it is implemented right
>> now, possible bugs aside).
>> ...and try to find and somehow handle any abort conditions that might leave
>> the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
>> check/update on 'svn cleanup'.
>
> I don't really see that a propset to svn:externals, followed immediately
> by a propget, should yield exactly the same content, as long as it's
> semantically identical. In other words, it'd be quite OK to treat such a
> propset differently from other propsets, i.e., to parse whatever was set
> into the EXTERNALS table, and have the propget read from there. It's not
> as if we ever promised that magic properties are anything but magic. :)
EXTERNALS table is in wc only, but propget can be called on repository URL.
I think it would be a mess if results of calling propget on URL and on
WC do not match.
Best regards,
Konstantin Kolinko
Re: EXTERNALS table -- good or bad?
Posted by Branko Čibej <br...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
On 10.10.2011 14:12, Neels J Hofmeyr wrote:
> On 10/08/2011 05:20 PM, Branko Čibej wrote:
>>
>> On 06.10.2011 22:28, Neels J Hofmeyr wrote:
>>> This table is / would be very useful if one tries to find all externals
>>> defined or checked out in a given subtree, without having to first
find and
>>> then parse all externals skels.
>>
>>> But in fact it is little more than a cache for svn:externals props. It
>>> duplicates information in a way. But it adds the knowledge of exactly
which
>>> repository and relpath an external is from (stores URLs in repos-root and
>>> repos-relpath, readily parsed from the various formats found in
>>> svn:externals definitions).
>>
>> Answer's simple: never store the svn:externals property with the other
>> props, always parse it into and read it from the EXTERNALS table. After
>> all, it /is/ a magic property. This can be a completely wc-local
>> implementation-specific thing.
>
> ...and change the interface for managing externals, too?
>
> Currently we set externals by setting some multiline string (propset). When
> coming back later (propget/edit), we expect that string to be exactly
> identical to what was set earlier -- even if it contains my seven favourite
> quotes of Mark Twain. Parsing that value to a table immediately and
bringing
> it back later discards of Twain's quotes. Those aren't a use case, but line
> comments in svn:externals and various URL notations are ("^/foo" and
> "svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
> row, i.e. same repos_id and def_repos_relpath).
>
> Ok, we could also store that plain string in the externals table instead of
> as a prop value. Does it really make a difference?
>
> Or we're talking about a complete feature change, not using the
> propset/propget API. Maybe something like
>
> svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
>
> Hmm, I think we better leave the prop as is, and have hooks that update the
> EXTERNALS table on every prop change (i.e. the way it is implemented right
> now, possible bugs aside).
> ...and try to find and somehow handle any abort conditions that might leave
> the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
> check/update on 'svn cleanup'.
I don't really see that a propset to svn:externals, followed immediately
by a propget, should yield exactly the same content, as long as it's
semantically identical. In other words, it'd be quite OK to treat such a
propset differently from other propsets, i.e., to parse whatever was set
into the EXTERNALS table, and have the propget read from there. It's not
as if we ever promised that magic properties are anything but magic. :)
- -- Brane
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
iQEcBAEBCgAGBQJOkw4gAAoJECm4ktDIYoUBeoYH/3fxcP+/s8njPi0VKBoP6JSG
m9zPGChadAnTSB6JzG62WUfHHFKSo/t96acVFScGv0bwPG9fairHrUpgJUD2MfTQ
P9MR45AHQ9lQSmPrkdt8YY7IsAG+6hPEm9kXdXWyXa6R86QjJrZtrrz0JZKTJF9N
hnQVCF6TRHrzUIPCETiTWIBvkS2x1bOtal+t3hY80ZqZyds8KG5jqz4VDqIQm5by
I7BpQI/VM+1bU7s8GL3svt/uYrbUvEJgBU45ZgyWxdMA21b8RIiA1LmJfwrFboH3
174fK/kT2wRAsdHeQ0HX1Z5Np59vducp1jHdLp8sB1wFL4JEW+abN3xw7lWFFMs=
=whcQ
-----END PGP SIGNATURE-----
Re: EXTERNALS table -- good or bad?
Posted by Neels J Hofmeyr <ne...@elego.de>.
On 10/08/2011 05:20 PM, Branko Čibej wrote:
>
> On 06.10.2011 22:28, Neels J Hofmeyr wrote:
>> This table is / would be very useful if one tries to find all externals
>> defined or checked out in a given subtree, without having to first find and
>> then parse all externals skels.
>
>> But in fact it is little more than a cache for svn:externals props. It
>> duplicates information in a way. But it adds the knowledge of exactly which
>> repository and relpath an external is from (stores URLs in repos-root and
>> repos-relpath, readily parsed from the various formats found in
>> svn:externals definitions).
>
> Answer's simple: never store the svn:externals property with the other
> props, always parse it into and read it from the EXTERNALS table. After
> all, it /is/ a magic property. This can be a completely wc-local
> implementation-specific thing.
...and change the interface for managing externals, too?
Currently we set externals by setting some multiline string (propset). When
coming back later (propget/edit), we expect that string to be exactly
identical to what was set earlier -- even if it contains my seven favourite
quotes of Mark Twain. Parsing that value to a table immediately and bringing
it back later discards of Twain's quotes. Those aren't a use case, but line
comments in svn:externals and various URL notations are ("^/foo" and
"svn://example.com/repos/foo" currently amount to the exact same EXTERNALS
row, i.e. same repos_id and def_repos_relpath).
Ok, we could also store that plain string in the externals table instead of
as a prop value. Does it really make a difference?
Or we're talking about a complete feature change, not using the
propset/propget API. Maybe something like
svn external add --anchor="." --url="^/foo" --path="sub/dir/foo"
Hmm, I think we better leave the prop as is, and have hooks that update the
EXTERNALS table on every prop change (i.e. the way it is implemented right
now, possible bugs aside).
...and try to find and somehow handle any abort conditions that might leave
the EXTERNALS table out-of-sync. As a safety net include an EXTERNALS table
check/update on 'svn cleanup'.
~Neels
Re: EXTERNALS table -- good or bad?
Posted by Branko Čibej <br...@e-reka.si>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
On 06.10.2011 22:28, Neels J Hofmeyr wrote:
> This table is / would be very useful if one tries to find all externals
> defined or checked out in a given subtree, without having to first find and
> then parse all externals skels.
>
> But in fact it is little more than a cache for svn:externals props. It
> duplicates information in a way. But it adds the knowledge of exactly which
> repository and relpath an external is from (stores URLs in repos-root and
> repos-relpath, readily parsed from the various formats found in
> svn:externals definitions).
Answer's simple: never store the svn:externals property with the other
props, always parse it into and read it from the EXTERNALS table. After
all, it /is/ a magic property. This can be a completely wc-local
implementation-specific thing.
- -- Brane
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
iQEcBAEBCgAGBQJOkGo6AAoJECm4ktDIYoUByywIAIwkZNnkBNP1A2YVWe2UcATs
1BXzfnooTowbdCIjgWtXD1xncpiv/+VrcH92yPg5CfuP9V++T5fSrkymRxWqt59y
ls6U50Z9HY2s+xoj3tiKpl7eUn6mBsYc6YdGkqbXg2GVGa8poWIKAGSodPm/LMOt
CsfjpUXbrx34573KFDKL7oKcr16alR/mrzhQHwOFVsAbM41yvFm+TEVnzxLhTYWe
jlwdWiQecPZOpjPfOWVduD8xFEKGJu2ecLt4PLV6YTnCKZH3S9JgleVX13hFIsIw
iHccc2q3NSEFChLSR0F340NCnXCwptSMIh8Fevqkw0pdWKjPMCHrzQYZdZYBl7k=
=N3QR
-----END PGP SIGNATURE-----
Re: EXTERNALS table -- good or bad?
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Oct 12, 2011 at 3:38 AM, Neels J Hofmeyr <ne...@elego.de> wrote:
> On 10/11/2011 01:29 PM, Johan Corveleyn wrote:
>>>> AFAIUI, the EXTERNALS table is indeed not a cache, but it notes the
>>>> externals "as rolled out by the last svn update", i.e. the externals
>>>> "as currently expanded in the WC" (at least, that's what I understood
>
> The EXTERNALS table does not track what is currently rolled out (e.g. can't
> possibly keep track of file-system rm/mv). Rather, it tracks what should be
> rolled out. An external definition may already be in the EXTERNALS table,
> but not yet checked out in the WC; for example, if, at the time the external
> def was added, there was an obstruction at the target path. Until that
> target path is freed up, the row will remain in the EXTERNALS table even
> though it does not reflect what is currently rolled out.
Hm, "should have been rolled out (by update)" vs. "has been rolled
out". We could also say that this is simply a bug: if something
obstructs the rolling out of an external, EXTERNALS should not reflect
that it was successfully rolled out. But again: I'm not the expert
here, I'm not 100% sure about all this.
> Specifically, a *file* external that is fully and correctly rolled out has a
> file_external==1 column in its NODES table's BASE row (op_depth==0) as well
> as its own repos_id and repos_path.
> But it is possible that a file external is already noted in an EXTERNALS row
> while a different dir or even file (with file_external==0) is still
> currently checked out at that path. From examining NODES and the fs, it can
> be identified whether the file external is rolled out yet or just
> obstructed. There can even be no NODES row at all, in which case a file
> lying at the target path would be known as unversioned, obstructing the
> external. I think there can even be another file external at the same path,
> and even though its BASE row says file_external==1, lib_wc would still
> notice if it was a file extl from a different URL (haven't checked this one).
>
> To know that a *dir* external is fully rolled out, a WC dir at the
> external's target path has to be a WC-root and has to match the external URL.
>
>
> Now take the situation that some externals are defined and rolled out. All
> is well. If now someone changes the svn:externals prop, it is necessary to
> know what the desired externals situation was before that change.
>
> In notes/wc-ng/externals, the EXTERNALS table is stated to solve this
> problem. Yet the problem only exists for file externals and only when
> locally changing the svn:externals prop multiple times without committing?
This problem exists just as well for dir externals. See "Handling
changes to svn:externals" in /notes/wc-ng/externals (the example given
there is with dir externals).
> As it looks to me, the area where the EXTERNALS table is not a cache is
> really quite narrow. IIUC the area is exactly confined to this use case:
>
> User does several propchanges on svn:externals,
> with intermediate 'svn update' calls.
Ok, but this is a pretty important use-case. Some users want to test
externals before committing them (which may entail changing+updating
them a couple of times before committing) . If EXTERNALS' sole purpose
would be to support this use-case: so what? Are you suggesting we
simply stop supporting that use-case?
> In this case the WC only notices the BASE's svn:externals prop; but upon
> comparing to the currently checked out node at the target path, to determine
> whether that node is an obstruction or the actual external, the WC has to
> remember what this externals node was desired to be, during the last 'svn
> update'. -Only- in the above use case, this differs from BASE's
> svn:externals prop. ... A 'commit' moves the new prop to BASE and this
> problem disappears. So "most of the time", EXTERNALS *is* just a cache.
>
> I still think this table is useful for quick scanning for presence of
> externals[1]. But for that purpose I guess it *could* simply list which
> paths are externals and where they are defined, so that you can go straight
> to the defining path and parse the details from the property. It currently
> seems to be a mere added bonus/optimization that the table reflects the URL,
> kind etc. all of the time, instead of just in the use case of multiple
> 'propset svn:externals' & 'update' cycles.
>
> EXTERNALS is almost pure optimization, just one corner case really dearly
> *needs* it. Please correct me if I'm wrong.
Not sure about the "just one corner case", but I still disagree that
it's a cache, and its purpose is optimization. No, its purpose is
keeping track of what was rolled out (maybe there are bugs in the
implementation, but I still think that that's the main responsibility
of this table). And that's a piece of information that isn't kept
anywhere else.
--
Johan
> Still I'm +1 for keeping it this way, because it seems silly to parse all of
> the props from a given node's skel blob, and to then parse all of the lines
> in the svn:externals prop, just to get at one extl definition.
>
> I wish I could have said this in fewer words and same detail...
>
> Re: documentation of the EXTERNALS table: large amounts of wc-ng are
> chronically under- and even misdocumented. Greg kept saying "moving
> target!", and for him code *is* comment, -- and hence this thread ;)
>
> ~Neels
>
> [1] need to scan for externals e.g. during svn cp wc/a wc/b, and in svn
> status for externals-below-unversioned or -below-other-wc (WIP)
>
>
Re: EXTERNALS table -- good or bad?
Posted by Neels J Hofmeyr <ne...@elego.de>.
On 10/11/2011 01:29 PM, Johan Corveleyn wrote:
>>> AFAIUI, the EXTERNALS table is indeed not a cache, but it notes the
>>> externals "as rolled out by the last svn update", i.e. the externals
>>> "as currently expanded in the WC" (at least, that's what I understood
The EXTERNALS table does not track what is currently rolled out (e.g. can't
possibly keep track of file-system rm/mv). Rather, it tracks what should be
rolled out. An external definition may already be in the EXTERNALS table,
but not yet checked out in the WC; for example, if, at the time the external
def was added, there was an obstruction at the target path. Until that
target path is freed up, the row will remain in the EXTERNALS table even
though it does not reflect what is currently rolled out.
Specifically, a *file* external that is fully and correctly rolled out has a
file_external==1 column in its NODES table's BASE row (op_depth==0) as well
as its own repos_id and repos_path.
But it is possible that a file external is already noted in an EXTERNALS row
while a different dir or even file (with file_external==0) is still
currently checked out at that path. From examining NODES and the fs, it can
be identified whether the file external is rolled out yet or just
obstructed. There can even be no NODES row at all, in which case a file
lying at the target path would be known as unversioned, obstructing the
external. I think there can even be another file external at the same path,
and even though its BASE row says file_external==1, lib_wc would still
notice if it was a file extl from a different URL (haven't checked this one).
To know that a *dir* external is fully rolled out, a WC dir at the
external's target path has to be a WC-root and has to match the external URL.
Now take the situation that some externals are defined and rolled out. All
is well. If now someone changes the svn:externals prop, it is necessary to
know what the desired externals situation was before that change.
In notes/wc-ng/externals, the EXTERNALS table is stated to solve this
problem. Yet the problem only exists for file externals and only when
locally changing the svn:externals prop multiple times without committing?
As it looks to me, the area where the EXTERNALS table is not a cache is
really quite narrow. IIUC the area is exactly confined to this use case:
User does several propchanges on svn:externals,
with intermediate 'svn update' calls.
In this case the WC only notices the BASE's svn:externals prop; but upon
comparing to the currently checked out node at the target path, to determine
whether that node is an obstruction or the actual external, the WC has to
remember what this externals node was desired to be, during the last 'svn
update'. -Only- in the above use case, this differs from BASE's
svn:externals prop. ... A 'commit' moves the new prop to BASE and this
problem disappears. So "most of the time", EXTERNALS *is* just a cache.
I still think this table is useful for quick scanning for presence of
externals[1]. But for that purpose I guess it *could* simply list which
paths are externals and where they are defined, so that you can go straight
to the defining path and parse the details from the property. It currently
seems to be a mere added bonus/optimization that the table reflects the URL,
kind etc. all of the time, instead of just in the use case of multiple
'propset svn:externals' & 'update' cycles.
EXTERNALS is almost pure optimization, just one corner case really dearly
*needs* it. Please correct me if I'm wrong.
Still I'm +1 for keeping it this way, because it seems silly to parse all of
the props from a given node's skel blob, and to then parse all of the lines
in the svn:externals prop, just to get at one extl definition.
I wish I could have said this in fewer words and same detail...
Re: documentation of the EXTERNALS table: large amounts of wc-ng are
chronically under- and even misdocumented. Greg kept saying "moving
target!", and for him code *is* comment, -- and hence this thread ;)
~Neels
[1] need to scan for externals e.g. during svn cp wc/a wc/b, and in svn
status for externals-below-unversioned or -below-other-wc (WIP)
Re: EXTERNALS table -- good or bad?
Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, Oct 11, 2011 at 12:03 PM, Julian Foad <ju...@wandisco.com> wrote:
> On Tue, 2011-10-11 at 10:20 +0200, Johan Corveleyn wrote:
>> On Mon, Oct 10, 2011 at 3:25 PM, Bert Huijben <be...@qqmail.nl> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Neels J Hofmeyr [mailto:neels@elego.de]
>> >> Sent: zaterdag 8 oktober 2011 7:58
>> >> To: Bert Huijben
>> >> Cc: 'Subversion Development'
>> >> Subject: Re: EXTERNALS table -- good or bad?
>> >>
>> >> On 10/07/2011 02:18 PM, Bert Huijben wrote:
>> >> > I think you were the one that enabled processing svn:externals on local
>> >> > additions?
>> >>
>> >> Can't say that I did :)
>> >>
>> >> > (Impossible without EXTERNALS)
>> >>
>> >> "Impossible" is such a strong word. Brief explanation?
>> >>
>> >> > And file externals would be impossible to delete, except when using
>> >> > EXTERNALS.
>> >>
>> >> Not sure about that, we do have that update_root column...
>> >>
>> >> > (etc.)
>> >> (/me greps svn:log,dev@)
>> >>
>> >> So in short, we are going to rely on the EXTERNALS table as the no.1
>> > source
>> >> for information about svn:externals definitions, i.e. keep the table
>> >> up-to-date at all times. It's more than just a cache. Right?
>> >
>> > ^/subversion/trunk/notes/wc-ng/externals contains more on the reasoning of
>> > why 'EXTERNALS'.
>> >
>> > (It was written before it was decided not to move the real actual data from
>> > BASE, so it could use some updates)
>>
>> AFAIUI, the EXTERNALS table is indeed not a cache, but it notes the
>> externals "as rolled out by the last svn update", i.e. the externals
>> "as currently expanded in the WC" (at least, that's what I understood
>> from the dev@ discussions a while ago, and from the notes doc). This
>> is a criticial piece of information, which can be different from what
>> the externals prop says (in either their BASE or ACTUAL forms).
>>
>> And it's needed in a couple of scenario's which Bert noted. For
>> instance, there is the scenario (as described in the notes file),
>> where you change an external from pointing to A, to B, and then to C,
>> without committing in between (but updating, i.e. applying the
>> externals each time). In that case, you really need the information of
>> what the "actually applied externals" were, to do a correct
>> "migration" from B to C.
>>
>> At least, that's how I understood the need for the EXTERNALS table...
>
> Can we *please* add some such description to the source code? I'd
> suggest at the definition of the table in wc-metadata.sql.
+1
But I don't feel expert enough on the subject to do that myself :-).
I'm not even sure if my understanding, as laid out above, is correct.
Maybe one of the experts (Bert?) can add the necessary documentation?
--
Johan
Re: EXTERNALS table -- good or bad?
Posted by Julian Foad <ju...@wandisco.com>.
On Tue, 2011-10-11 at 10:20 +0200, Johan Corveleyn wrote:
> On Mon, Oct 10, 2011 at 3:25 PM, Bert Huijben <be...@qqmail.nl> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Neels J Hofmeyr [mailto:neels@elego.de]
> >> Sent: zaterdag 8 oktober 2011 7:58
> >> To: Bert Huijben
> >> Cc: 'Subversion Development'
> >> Subject: Re: EXTERNALS table -- good or bad?
> >>
> >> On 10/07/2011 02:18 PM, Bert Huijben wrote:
> >> > I think you were the one that enabled processing svn:externals on local
> >> > additions?
> >>
> >> Can't say that I did :)
> >>
> >> > (Impossible without EXTERNALS)
> >>
> >> "Impossible" is such a strong word. Brief explanation?
> >>
> >> > And file externals would be impossible to delete, except when using
> >> > EXTERNALS.
> >>
> >> Not sure about that, we do have that update_root column...
> >>
> >> > (etc.)
> >> (/me greps svn:log,dev@)
> >>
> >> So in short, we are going to rely on the EXTERNALS table as the no.1
> > source
> >> for information about svn:externals definitions, i.e. keep the table
> >> up-to-date at all times. It's more than just a cache. Right?
> >
> > ^/subversion/trunk/notes/wc-ng/externals contains more on the reasoning of
> > why 'EXTERNALS'.
> >
> > (It was written before it was decided not to move the real actual data from
> > BASE, so it could use some updates)
>
> AFAIUI, the EXTERNALS table is indeed not a cache, but it notes the
> externals "as rolled out by the last svn update", i.e. the externals
> "as currently expanded in the WC" (at least, that's what I understood
> from the dev@ discussions a while ago, and from the notes doc). This
> is a criticial piece of information, which can be different from what
> the externals prop says (in either their BASE or ACTUAL forms).
>
> And it's needed in a couple of scenario's which Bert noted. For
> instance, there is the scenario (as described in the notes file),
> where you change an external from pointing to A, to B, and then to C,
> without committing in between (but updating, i.e. applying the
> externals each time). In that case, you really need the information of
> what the "actually applied externals" were, to do a correct
> "migration" from B to C.
>
> At least, that's how I understood the need for the EXTERNALS table...
Can we *please* add some such description to the source code? I'd
suggest at the definition of the table in wc-metadata.sql.
- Julian
Re: EXTERNALS table -- good or bad?
Posted by Johan Corveleyn <jc...@gmail.com>.
On Mon, Oct 10, 2011 at 3:25 PM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Neels J Hofmeyr [mailto:neels@elego.de]
>> Sent: zaterdag 8 oktober 2011 7:58
>> To: Bert Huijben
>> Cc: 'Subversion Development'
>> Subject: Re: EXTERNALS table -- good or bad?
>>
>> On 10/07/2011 02:18 PM, Bert Huijben wrote:
>> > I think you were the one that enabled processing svn:externals on local
>> > additions?
>>
>> Can't say that I did :)
>>
>> > (Impossible without EXTERNALS)
>>
>> "Impossible" is such a strong word. Brief explanation?
>>
>> > And file externals would be impossible to delete, except when using
>> > EXTERNALS.
>>
>> Not sure about that, we do have that update_root column...
>>
>> > (etc.)
>> (/me greps svn:log,dev@)
>>
>> So in short, we are going to rely on the EXTERNALS table as the no.1
> source
>> for information about svn:externals definitions, i.e. keep the table
>> up-to-date at all times. It's more than just a cache. Right?
>
> ^/subversion/trunk/notes/wc-ng/externals contains more on the reasoning of
> why 'EXTERNALS'.
>
> (It was written before it was decided not to move the real actual data from
> BASE, so it could use some updates)
AFAIUI, the EXTERNALS table is indeed not a cache, but it notes the
externals "as rolled out by the last svn update", i.e. the externals
"as currently expanded in the WC" (at least, that's what I understood
from the dev@ discussions a while ago, and from the notes doc). This
is a criticial piece of information, which can be different from what
the externals prop says (in either their BASE or ACTUAL forms).
And it's needed in a couple of scenario's which Bert noted. For
instance, there is the scenario (as described in the notes file),
where you change an external from pointing to A, to B, and then to C,
without committing in between (but updating, i.e. applying the
externals each time). In that case, you really need the information of
what the "actually applied externals" were, to do a correct
"migration" from B to C.
At least, that's how I understood the need for the EXTERNALS table...
--
Johan
RE: EXTERNALS table -- good or bad?
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Neels J Hofmeyr [mailto:neels@elego.de]
> Sent: zaterdag 8 oktober 2011 7:58
> To: Bert Huijben
> Cc: 'Subversion Development'
> Subject: Re: EXTERNALS table -- good or bad?
>
> On 10/07/2011 02:18 PM, Bert Huijben wrote:
> > I think you were the one that enabled processing svn:externals on local
> > additions?
>
> Can't say that I did :)
>
> > (Impossible without EXTERNALS)
>
> "Impossible" is such a strong word. Brief explanation?
>
> > And file externals would be impossible to delete, except when using
> > EXTERNALS.
>
> Not sure about that, we do have that update_root column...
>
> > (etc.)
> (/me greps svn:log,dev@)
>
> So in short, we are going to rely on the EXTERNALS table as the no.1
source
> for information about svn:externals definitions, i.e. keep the table
> up-to-date at all times. It's more than just a cache. Right?
^/subversion/trunk/notes/wc-ng/externals contains more on the reasoning of
why 'EXTERNALS'.
(It was written before it was decided not to move the real actual data from
BASE, so it could use some updates)
Bert
Re: EXTERNALS table -- good or bad?
Posted by Neels J Hofmeyr <ne...@elego.de>.
On 10/07/2011 02:18 PM, Bert Huijben wrote:
> I think you were the one that enabled processing svn:externals on local
> additions?
Can't say that I did :)
> (Impossible without EXTERNALS)
"Impossible" is such a strong word. Brief explanation?
> And file externals would be impossible to delete, except when using
> EXTERNALS.
Not sure about that, we do have that update_root column...
> (etc.)
(/me greps svn:log,dev@)
So in short, we are going to rely on the EXTERNALS table as the no.1 source
for information about svn:externals definitions, i.e. keep the table
up-to-date at all times. It's more than just a cache. Right?
~Neels
RE: EXTERNALS table -- good or bad?
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Neels J Hofmeyr [mailto:neels@elego.de]
> Sent: donderdag 6 oktober 2011 22:29
> To: Subversion Development
> Subject: EXTERNALS table -- good or bad?
>
> Hi all,
>
> I have committed code to trunk that populates the EXTERNALS table during
> upgrade from 1.6. With a nudge from Philip, I noticed that the EXTERNALS
> table is still a neither-here-nor-there implementation. Now I'm unsure.
>
> This table is / would be very useful if one tries to find all externals
> defined or checked out in a given subtree, without having to first find
and
> then parse all externals skels.
>
> But in fact it is little more than a cache for svn:externals props. It
> duplicates information in a way. But it adds the knowledge of exactly
which
> repository and relpath an external is from (stores URLs in repos-root and
> repos-relpath, readily parsed from the various formats found in
> svn:externals definitions).
>
> So it is useful for speeding up things, but it also has a danger: heavens
> forbid if the EXTERNALS table is ever out of sync with the svn:externals
> props. So, we may often be forced to parse the props anyway (cleanup?).
>
> What do you guys think about the EXTERNALS table? Yay or nay? What
> thought
> has gone into this so far?
There are also quite a few issues resolved by separating the definition and
applied externals in the working copy.
I think you were the one that enabled processing svn:externals on local
additions?
(Impossible without EXTERNALS)
And file externals would be impossible to delete, except when using
EXTERNALS.
(etc.)
Bert