You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Dr. Thomas Orgis" <th...@uni-hamburg.de> on 2019/12/20 18:19:27 UTC
Handling of unexpectedly changed kind, accepting local changes in
versioning changes by external forces
Hi,
it was suggested to me to discuss Subversion behaviour that is causing
me some nuisance while versioning configuration files, namely the /etc
dir of Linux servers. It may be smarter to just check in the files I
edit (I do that on some), but there is some convenience in telling svn
to just accept whatever an (automated) update did to a directory as
new state. Some semantic info gets lost (files being copied/moved around)
without the content fixation that git uses to derive such things, but I
prefer Subversion for such a task, as it more closely follows the idea
of a directory tree, including treating empty directories as valid
objects.
Please let's avoid the discussion whether one should version config
files of the actual servers or the pupped/ansible/… scripts that
generate them and just consider that
1. a directory tree on a live system is versioned in a Subversion
repository,
2. various automated processes (package manager) modify that directory
tree, and
3. Subversion is tripped up by nodes changing their kind from/to
symbolic link, plain file, directory.
I do have a script that looks at `svn st` and adds/removes files
to turn ! into D, ? into A, basically. A mythical `svn synctree`
command might do that, too, in a safer way.
If a file in the working copy is replaced by a symlink, as
reorganizations on Linux distribution upgrades make it happen, svn
refuses the commit:
svn: E145001: Commit failed (details follow):
svn: E145001: Node '...' has unexpectedly changed kind
Now, this can be worked around by moving the changed file/link aside,
telling svn to forget about it, then move back and re-add. Commit. Like
this example (out of one of the attached scripts that demonstrate the
issue and resolution in some variations of changes in kind:
# Known state to svn:
echo a > a
echo b > b
svn add a b
svn commit -m 'created two test files'
# The set up:
echo "The world happens and switches a for a symlink to b."
rm a
ln -s b a
# The workaround:
echo "Workaround: move around, remove, add."
tmp=$(mktemp a.XXXX)
mv a "$tmp"
svn rm a
mv "$tmp" a
svn add a
svn commit -m "Now a is a symlink to b in the repo."
Any thoughts on how svn could be made fitter/adaptible to a scenariro
where $anything can happen to the working copy and one just wants it to
make sense out of the current state?
This message of unexpected kind can be frustrating for the user that
for sure can see that the erstwhile file is a symlink now and is
wishing for a concise command that tells svn to just accept that fact.
At the very least, it would be nice to be able to resolve the
situation without having to delete/move the actual file (here or in
another working copy as a crutch). Just tell subversion to update it's
metadata with respect to this file. If it is a change in kind, I guess
it would be fine if the internal workings would be equivalent to `svn rm`
and `svn add`.
Comments? Suggeststions? Solutions?
Alrighty then,
Thomas
PS: Of course, as a backup/restore solution, svn on /etc is not
perfect, with it not bothering with file ownership and detailed
permissions. That can be handled by versioning the output of
find "$@" -print0 \
| sort -z | xargs -0 stat -c '%a %u:%g %U:%G %n'
(not too much thought on line breaks in file names). Of course it would
be nice if there was the option to version ownership and permission
info, too (not sure if extended attributes should be relevant for
configuration). I faintly remember a project that built on svn to
version whole filesystems including special/device files, but cannot
find it right now. I still do think that exploiting the file system
metaphor of svn provides the niche for it to continue to exist besides
the all-mighty git.
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
Re: Handling of unexpectedly changed kind, accepting local changes in
versioning changes by external forces
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, Dec 23, 2019 at 8:04 AM Dr. Thomas Orgis
<th...@uni-hamburg.de> wrote:
>
> Am Fri, 20 Dec 2019 19:36:36 +0100
> schrieb Stefan Sperling <st...@elego.de>:
>
> > There is a work-in-progress branch for such a feature:
> > https://svn.apache.org/viewvc/subversion/branches/addremove/
> > Would you be interested in working on this?
>
> > At present it doesn't implement much more than a wrapper around
> > 'svn status' could do, mapping '!' to 'D' and '?' to 'A' status.
>
> … but safer, I presume. Would that command be called 'svn addremove'?
> Somehow I prefer some wording involving the term 'sync', meaning to
> bring the mental state of subversion back into sync with the actual
> state of the tree. But that's the bike shed's colour.
>
> Indeed, existence of such a branch is a welcomed surprise.
>
> > I intended to work on an ad-hoc rename detection feature for this
> > branch, like Git would do it. But I ran out of time.
>
> Hm. I believe git is rather smart about detecting the content in the
> files, even if bits changed, right? You were just thinking about
> finding the same file under a different name, strictly for cases where
> one file vanished and another appeared?
>
> Actually, I have doubts that adding such smarts to subversion is
> beneficial. First, you won't outsmart git on content tracking, second:
> For me, Subversion is the system that does what it is told, not more.
> No magic. When I say that I derived a file from another via 'svn copy',
> even if I replace all contents, I want that fact recorded. Any magic
> behind the scenes, any guessing of relationships, would hit me
> unexpectedly.
I haven't looked at the addremove branch yet. But I've had some
thoughts about out-of-band move/renames over the years, so just to get
them out there...
Currently:
$ mv foo bar
$ svn status
! foo
? bar
$ svn mv foo bar
svn: E155010: Path '/path/to/bar' is not a directory
You have to do the two-step dance:
$ mv bar foo
$ svn mv foo bar
TortoiseSVN has a "Repair Move" command.
The 'is not a directory' error is consistent with the behavior of the
system move command. But maybe we can do a little better. Some ideas:
(1) Add a '--repair' option to 'svn mv'. With this option, only moving
a !foo to a ?bar is possible:
$ mv foo bar
$ svn status
! foo
? bar
$ svn mv foo bar --repair
$ svn st
A + bar
> moved from foo
D foo
> moved to bar
(2) Don't add a '--repair' option, but automatically detect the !foo
?bar case and allow 'svn mv foo bar' to "repair" the move. I'm sure
this one is fraught with peril, but I mention it for completeness:
$ mv foo bar
$ svn status
! foo
? bar
$ svn mv foo bar
$ svn st
A + bar
> moved from foo
D foo
> moved to bar
(3) If someone really wants to get fancy, do either (1) or (2) and
then create a 'svn mv --automatic' with no other arguments. This
searches for all ! and ? in the wc, does git-style "similar content"
magic, and repairs the moves. But unlike git, this never runs unless
you ask for it.
$ mv foo bar
$ svn status
! foo
? bar
$ svn mv --automatic
$ svn st
A + bar
> moved from foo
D foo
> moved to bar
Note that svn_client_move7(), added in 1.8, has a 'metadata_only'
parameter: "If metadata_only is TRUE and moving a file in a working
copy, everything in the metadata is updated as if the node is moved,
but the actual disk move operation is not performed. This feature is
useful for clients that want to keep the working copy in sync while
the actual working copy is updated by some other task."
But svn_cl__move() in subversion/svn/move-cmd.c always passes FALSE
for metadata_only and there is no other invocation of
svn_client_move7() with metadata_only TRUE (at least not in trunk).
There don't appear to be any tests for metadata_only == TRUE.
Nathan
Re: Handling of unexpectedly changed kind, accepting local changes
in versioning changes by external forces
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Dec 23, 2019 at 02:03:46PM +0100, Dr. Thomas Orgis wrote:
> Am Fri, 20 Dec 2019 19:36:36 +0100
> schrieb Stefan Sperling <st...@elego.de>:
>
> > There is a work-in-progress branch for such a feature:
> > https://svn.apache.org/viewvc/subversion/branches/addremove/
> > Would you be interested in working on this?
>
> > At present it doesn't implement much more than a wrapper around
> > 'svn status' could do, mapping '!' to 'D' and '?' to 'A' status.
>
> … but safer, I presume. Would that command be called 'svn addremove'?
> Somehow I prefer some wording involving the term 'sync', meaning to
> bring the mental state of subversion back into sync with the actual
> state of the tree. But that's the bike shed's colour.
I think it makes sense to use that name since it is also used by Mercurial.
Mercurial put a lot of thought into their UI so I'm happy to use naming
they have come up with, rather than duplicating effort.
> > I intended to work on an ad-hoc rename detection feature for this
> > branch, like Git would do it. But I ran out of time.
>
> Hm. I believe git is rather smart about detecting the content in the
> files, even if bits changed, right? You were just thinking about
> finding the same file under a different name, strictly for cases where
> one file vanished and another appeared?
>
> Actually, I have doubts that adding such smarts to subversion is
> beneficial. First, you won't outsmart git on content tracking, second:
> For me, Subversion is the system that does what it is told, not more.
> No magic. When I say that I derived a file from another via 'svn copy',
> even if I replace all contents, I want that fact recorded. Any magic
> behind the scenes, any guessing of relationships, would hit me
> unexpectedly.
>
> But then, if this is a feature I explicitly ask for … 'svn
> rename-find-copies-magic', then it fits, I guess. Could even involve
> some threshold for amount of matching content, or how much value is put
> on the same file name appearing at a different place. Then it's a tool
> again, employed by the user. Was that the plan?
There was no finalized plan. This is an experimental branch.
I tend to work towards a certain goal but given limited time existing
contraints in the implementation have to be accounted for.
The simple case where file are 100% identical already works, anything
further can be discussed and evaluated. It only has to be as smart as
we can reasonably make it.
> > Regardless, if this feature branch also solved the symlink issue you
> > describe below, the branch might already be worth merging to trunk.
>
> Seems to be a simple addition, just have to add the cases of changed
> kind and treat them accordingly as removes/adds.
>
> Maybe I have some time to look at this branch and cargo-code from its
> diff to add the symlink thing. After getting that other important work
> done. Probably in the new year.
Great :) Let me know if you have any questions.
And I could rebase the branch to trunk for you if needed.
> > Technically, SVN expects tree modifications to be made with
> > Subversion commands like 'svn add', 'svn rm' etc. Modifying
> > the working copy's tree structure with external commands is
> > actually somewhat "out of spec".
>
> Sure, it is. But expecially when more people are used to the git way of
> just doing things, having a 'just figure it out' command to at least
> get the changes tracked, even if not with optimal metadata, would be
> helpful for those who forget to use 'svn mv' instead of 'mv'.
It makes no sense to "record a move" which won't be recognized as such
by the conflict resolver. A move must have copyfrom + delete recorded
within the same revision. Otherwise it is not a move.
> (The rename/copy detection probably should be an --option, or set of
> those with thresholds, to the man sync/addremove command. Having your
> initial design plan spelled out would help.)
It has been a while. I'm afraid I didn't keep notes beyond those recorded
in log messages on the branch.
Re: Handling of unexpectedly changed kind, accepting local changes
in versioning changes by external forces
Posted by "Dr. Thomas Orgis" <th...@uni-hamburg.de>.
Am Fri, 20 Dec 2019 19:36:36 +0100
schrieb Stefan Sperling <st...@elego.de>:
> There is a work-in-progress branch for such a feature:
> https://svn.apache.org/viewvc/subversion/branches/addremove/
> Would you be interested in working on this?
> At present it doesn't implement much more than a wrapper around
> 'svn status' could do, mapping '!' to 'D' and '?' to 'A' status.
… but safer, I presume. Would that command be called 'svn addremove'?
Somehow I prefer some wording involving the term 'sync', meaning to
bring the mental state of subversion back into sync with the actual
state of the tree. But that's the bike shed's colour.
Indeed, existence of such a branch is a welcomed surprise.
> I intended to work on an ad-hoc rename detection feature for this
> branch, like Git would do it. But I ran out of time.
Hm. I believe git is rather smart about detecting the content in the
files, even if bits changed, right? You were just thinking about
finding the same file under a different name, strictly for cases where
one file vanished and another appeared?
Actually, I have doubts that adding such smarts to subversion is
beneficial. First, you won't outsmart git on content tracking, second:
For me, Subversion is the system that does what it is told, not more.
No magic. When I say that I derived a file from another via 'svn copy',
even if I replace all contents, I want that fact recorded. Any magic
behind the scenes, any guessing of relationships, would hit me
unexpectedly.
But then, if this is a feature I explicitly ask for … 'svn
rename-find-copies-magic', then it fits, I guess. Could even involve
some threshold for amount of matching content, or how much value is put
on the same file name appearing at a different place. Then it's a tool
again, employed by the user. Was that the plan?
> Regardless, if this feature branch also solved the symlink issue you
> describe below, the branch might already be worth merging to trunk.
Seems to be a simple addition, just have to add the cases of changed
kind and treat them accordingly as removes/adds.
Maybe I have some time to look at this branch and cargo-code from its
diff to add the symlink thing. After getting that other important work
done. Probably in the new year.
> Technically, SVN expects tree modifications to be made with
> Subversion commands like 'svn add', 'svn rm' etc. Modifying
> the working copy's tree structure with external commands is
> actually somewhat "out of spec".
Sure, it is. But expecially when more people are used to the git way of
just doing things, having a 'just figure it out' command to at least
get the changes tracked, even if not with optimal metadata, would be
helpful for those who forget to use 'svn mv' instead of 'mv'.
(The rename/copy detection probably should be an --option, or set of
those with thresholds, to the man sync/addremove command. Having your
initial design plan spelled out would help.)
> Are you aware of 'asvn'?
> https://svn.apache.org/viewvc/subversion/trunk/contrib/client-side/asvn?revision=1295006&view=markup
I was not. It's not what I remember, but something to look at. Thanks
for the pointer. Would help if it'd be contained in standard installs of
svn, but I could roll it out on the servers. It reminds me of these
idle ideas to have signed commits via added svn properties …
Alrighty then,
Thomas
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
Re: Handling of unexpectedly changed kind, accepting local changes
in versioning changes by external forces
Posted by Stefan Sperling <st...@elego.de>.
On Fri, Dec 20, 2019 at 07:19:27PM +0100, Dr. Thomas Orgis wrote:
> Please let's avoid the discussion whether one should version config
> files of the actual servers or the pupped/ansible/… scripts that
> generate them and just consider that
>
> 1. a directory tree on a live system is versioned in a Subversion
> repository,
> 2. various automated processes (package manager) modify that directory
> tree, and
> 3. Subversion is tripped up by nodes changing their kind from/to
> symbolic link, plain file, directory.
>
> I do have a script that looks at `svn st` and adds/removes files
> to turn ! into D, ? into A, basically. A mythical `svn synctree`
> command might do that, too, in a safer way.
There is a work-in-progress branch for such a feature:
https://svn.apache.org/viewvc/subversion/branches/addremove/
Would you be interested in working on this?
At present it doesn't implement much more than a wrapper around
'svn status' could do, mapping '!' to 'D' and '?' to 'A' status.
I intended to work on an ad-hoc rename detection feature for this
branch, like Git would do it. But I ran out of time.
Regardless, if this feature branch also solved the symlink issue you
describe below, the branch might already be worth merging to trunk.
> If a file in the working copy is replaced by a symlink, as
> reorganizations on Linux distribution upgrades make it happen, svn
> refuses the commit:
>
> svn: E145001: Commit failed (details follow):
> svn: E145001: Node '...' has unexpectedly changed kind
>
> Now, this can be worked around by moving the changed file/link aside,
> telling svn to forget about it, then move back and re-add. Commit. Like
> this example (out of one of the attached scripts that demonstrate the
> issue and resolution in some variations of changes in kind:
>
> # Known state to svn:
> echo a > a
> echo b > b
> svn add a b
> svn commit -m 'created two test files'
>
> # The set up:
> echo "The world happens and switches a for a symlink to b."
> rm a
> ln -s b a
Technically, SVN expects tree modifications to be made with
Subversion commands like 'svn add', 'svn rm' etc. Modifying
the working copy's tree structure with external commands is
actually somewhat "out of spec".
> This message of unexpected kind can be frustrating for the user that
> for sure can see that the erstwhile file is a symlink now and is
> wishing for a concise command that tells svn to just accept that fact.
I agree that it would be nice to make this case easier to deal with.
> (not too much thought on line breaks in file names). Of course it would
> be nice if there was the option to version ownership and permission
> info, too (not sure if extended attributes should be relevant for
> configuration). I faintly remember a project that built on svn to
> version whole filesystems including special/device files, but cannot
> find it right now.
Are you aware of 'asvn'?
https://svn.apache.org/viewvc/subversion/trunk/contrib/client-side/asvn?revision=1295006&view=markup