You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by David Wallace <da...@bluepearlsoftware.com> on 2011/08/18 01:38:57 UTC

Resoving tree conflicts results in inconsistent state between two working copies of same branch

I just described my experience inadvertently creating an inconsistent state between two working copies of the same branch while trying to resolve tree conflicts over at: http://stackoverflow.com/questions/767763/svn-how-to-resolve-new-tree-conflicts-when-file-is-added-on-two-branches/7100512#7100512
To make this more self-contained, I'll repeat the key text here:

(1): I had added some files while working on my initial branch, branch1; (2) I created a new branch, branch2 for further development, branching it off from the trunk and then merging in my changes from branch1 (3) A co-worker had copied my mods from branch1 to his own branch, added further mods, and then merged back to the trunk; (4) I now wanted to merge the latest changes from trunk into my current working branch, branch2. This is with svn 1.6.17.

The merge had tree conflicts with the new files, and I wanted the new version from the trunk where they differed, so from a clean copy of branch2, I did an svn delete of the conflicting files, committed these branch2 changes (thus creating a temporary version of branch2 without the files in question), and then did my merge from the trunk. I did this because I wanted the history to match the trunk version so that I wouldn't have more problems later when trying to merge back to trunk. Merge went fine, I got the trunk version of the files, svn st shows all ok, and then I hit more tree conflicts while trying to commit the changes, between the delete I had done earlier and the add from the merge. Did an svn resolve of the conflicts in favor of my working copy (which now had the trunk version of the files), and got it to commit. All should be good, right?

Well, no. An update of another copy of branch2 resulted in the old version of the files (pre-trunk merge). So now I have two different working copies of branch2, supposedly updated to the same version, with two different versions of the files, and both insisting that they are fully up to date! Checking out a clean copy of branch2 resulted in the old (pre-trunk) version of the files. I manually update these to the trunk version and commit the changes, go back to my first working copy (from which I had submitted the trunk changes originally), try to update it, and now get a checksum error on the files in question. Blow the directory in question away, get a new version via update, and finally I have what should be a good version of branch2 with the trunk changes. I hope.

Further notes:  Our standard working practice is that all merges except merges from the trunk are done with a specific range of revision numbers, getting the relevant revisions via svn log -stop-on-copy on the appropriate branch.  This applied to my merge from branch1 to branch2, my co-worker's merge from my branch1 to his branch, and his merge from his branch back to the trunk, but not to my merges from the trunk described above.  I'm not sure if this matters, but I report it here in case it makes a difference when trying to reproduce the problem.  Most of the svn operations took place on Windows 7 Pro from a Cygwin command line, but the other copy of branch2 that I was comparing to was on a Linux VM.
Dave Wallace


Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, Aug 23, 2011 at 7:27 PM, Neels J Hofmeyr <ne...@elego.de> wrote:
> On 08/23/2011 09:07 AM, Johan Corveleyn wrote:
>> On Tue, Aug 23, 2011 at 5:13 AM, Neels J Hofmeyr <ne...@elego.de> wrote:
>>> I can reproduce the corrupted WC reported by David Wallace with 1.6.17.
>>> 1.7.x does not show this bug.
>>>
>>> I have attached a reproduction script that is roughly based on David's
>>> typescript, but is only a fraction of the size.
>>>
>>> The error occurs like this:
>>>
>>> - use Subversion release 1.6.17
>>> - update WC
>>> - locally delete file 'foo'
>>> - commit
>>> - do *not* update.
>>> - merge, so that file 'foo' is added (e.g. from another branch).
>>>  status becomes: A    +  foo
>>> - update
>>> - A tree conflict "local edit, incoming delete" is flagged.
>>
>> At this point, this seems remotely similar to
>> http://subversion.tigris.org/issues/show_bug.cgi?id=3526 "Commit of
>
> Indeed, looks like we misinterpret absent/not-present states in both ways.
> Or should I say, misinterpetED :)
>
>> newly added file followed by move (or delete) of parent dir causes
>> tree conflict". Except that it's more like "Commit of deleted file
>> followed by a merge adding the same file to the parent dir causes tree
>> conflict". BTW: Does it have to be a merge adding the same file (A+),
>> or can it also happen with a plain 'add' of that same file (A)?
>
> No, it has to be an
> - add with history (copy or merge)
> - from a different branch or an earlier rev,
>  i.e. copy-from differing content.
>
>>
>> In issue #3526 the conflict came from one of the 'entry-props' being
>> updated (the last-changed-revision number), while it was locally
>> deleted. I don't understand the internals too much here, but on the
>> surface it looks like a conflict between a local entry-prop being
>> updated (the copyfrom perhaps?), and the remote delete coming in. Or
>> something like that :-).
>
> I guess that this situation comes from the parent dir thinking the file
> should be there/not be there (as the parent dir is out-of-date) while the
> file's own status is the reverse, as it is up-to-date. Will test.
>
>>> The committed result (svn cat <URL>) is different from the on-disk contents,
>>> even though the file is claimed to be up-to-date.
>>
>> That's certainly a new result, one which I hadn't reached with #3526.
>> Nice! (well, not nice of course, but ... interesting :-)).
>
> Hmm, let me try that...
> No, since both the local state and the committed state have the dir deleted,
> there is no way to have differing contents. The WC corruption happens when
> the BASE text differs. (probably props affected too?)
>
>
> I've discussed with stsp on #svn-dev that we wonder why this hasn't been
> found in the two years that have passed, and that, looking at the upcoming
> 1.7.0 release that fixes these issues and noting that 1.6.x will only
> receive security fixes in the near future, we only need to bother fixing
> this on 1.6.x *if* it is possible to lose data with these bugs.
>
> That happens not to be the case.
>
> Issue #3526 ends up with a deleted dir and doesn't even reach a corrupted
> working copy state other than a nonsensical tree conflict.
>
> The issue found by David Wallace, verified by tests, does not happen when
> there are local modifications on the file in question. When there are local
> mods, the new content is committed and no WC corruption occurs.
>
> Such WC corruption can *NOT* be cleaned up by removing just the file in
> question and letting it be restored by 'svn update'. The *parent* folder of
> the file in question must be plain-'rm -rf'd and restored by update.
> An entirely new checkout is also a way out of such a situation.
>
> While the WC is corrupted, any new mods on the file as well as any commit
> will bail with mismatching checksums. It is not possible to corrupt the
> repos, beyond the fact that the wrong BASE was committed by the client.
>
>
> In conclusion, we will not bother to fix this issue for 1.6.x.
> Agreed?

Fine by me. This is probably one of those things that will be quite
hard and painful to fix in 1.6. And since, as you say, 1.7 is just
around the corner ...

-- 
Johan

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Neels J Hofmeyr <ne...@elego.de>.
On 08/23/2011 09:07 AM, Johan Corveleyn wrote:
> On Tue, Aug 23, 2011 at 5:13 AM, Neels J Hofmeyr <ne...@elego.de> wrote:
>> I can reproduce the corrupted WC reported by David Wallace with 1.6.17.
>> 1.7.x does not show this bug.
>>
>> I have attached a reproduction script that is roughly based on David's
>> typescript, but is only a fraction of the size.
>>
>> The error occurs like this:
>>
>> - use Subversion release 1.6.17
>> - update WC
>> - locally delete file 'foo'
>> - commit
>> - do *not* update.
>> - merge, so that file 'foo' is added (e.g. from another branch).
>>  status becomes: A    +  foo
>> - update
>> - A tree conflict "local edit, incoming delete" is flagged.
> 
> At this point, this seems remotely similar to
> http://subversion.tigris.org/issues/show_bug.cgi?id=3526 "Commit of

Indeed, looks like we misinterpret absent/not-present states in both ways.
Or should I say, misinterpetED :)

> newly added file followed by move (or delete) of parent dir causes
> tree conflict". Except that it's more like "Commit of deleted file
> followed by a merge adding the same file to the parent dir causes tree
> conflict". BTW: Does it have to be a merge adding the same file (A+),
> or can it also happen with a plain 'add' of that same file (A)?

No, it has to be an
- add with history (copy or merge)
- from a different branch or an earlier rev,
  i.e. copy-from differing content.

> 
> In issue #3526 the conflict came from one of the 'entry-props' being
> updated (the last-changed-revision number), while it was locally
> deleted. I don't understand the internals too much here, but on the
> surface it looks like a conflict between a local entry-prop being
> updated (the copyfrom perhaps?), and the remote delete coming in. Or
> something like that :-).

I guess that this situation comes from the parent dir thinking the file
should be there/not be there (as the parent dir is out-of-date) while the
file's own status is the reverse, as it is up-to-date. Will test.

>> The committed result (svn cat <URL>) is different from the on-disk contents,
>> even though the file is claimed to be up-to-date.
> 
> That's certainly a new result, one which I hadn't reached with #3526.
> Nice! (well, not nice of course, but ... interesting :-)).

Hmm, let me try that...
No, since both the local state and the committed state have the dir deleted,
there is no way to have differing contents. The WC corruption happens when
the BASE text differs. (probably props affected too?)


I've discussed with stsp on #svn-dev that we wonder why this hasn't been
found in the two years that have passed, and that, looking at the upcoming
1.7.0 release that fixes these issues and noting that 1.6.x will only
receive security fixes in the near future, we only need to bother fixing
this on 1.6.x *if* it is possible to lose data with these bugs.

That happens not to be the case.

Issue #3526 ends up with a deleted dir and doesn't even reach a corrupted
working copy state other than a nonsensical tree conflict.

The issue found by David Wallace, verified by tests, does not happen when
there are local modifications on the file in question. When there are local
mods, the new content is committed and no WC corruption occurs.

Such WC corruption can *NOT* be cleaned up by removing just the file in
question and letting it be restored by 'svn update'. The *parent* folder of
the file in question must be plain-'rm -rf'd and restored by update.
An entirely new checkout is also a way out of such a situation.

While the WC is corrupted, any new mods on the file as well as any commit
will bail with mismatching checksums. It is not possible to corrupt the
repos, beyond the fact that the wrong BASE was committed by the client.


In conclusion, we will not bother to fix this issue for 1.6.x.
Agreed?

~Neels


Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, Aug 23, 2011 at 5:13 AM, Neels J Hofmeyr <ne...@elego.de> wrote:
> I can reproduce the corrupted WC reported by David Wallace with 1.6.17.
> 1.7.x does not show this bug.
>
> I have attached a reproduction script that is roughly based on David's
> typescript, but is only a fraction of the size.
>
> The error occurs like this:
>
> - use Subversion release 1.6.17
> - update WC
> - locally delete file 'foo'
> - commit
> - do *not* update.
> - merge, so that file 'foo' is added (e.g. from another branch).
>  status becomes: A    +  foo
> - update
> - A tree conflict "local edit, incoming delete" is flagged.

At this point, this seems remotely similar to
http://subversion.tigris.org/issues/show_bug.cgi?id=3526 "Commit of
newly added file followed by move (or delete) of parent dir causes
tree conflict". Except that it's more like "Commit of deleted file
followed by a merge adding the same file to the parent dir causes tree
conflict". BTW: Does it have to be a merge adding the same file (A+),
or can it also happen with a plain 'add' of that same file (A)?

In issue #3526 the conflict came from one of the 'entry-props' being
updated (the last-changed-revision number), while it was locally
deleted. I don't understand the internals too much here, but on the
surface it looks like a conflict between a local entry-prop being
updated (the copyfrom perhaps?), and the remote delete coming in. Or
something like that :-).

Anyway, it's just a vague similarity, maybe they have nothing to do
with each other, so I could just as well be saying nonsense here :-).

> This is obviously wrong. The local 'foo' has been deleted (and committed).
> Taking this situation further results in a truly corrupted WC:
>
> - resolve the tree conflict
> - commit
>
> The committed result (svn cat <URL>) is different from the on-disk contents,
> even though the file is claimed to be up-to-date.

That's certainly a new result, one which I hadn't reached with #3526.
Nice! (well, not nice of course, but ... interesting :-)).

> Trunk doesn't show this behavior (thumbs up for wc-ng), but we should try to
> find a fix for 1.6.x.
>
> stsp, you wanna hunt this down?
>
> Thank you very much, Dave! Seriously bad mojo, as you said. Good work!!
> ...even though it took ages to deflate your darn typescript ;)
>
> ~Neels
>

-- 
Johan

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Hyrum K Wright <hy...@wandisco.com>.
For what its worth, this sounds very much like some tree-conflicts
badness that I ran into on occasion with 1.6.  I was never able to
reproduce it, since by the time I un-hosed my working copy, I'd
forgotten what I had done to get myself into the mess in the first
place.  It's good to hear that it's fixed in 1.7.

-Hyrum

On Mon, Aug 22, 2011 at 10:13 PM, Neels J Hofmeyr <ne...@elego.de> wrote:
> I can reproduce the corrupted WC reported by David Wallace with 1.6.17.
> 1.7.x does not show this bug.
>
> I have attached a reproduction script that is roughly based on David's
> typescript, but is only a fraction of the size.
>
> The error occurs like this:
>
> - use Subversion release 1.6.17
> - update WC
> - locally delete file 'foo'
> - commit
> - do *not* update.
> - merge, so that file 'foo' is added (e.g. from another branch).
>  status becomes: A    +  foo
> - update
> - A tree conflict "local edit, incoming delete" is flagged.
>
> This is obviously wrong. The local 'foo' has been deleted (and committed).
> Taking this situation further results in a truly corrupted WC:
>
> - resolve the tree conflict
> - commit
>
> The committed result (svn cat <URL>) is different from the on-disk contents,
> even though the file is claimed to be up-to-date.
>
> Trunk doesn't show this behavior (thumbs up for wc-ng), but we should try to
> find a fix for 1.6.x.
>
> stsp, you wanna hunt this down?
>
> Thank you very much, Dave! Seriously bad mojo, as you said. Good work!!
> ...even though it took ages to deflate your darn typescript ;)
>
> ~Neels
>



-- 

uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Neels J Hofmeyr <ne...@elego.de>.
I can reproduce the corrupted WC reported by David Wallace with 1.6.17.
1.7.x does not show this bug.

I have attached a reproduction script that is roughly based on David's
typescript, but is only a fraction of the size.

The error occurs like this:

- use Subversion release 1.6.17
- update WC
- locally delete file 'foo'
- commit
- do *not* update.
- merge, so that file 'foo' is added (e.g. from another branch).
  status becomes: A    +  foo
- update
- A tree conflict "local edit, incoming delete" is flagged.

This is obviously wrong. The local 'foo' has been deleted (and committed).
Taking this situation further results in a truly corrupted WC:

- resolve the tree conflict
- commit

The committed result (svn cat <URL>) is different from the on-disk contents,
even though the file is claimed to be up-to-date.

Trunk doesn't show this behavior (thumbs up for wc-ng), but we should try to
find a fix for 1.6.x.

stsp, you wanna hunt this down?

Thank you very much, Dave! Seriously bad mojo, as you said. Good work!!
...even though it took ages to deflate your darn typescript ;)

~Neels

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Neels J Hofmeyr <ne...@elego.de>.
On 08/22/2011 05:56 PM, David Wallace wrote:
> Hi, Neels.  Thanks for the suggestions.  It will probably be a few days before I have time to get to this.
> 
> Dave Wallace

Ok, I or Stefan might be faster than that. Take a look whether we were
before you start...

Thanks!

~Neeks


RE: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by David Wallace <da...@bluepearlsoftware.com>.
Hi, Neels.  Thanks for the suggestions.  It will probably be a few days before I have time to get to this.

Dave Wallace

-----Original Message-----
From: Neels J Hofmeyr [mailto:neels@elego.de] 
Sent: Saturday, August 20, 2011 7:18 PM
To: dev@subversion.apache.org; David Wallace
Subject: Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Hi Dave,

if you've found an inconsistency in Subversion, this is highly interesting information! The typescript you've attached is certainly useful. But (I'm sorry to nitpick more) there is a lot of noise in the file and even after scanning for minutes I have not spotted anything unusual except the final checksum error.

You've already put a lot of effort in this, and I can certainly translate your typescript myself. But Stefan would have to do the same, and others too. So, if you still have more time to spare and are willing to do so, it would be very helpful if you could distill the reproduction into a runnable script, that, for added bonus, is as minimal as you can get it while still producing the error every time.

I have attached a test script blueprint that I use to write my tests. Where it says "# ACTUAL TEST" you can simply start hacking svn add,commit,... I've included a few basic examples there for you to get started easily.

This test script creates a new temp folder under /tmp and creates a repository and a working copy all inside, and does a cd into the working copy ('cd "$WC"'). All you do is write your svn commands, then run it. It uses simple file:/// URLs to access the repository, so it cuts out all network and authentication issues. So if you get valid commands to fail there, it's definitely Subversion's fault.

BTW, I usually have just one large working copy of all branches in a small test like this, unless I want to test the checkout command. But if you need them, there are also examples of creating more working copies in there.

When you're done testing it, run your script like  ./my_test >my_test.output 2>&1 and attach both test script and the output file.

For illustration I've done just that.

Oh, and, it would be sufficient to write the test up to the point where, as you say:

>> So now I have two different
>> working copies of branch2, supposedly updated to the same version, 
>> with two different versions of the files, and both insisting that 
>> they are fully up to date!

No need to try to resolve that any further in the test, as this alone would be catastrophic enough ;)

So, if you can spare more time on this issue, we would very much appreciate your effort!

Many thanks,
~Neels



On 08/19/2011 12:02 AM, David Wallace wrote:
> Ok, I've attached a filtered typescript in which I was able to reproduce the problem, using a subtree in our repository as if it was the full repository.   I don't have a full reproduction script, because that would require me to (1) set up a new repository and (2) parse the output from svn log to get the proper range of revision numbers to attach to subsequent commands.  But this typescript should be pretty easy to follow manually.
> 
> I trust that my comments in the script should illustrate why I think the behavior is a major bug, but to summarize:
> 1) I have created a scenario following a fairly normal workflow, in 
> which I have checked in changes from a working directory which 
> Subversion then reports as being up to date with the repository, when 
> in fact some of the changes have not actually hit the repository, and
> 2) Subsequent changes to the file in question from another working directory cause updates on my original working directory to fail with a checksum error, potentially causing loss of data from any subsequent work I may have done in that working directory, thinking it was properly up to date.
> 
> Data integrity is a pretty important part of any source code control system: the promise is that your data is safe once you check it in to the repository.  This scenario represents a violation of that promise.
> 
> Dave Wallace
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: Thursday, August 18, 2011 3:16 AM
> To: David Wallace
> Cc: dev@subversion.apache.org
> Subject: Re: Resoving tree conflicts results in inconsistent state 
> between two working copies of same branch
> 
> On Wed, Aug 17, 2011 at 11:38:57PM +0000, David Wallace wrote:
>> I just described my experience inadvertently creating an inconsistent 
>> state between two working copies of the same branch while trying to 
>> resolve tree conflicts over at:
>> http://stackoverflow.com/questions/767763/svn-how-to-resolve-new-tree
>> -
>> conflicts-when-file-is-added-on-two-branches/7100512#7100512
>> To make this more self-contained, I'll repeat the key text here:
>>
>> (1): I had added some files while working on my initial branch, 
>> branch1; (2) I created a new branch, branch2 for further development, 
>> branching it off from the trunk and then merging in my changes from
>> branch1 (3) A co-worker had copied my mods from branch1 to his own 
>> branch, added further mods, and then merged back to the trunk; (4) I 
>> now wanted to merge the latest changes from trunk into my current 
>> working branch, branch2. This is with svn 1.6.17.
>>
>> The merge had tree conflicts with the new files, and I wanted the new 
>> version from the trunk where they differed, so from a clean copy of 
>> branch2, I did an svn delete of the conflicting files, committed 
>> these
>> branch2 changes (thus creating a temporary version of branch2 without 
>> the files in question), and then did my merge from the trunk. I did 
>> this because I wanted the history to match the trunk version so that 
>> I wouldn't have more problems later when trying to merge back to trunk.
>>
>> Merge went fine, I got the trunk version of the files, svn st shows 
>> all ok, and then I hit more tree conflicts while trying to commit the 
>> changes, between the delete I had done earlier and the add from the 
>> merge.
>>
>> Did an svn resolve of the conflicts in favor of my working copy 
>> (which now had the trunk version of the files), and got it to commit.
>> All should be good, right?
>>
>> Well, no. An update of another copy of branch2 resulted in the old 
>> version of the files (pre-trunk merge). So now I have two different 
>> working copies of branch2, supposedly updated to the same version, 
>> with two different versions of the files, and both insisting that 
>> they are fully up to date!
>>
>> Checking out a clean copy of branch2 resulted in the old (pre-trunk) 
>> version of the files. I manually update these to the trunk version 
>> and commit the changes, go back to my first working copy (from which 
>> I had submitted the trunk changes originally), try to update it, and 
>> now get a checksum error on the files in question.
>>
>> Blow the directory in question away, get a new version via update, 
>> and finally I have what should be a good version of branch2 with the 
>> trunk changes. I hope.
>>
>> Further notes:  Our standard working practice is that all merges 
>> except merges from the trunk are done with a specific range of 
>> revision numbers, getting the relevant revisions via svn log 
>> -stop-on-copy on the appropriate branch.  This applied to my merge 
>> from branch1 to branch2, my co-worker's merge from my branch1 to his 
>> branch, and his merge from his branch back to the trunk, but not to 
>> my merges from the trunk described above.  I'm not sure if this 
>> matters, but I report it here in case it makes a difference when 
>> trying to reproduce the problem.
> 
> Hi David,
> 
> you've provided a long explanation of a problem which is very complicated and very hard to reproduce remotely. I am also not sure which part of Subversion's behaviour you consider a bug. There are certainly annoying problems in the situations you describe but where do you think the fault lies with Subversion exactly?
> 
> Long prose is not an effective medium for communicating problem reports and reproduction recipes via email. Prose lacks detail and is ambiguous.
> Prose works well when talking to someone next to you who can ask questions back, look at your screen, and maybe even try out some things on your computer. But it is very hard for recipients of email to guess what really happened on your computer. So if you consider your email a bug report then please follow the guidelines on this page:
> http://subversion.apache.org/docs/community-guide/issues.html#reportin
> g-bugs
> 
> Apart from clearly explaining what you consider a bug, please provide a full command line transcript of the situation you are describing.
> Even better would be a full reproduction script starting off with an empty repository and running a sequence of commands that trigger the problem.
> The time you spend on this will be worth it. Thanks!

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Neels J Hofmeyr <ne...@elego.de>.
Hi Dave,

if you've found an inconsistency in Subversion, this is highly interesting
information! The typescript you've attached is certainly useful. But (I'm
sorry to nitpick more) there is a lot of noise in the file and even after
scanning for minutes I have not spotted anything unusual except the final
checksum error.

You've already put a lot of effort in this, and I can certainly translate
your typescript myself. But Stefan would have to do the same, and others
too. So, if you still have more time to spare and are willing to do so, it
would be very helpful if you could distill the reproduction into a runnable
script, that, for added bonus, is as minimal as you can get it while still
producing the error every time.

I have attached a test script blueprint that I use to write my tests. Where
it says "# ACTUAL TEST" you can simply start hacking svn add,commit,... I've
included a few basic examples there for you to get started easily.

This test script creates a new temp folder under /tmp and creates a
repository and a working copy all inside, and does a cd into the working
copy ('cd "$WC"'). All you do is write your svn commands, then run it. It
uses simple file:/// URLs to access the repository, so it cuts out all
network and authentication issues. So if you get valid commands to fail
there, it's definitely Subversion's fault.

BTW, I usually have just one large working copy of all branches in a small
test like this, unless I want to test the checkout command. But if you need
them, there are also examples of creating more working copies in there.

When you're done testing it, run your script like
 ./my_test >my_test.output 2>&1
and attach both test script and the output file.

For illustration I've done just that.

Oh, and, it would be sufficient to write the test up to the point where, as
you say:

>> So now I have two different
>> working copies of branch2, supposedly updated to the same version,
>> with two different versions of the files, and both insisting that they
>> are fully up to date!

No need to try to resolve that any further in the test, as this alone would
be catastrophic enough ;)

So, if you can spare more time on this issue, we would very much appreciate
your effort!

Many thanks,
~Neels



On 08/19/2011 12:02 AM, David Wallace wrote:
> Ok, I've attached a filtered typescript in which I was able to reproduce the problem, using a subtree in our repository as if it was the full repository.   I don't have a full reproduction script, because that would require me to (1) set up a new repository and (2) parse the output from svn log to get the proper range of revision numbers to attach to subsequent commands.  But this typescript should be pretty easy to follow manually.
> 
> I trust that my comments in the script should illustrate why I think the behavior is a major bug, but to summarize:
> 1) I have created a scenario following a fairly normal workflow, in which I have checked in changes from a working directory which Subversion then reports as being up to date with the repository, when in fact some of the changes have not actually hit the repository, and
> 2) Subsequent changes to the file in question from another working directory cause updates on my original working directory to fail with a checksum error, potentially causing loss of data from any subsequent work I may have done in that working directory, thinking it was properly up to date.
> 
> Data integrity is a pretty important part of any source code control system: the promise is that your data is safe once you check it in to the repository.  This scenario represents a violation of that promise.
> 
> Dave Wallace
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de] 
> Sent: Thursday, August 18, 2011 3:16 AM
> To: David Wallace
> Cc: dev@subversion.apache.org
> Subject: Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch
> 
> On Wed, Aug 17, 2011 at 11:38:57PM +0000, David Wallace wrote:
>> I just described my experience inadvertently creating an inconsistent 
>> state between two working copies of the same branch while trying to 
>> resolve tree conflicts over at:
>> http://stackoverflow.com/questions/767763/svn-how-to-resolve-new-tree-
>> conflicts-when-file-is-added-on-two-branches/7100512#7100512
>> To make this more self-contained, I'll repeat the key text here:
>>
>> (1): I had added some files while working on my initial branch, 
>> branch1; (2) I created a new branch, branch2 for further development, 
>> branching it off from the trunk and then merging in my changes from
>> branch1 (3) A co-worker had copied my mods from branch1 to his own 
>> branch, added further mods, and then merged back to the trunk; (4) I 
>> now wanted to merge the latest changes from trunk into my current 
>> working branch, branch2. This is with svn 1.6.17.
>>
>> The merge had tree conflicts with the new files, and I wanted the new 
>> version from the trunk where they differed, so from a clean copy of 
>> branch2, I did an svn delete of the conflicting files, committed these
>> branch2 changes (thus creating a temporary version of branch2 without 
>> the files in question), and then did my merge from the trunk. I did 
>> this because I wanted the history to match the trunk version so that I 
>> wouldn't have more problems later when trying to merge back to trunk.
>>
>> Merge went fine, I got the trunk version of the files, svn st shows 
>> all ok, and then I hit more tree conflicts while trying to commit the 
>> changes, between the delete I had done earlier and the add from the 
>> merge.
>>
>> Did an svn resolve of the conflicts in favor of my working copy (which 
>> now had the trunk version of the files), and got it to commit.
>> All should be good, right?
>>
>> Well, no. An update of another copy of branch2 resulted in the old 
>> version of the files (pre-trunk merge). So now I have two different 
>> working copies of branch2, supposedly updated to the same version, 
>> with two different versions of the files, and both insisting that they 
>> are fully up to date!
>>
>> Checking out a clean copy of branch2 resulted in the old (pre-trunk) 
>> version of the files. I manually update these to the trunk version and 
>> commit the changes, go back to my first working copy (from which I had 
>> submitted the trunk changes originally), try to update it, and now get 
>> a checksum error on the files in question.
>>
>> Blow the directory in question away, get a new version via update, and 
>> finally I have what should be a good version of branch2 with the trunk 
>> changes. I hope.
>>
>> Further notes:  Our standard working practice is that all merges 
>> except merges from the trunk are done with a specific range of 
>> revision numbers, getting the relevant revisions via svn log 
>> -stop-on-copy on the appropriate branch.  This applied to my merge 
>> from branch1 to branch2, my co-worker's merge from my branch1 to his 
>> branch, and his merge from his branch back to the trunk, but not to my 
>> merges from the trunk described above.  I'm not sure if this matters, 
>> but I report it here in case it makes a difference when trying to 
>> reproduce the problem.
> 
> Hi David,
> 
> you've provided a long explanation of a problem which is very complicated and very hard to reproduce remotely. I am also not sure which part of Subversion's behaviour you consider a bug. There are certainly annoying problems in the situations you describe but where do you think the fault lies with Subversion exactly?
> 
> Long prose is not an effective medium for communicating problem reports and reproduction recipes via email. Prose lacks detail and is ambiguous.
> Prose works well when talking to someone next to you who can ask questions back, look at your screen, and maybe even try out some things on your computer. But it is very hard for recipients of email to guess what really happened on your computer. So if you consider your email a bug report then please follow the guidelines on this page:
> http://subversion.apache.org/docs/community-guide/issues.html#reporting-bugs
> 
> Apart from clearly explaining what you consider a bug, please provide a full command line transcript of the situation you are describing.
> Even better would be a full reproduction script starting off with an empty repository and running a sequence of commands that trigger the problem.
> The time you spend on this will be worth it. Thanks!

RE: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by David Wallace <da...@bluepearlsoftware.com>.
Thanks, Stefan.  In addition to the specific bug issues I mentioned below, there's also the design issue of whether this should properly be considered a tree conflict at all.  After all, file2.txt was originally added in changeset 8036 back on davebr1, and every other branch or trunk that subsequently adds the file does so by merging in a changeset that either directly or indirectly traces back to changeset 8036.  So at least in theory, it should be possible for subversion to realize "Oh, this is all the same file that was originally added back in 8036" and keep the histories straight.  But I realize that it might require significant architectural changes to the tool to get this to happen.

Dave Wallace

-----Original Message-----
From: Stefan Sperling [mailto:stsp@elego.de] 
Sent: Thursday, August 18, 2011 3:43 PM
To: David Wallace
Cc: dev@subversion.apache.org
Subject: Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

On Thu, Aug 18, 2011 at 10:02:45PM +0000, David Wallace wrote:
> Ok, I've attached a filtered typescript in which I was able to 
> reproduce the problem, using a subtree in our repository as if it was 
> the full repository.

Thanks!

> I don't have a full reproduction script, because that would require me 
> to (1) set up a new repository

You can create a temporary repository for experiments on the local hard drive like this:

  svnadmin create /tmp/repos

and then check out a working copy:

  svn co file:///tmp/repos wc

There is no need to set it up a repository on your Subversion server or even use your production repository for such things.

> and (2)
> parse the output from svn log to get the proper range of revision 
> numbers to attach to subsequent commands.  But this typescript should 
> be pretty easy to follow manually.

It is quite long. At first glance it looks like a bug. I don't know if it is a known bug. I'll try to find some time to look at in detail next week.

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Stefan Sperling <st...@elego.de>.
On Thu, Aug 18, 2011 at 10:02:45PM +0000, David Wallace wrote:
> Ok, I've attached a filtered typescript in which I was able to
> reproduce the problem, using a subtree in our repository as if it was
> the full repository.

Thanks!

> I don't have a full reproduction script,
> because that would require me to (1) set up a new repository

You can create a temporary repository for experiments on the local
hard drive like this:

  svnadmin create /tmp/repos

and then check out a working copy:

  svn co file:///tmp/repos wc

There is no need to set it up a repository on your Subversion server
or even use your production repository for such things.

> and (2)
> parse the output from svn log to get the proper range of revision
> numbers to attach to subsequent commands.  But this typescript should
> be pretty easy to follow manually.

It is quite long. At first glance it looks like a bug. I don't
know if it is a known bug. I'll try to find some time to look at
in detail next week.

RE: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by David Wallace <da...@bluepearlsoftware.com>.
Ok, I've attached a filtered typescript in which I was able to reproduce the problem, using a subtree in our repository as if it was the full repository.   I don't have a full reproduction script, because that would require me to (1) set up a new repository and (2) parse the output from svn log to get the proper range of revision numbers to attach to subsequent commands.  But this typescript should be pretty easy to follow manually.

I trust that my comments in the script should illustrate why I think the behavior is a major bug, but to summarize:
1) I have created a scenario following a fairly normal workflow, in which I have checked in changes from a working directory which Subversion then reports as being up to date with the repository, when in fact some of the changes have not actually hit the repository, and
2) Subsequent changes to the file in question from another working directory cause updates on my original working directory to fail with a checksum error, potentially causing loss of data from any subsequent work I may have done in that working directory, thinking it was properly up to date.

Data integrity is a pretty important part of any source code control system: the promise is that your data is safe once you check it in to the repository.  This scenario represents a violation of that promise.

Dave Wallace
-----Original Message-----
From: Stefan Sperling [mailto:stsp@elego.de] 
Sent: Thursday, August 18, 2011 3:16 AM
To: David Wallace
Cc: dev@subversion.apache.org
Subject: Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

On Wed, Aug 17, 2011 at 11:38:57PM +0000, David Wallace wrote:
> I just described my experience inadvertently creating an inconsistent 
> state between two working copies of the same branch while trying to 
> resolve tree conflicts over at:
> http://stackoverflow.com/questions/767763/svn-how-to-resolve-new-tree-
> conflicts-when-file-is-added-on-two-branches/7100512#7100512
> To make this more self-contained, I'll repeat the key text here:
> 
> (1): I had added some files while working on my initial branch, 
> branch1; (2) I created a new branch, branch2 for further development, 
> branching it off from the trunk and then merging in my changes from
> branch1 (3) A co-worker had copied my mods from branch1 to his own 
> branch, added further mods, and then merged back to the trunk; (4) I 
> now wanted to merge the latest changes from trunk into my current 
> working branch, branch2. This is with svn 1.6.17.
>
> The merge had tree conflicts with the new files, and I wanted the new 
> version from the trunk where they differed, so from a clean copy of 
> branch2, I did an svn delete of the conflicting files, committed these
> branch2 changes (thus creating a temporary version of branch2 without 
> the files in question), and then did my merge from the trunk. I did 
> this because I wanted the history to match the trunk version so that I 
> wouldn't have more problems later when trying to merge back to trunk.
>
> Merge went fine, I got the trunk version of the files, svn st shows 
> all ok, and then I hit more tree conflicts while trying to commit the 
> changes, between the delete I had done earlier and the add from the 
> merge.
>
> Did an svn resolve of the conflicts in favor of my working copy (which 
> now had the trunk version of the files), and got it to commit.
> All should be good, right?
> 
> Well, no. An update of another copy of branch2 resulted in the old 
> version of the files (pre-trunk merge). So now I have two different 
> working copies of branch2, supposedly updated to the same version, 
> with two different versions of the files, and both insisting that they 
> are fully up to date!
>
> Checking out a clean copy of branch2 resulted in the old (pre-trunk) 
> version of the files. I manually update these to the trunk version and 
> commit the changes, go back to my first working copy (from which I had 
> submitted the trunk changes originally), try to update it, and now get 
> a checksum error on the files in question.
>
> Blow the directory in question away, get a new version via update, and 
> finally I have what should be a good version of branch2 with the trunk 
> changes. I hope.
>
> Further notes:  Our standard working practice is that all merges 
> except merges from the trunk are done with a specific range of 
> revision numbers, getting the relevant revisions via svn log 
> -stop-on-copy on the appropriate branch.  This applied to my merge 
> from branch1 to branch2, my co-worker's merge from my branch1 to his 
> branch, and his merge from his branch back to the trunk, but not to my 
> merges from the trunk described above.  I'm not sure if this matters, 
> but I report it here in case it makes a difference when trying to 
> reproduce the problem.

Hi David,

you've provided a long explanation of a problem which is very complicated and very hard to reproduce remotely. I am also not sure which part of Subversion's behaviour you consider a bug. There are certainly annoying problems in the situations you describe but where do you think the fault lies with Subversion exactly?

Long prose is not an effective medium for communicating problem reports and reproduction recipes via email. Prose lacks detail and is ambiguous.
Prose works well when talking to someone next to you who can ask questions back, look at your screen, and maybe even try out some things on your computer. But it is very hard for recipients of email to guess what really happened on your computer. So if you consider your email a bug report then please follow the guidelines on this page:
http://subversion.apache.org/docs/community-guide/issues.html#reporting-bugs

Apart from clearly explaining what you consider a bug, please provide a full command line transcript of the situation you are describing.
Even better would be a full reproduction script starting off with an empty repository and running a sequence of commands that trigger the problem.
The time you spend on this will be worth it. Thanks!

Re: Resoving tree conflicts results in inconsistent state between two working copies of same branch

Posted by Stefan Sperling <st...@elego.de>.
On Wed, Aug 17, 2011 at 11:38:57PM +0000, David Wallace wrote:
> I just described my experience inadvertently creating an inconsistent
> state between two working copies of the same branch while trying to
> resolve tree conflicts over at:
> http://stackoverflow.com/questions/767763/svn-how-to-resolve-new-tree-conflicts-when-file-is-added-on-two-branches/7100512#7100512
> To make this more self-contained, I'll repeat the key text here:
> 
> (1): I had added some files while working on my initial branch,
> branch1; (2) I created a new branch, branch2 for further development,
> branching it off from the trunk and then merging in my changes from
> branch1 (3) A co-worker had copied my mods from branch1 to his own
> branch, added further mods, and then merged back to the trunk; (4) I
> now wanted to merge the latest changes from trunk into my current
> working branch, branch2. This is with svn 1.6.17.
>
> The merge had tree conflicts with the new files, and I wanted the new
> version from the trunk where they differed, so from a clean copy of
> branch2, I did an svn delete of the conflicting files, committed these
> branch2 changes (thus creating a temporary version of branch2 without
> the files in question), and then did my merge from the trunk. I did
> this because I wanted the history to match the trunk version so that I
> wouldn't have more problems later when trying to merge back to trunk.
>
> Merge went fine, I got the trunk version of the files, svn st shows
> all ok, and then I hit more tree conflicts while trying to commit the
> changes, between the delete I had done earlier and the add from the
> merge.
>
> Did an svn resolve of the conflicts in favor of my working copy
> (which now had the trunk version of the files), and got it to commit.
> All should be good, right?
> 
> Well, no. An update of another copy of branch2 resulted in the old
> version of the files (pre-trunk merge). So now I have two different
> working copies of branch2, supposedly updated to the same version,
> with two different versions of the files, and both insisting that they
> are fully up to date!
>
> Checking out a clean copy of branch2 resulted in
> the old (pre-trunk) version of the files. I manually update these to
> the trunk version and commit the changes, go back to my first working
> copy (from which I had submitted the trunk changes originally), try to
> update it, and now get a checksum error on the files in question.
>
> Blow the directory in question away, get a new version via update, and
> finally I have what should be a good version of branch2 with the trunk
> changes. I hope.
>
> Further notes:  Our standard working practice is that all merges
> except merges from the trunk are done with a specific range of
> revision numbers, getting the relevant revisions via svn log
> -stop-on-copy on the appropriate branch.  This applied to my merge
> from branch1 to branch2, my co-worker's merge from my branch1 to his
> branch, and his merge from his branch back to the trunk, but not to my
> merges from the trunk described above.  I'm not sure if this matters,
> but I report it here in case it makes a difference when trying to
> reproduce the problem.

Hi David,

you've provided a long explanation of a problem which is very complicated
and very hard to reproduce remotely. I am also not sure which part of
Subversion's behaviour you consider a bug. There are certainly annoying
problems in the situations you describe but where do you think the fault
lies with Subversion exactly?

Long prose is not an effective medium for communicating problem reports
and reproduction recipes via email. Prose lacks detail and is ambiguous.
Prose works well when talking to someone next to you who can ask questions
back, look at your screen, and maybe even try out some things on your
computer. But it is very hard for recipients of email to guess what
really happened on your computer. So if you consider your email a bug
report then please follow the guidelines on this page:
http://subversion.apache.org/docs/community-guide/issues.html#reporting-bugs

Apart from clearly explaining what you consider a bug, please provide a
full command line transcript of the situation you are describing.
Even better would be a full reproduction script starting off with an
empty repository and running a sequence of commands that trigger the problem.
The time you spend on this will be worth it. Thanks!