You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Ryan Schmidt <su...@ryandesign.com> on 2017/10/15 17:49:53 UTC
GitHub svn bridge corrupting working copies
Hi, I'm sure this is GitHub's problem, because it happens only when accessing a git repository through the GitHub Subversion bridge using the Subversion client, but it's been months since I reported the problem to them and they haven't fixed it so I thought I'd ask here if anybody has any idea how to try to debug this so that I can give them some better information to work with.
I'm using Subversion 1.9.7 installed with MacPorts on macOS 10.12.6 "Sierra".
It's a working copy of the macports-ports repository, checked out as:
svn checkout https://github.com/macports/macports-ports/trunk macports-ports-trunk
Everything is fine for a random number of days, and then at some point "svn update" fails with a message like this:
svn: E155010: The node '/path/to/macports-ports-trunk/kdepimlibs4' was not found.
There is not supposed to be any kdepimlibs4 directly inside trunk. In fact, it's inside the kde directory which is in trunk.
I can successfully "svn update" all the top-level directories except for kde. I can commit from other directories too. But to finally fix the problem, I have to throw away the working copy and check out a new one. This is inconvenient because it's the one I do all my work in.
And the problem will just occur again sometime later with a different node. The node it complains about is always a directory that someone else committed to recently, and is one that is supposed to be inside one of the top-level directories, but the error message always shows it is looking for it directly in trunk.
Doing work in the working copy is not required for the corruption to occur. Until recently, we were using such an svn working copy in a server-side script which ran every half hour and would only ever "svn update" it, then index it and copy it to an rsync server; this would corrupt itself regularly, sometimes more than once a day.
I can "rm -rf kde && svn update kde" it, which restores it from the pristines, and then again fails to update it because of the corrupted node.
I assume the GitHub svn bridge has delivered an incorrect response to the client, the results of which have been recorded somewhere in the .svn directory, presumably inside wc.db because that's the only file in there, other than pristines, which matches a grep for "kdepimlibs4". I've poked around at it in an SQLite viewer, but searching for records where local_relpath, parent_relpath, or repos_path contain "kdepimlibs4" hasn't revealed any entries where it's not prefixed with "kde".
I haven't tried to capture the network traffic. I haven't used such tools before, so I'm unfamiliar with them, and I can't reproduce the problem on demand so it seems problematic to try capturing all network traffic for days until the problem happens to occur.
Thanks for any suggestions you might have! (Other than "use the git client instead"; I've tried that for months and it doesn't seem to be compatible with my brain.)
Re: GitHub svn bridge corrupting working copies
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 16, 2017, at 08:01, Bert Huijben wrote:
> One probable cause for this would be that they somehow changed the revision number to hash mapping. I would hope they change the repository UUID in this case, but given how easy it is to change history in git, I wouldn't count on this.
>
> Is this problem reproducible on a clean checkout from the same base revision?
The repository UUID does not change.
We don't allow changing history -- GitHub is configured to prohibit force-pushing to master -- so I am certain that is not occurring. We do allow force-pushing to pull requests before they're merged, but AFAIK pull requests are not stored in the main git repository so that shouldn't be a problem.
The same node corruption is not necessarily reproducible when a new working copy is checked out, but a different corruption occurs sometime later. Or, sometimes corruption occurs partway through the initial checkout, and trying another checkout a moment later works fine.
But I have several working copies of macports-ports that I have been using on my machine for awhile, and I have seen the some of the same corruption in multiple working copies. For example, a couple days ago, I updated one working copy and had problems with, among other directories, TeXShop3 and msp430-gcc-devel. I fixed that working copy by "svn up"ing the parent directory of those problem directories to a prior revision, then "svn up"ing back to HEAD. And now I have those same directories -- and others -- giving me problems in a separate working copy that had not been updated for awhile.
I'm attaching a terminal transcript of that session. It's long because the "svn up" pulled in 2 weeks worth of changes, and Subversion was compiled with the debugging change Daniel suggested. I took a working copy that had no uncommitted changes, and was last used to update boost and its dependents on November 10 (https://trac.macports.org/ticket/50671#comment:69) and ran "svn up". After pulling in some changes, it ended with "svn: E175002: REPORT request on '/macports/macports-ports/!svn/vcc/default' failed". "svn st" still showed nothing. Re-attempting "svn up" failed after 2 minutes with "svn: E175002: Unexpected HTTP status 504 'Gateway Timeout' on '/macports/macports-ports/!svn/vcc/default'". (GitHub support previously explained that when this happens, Subversion for whatever reason needs to compare all files in the working copy with the server, and the server takes too long to do that.) Then I ran "svn up" individually on all top-level items which revealed 19 separate directories experiencing the corruption (all of which are shown being updated in the first "svn up"). (But hundreds of other items were updated without complaint.)
Could you look at the debug output in the transcript and see if anything jumps out at you that would explain what's different about those 19 changes? If not, I wonder if corruption already happened earlier, and I should be making a new working copy, and occasionally running "svn up" on it, and saving all of the transcripts for later analysis.
On Oct 15, 2017, at 14:28, Daniel Shahaf wrote:
> Ryan Schmidt wrote on Sun, Oct 15, 2017 at 14:00:10 -0500:
>
>> Inside the kde directory, I ran:
> ⋮
>> $ svn up -r 139308
>> Updating '.':
>> UU kdepimlibs4/Portfile
>> Skipped 'kdepimlibs4/files/patch-do_not_include_cpp.diff' -- An obstructing working copy was found
>
> That's interesting; why would there be an obstruction?
>
> Maybe a file by this name already existed at some point, and was not removed cleanly?
>
> Or perhaps github reported the 'add' to the client twice?
>
> On your build server you can try recording the output of 'svn status --no-ignore' before the update; that will show whether the file existed before the update.
Oh strange. In this working copy (not the one I submitted the transcript for above) I've just found this patchfile -- and another one -- elsewhere in the working copy. The client thinks it belongs there -- "svn st" shows nothing, and "svn info" shows information -- but "svn log" shows the server doesn't know about it, as of course it shouldn't, since that's not where it is on the server.
$ svn st patch-do_not_include_cpp.diff
$ svn info patch-do_not_include_cpp.diff
Path: patch-do_not_include_cpp.diff
Name: patch-do_not_include_cpp.diff
Working Copy Root Path: /Users/rschmidt/macports/macports-ports-svn-trunk
URL: https://github.com/macports/macports-ports/trunk/patch-do_not_include_cpp.diff
Relative URL: ^/trunk/patch-do_not_include_cpp.diff
Repository Root: https://github.com/macports/macports-ports
Repository UUID: 0bf748bf-1d32-881d-ce5d-ce57b7db7bff
Revision: 140783
Node Kind: file
Schedule: normal
Last Changed Author: nicolas.pavillon
Last Changed Rev: 139308
Last Changed Date: 2017-10-13 08:22:43 -0500 (Fri, 13 Oct 2017)
Text Last Updated: 2017-10-15 13:35:55 -0500 (Sun, 15 Oct 2017)
Checksum: 2a8a228c0febb014a18a90066cc944c43e70e89e
$ svn log patch-do_not_include_cpp.diff
svn: E160013: '/macports/macports-ports/!svn/bc/140783/trunk/patch-do_not_include_cpp.diff' path not found
$ cd net
$ svn st patch-plugins-check_load.c.diff
$ svn info patch-plugins-check_load.c.diff
Path: patch-plugins-check_load.c.diff
Name: patch-plugins-check_load.c.diff
Working Copy Root Path: /Users/rschmidt/macports/macports-ports-svn-trunk
URL: https://github.com/macports/macports-ports/trunk/net/patch-plugins-check_load.c.diff
Relative URL: ^/trunk/net/patch-plugins-check_load.c.diff
Repository Root: https://github.com/macports/macports-ports
Repository UUID: 0bf748bf-1d32-881d-ce5d-ce57b7db7bff
Revision: 140783
Node Kind: file
Schedule: normal
Last Changed Author: nicolas.pavillon
Last Changed Rev: 140759
Last Changed Date: 2017-11-22 22:33:27 -0600 (Wed, 22 Nov 2017)
Text Last Updated: 2017-11-22 22:57:35 -0600 (Wed, 22 Nov 2017)
Checksum: 8618a8b2da28a5123c6591314d67c537d9e43d4f
$ svn log patch-plugins-check_load.c.diff
svn: E160013: '/macports/macports-ports/!svn/bc/140783/trunk/net/patch-plugins-check_load.c.diff' path not found
RE: GitHub svn bridge corrupting working copies
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Daniel Shahaf [mailto:d.s@daniel.shahaf.name]
> Sent: zondag 15 oktober 2017 21:28
> To: Ryan Schmidt <su...@ryandesign.com>
> Cc: Subversion Users <us...@subversion.apache.org>
> Subject: Re: GitHub svn bridge corrupting working copies
>
> Ryan Schmidt wrote on Sun, Oct 15, 2017 at 14:00:10 -0500:
> > On Oct 15, 2017, at 13:07, Daniel Shahaf wrote:
> > > Ryan Schmidt wrote on Sun, 15 Oct 2017 12:49 -0500:
> > >> And the problem will just occur again sometime later with a different
> node. The node it complains about is always a directory that someone else
> committed to recently, [...]
> > >
> > > Have you looked at these commits? Is there anything unusual in them,
> > > e.g., do they involve renames?
> >
> > Nothing unusual; just modifying one file and adding another:
> >
> > https://github.com/macports/macports-
> ports/commit/2198debd25b02bcf7f6b17f79b415690449a85f0
>
> Okay, so perhaps the problem has to do with revisions that add new
> files.
>
> > Somehow I don't think specific revisions are the problem. I mean, I
> > can now check out a new working copy at revision 139307 and
> > successfully update it to 139308. Meanwhile, on our server, once,
> > back in August, it encountered a corruption already halfway through
> > the process of checking out a new working copy. And on the next
> > checkout attempt it worked.
>
> Indeed, there is little we can do without knowledge of the bridge.
> There is any number of ways in which these observations might be
> confounders.
>
> > > You might be able to reproduce the problem by backdating your working
> > > copy, 'svn up -r N-1 && svn up -r N kde'.
> >
> > Inside the kde directory, I ran:
> ⋮
> > $ svn up -r 139308
> > Updating '.':
> > UU kdepimlibs4/Portfile
> > Skipped 'kdepimlibs4/files/patch-do_not_include_cpp.diff' -- An
> obstructing working copy was found
>
> That's interesting; why would there be an obstruction?
>
> Maybe a file by this name already existed at some point, and was not
> removed cleanly?
>
> Or perhaps github reported the 'add' to the client twice?
One probable cause for this would be that they somehow changed the revision number to hash mapping. I would hope they change the repository UUID in this case, but given how easy it is to change history in git, I wouldn't count on this.
Is this problem reproducible on a clean checkout from the same base revision?
Bert
Re: GitHub svn bridge corrupting working copies
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Ryan Schmidt wrote on Sun, Oct 15, 2017 at 14:00:10 -0500:
> On Oct 15, 2017, at 13:07, Daniel Shahaf wrote:
> > Ryan Schmidt wrote on Sun, 15 Oct 2017 12:49 -0500:
> >> And the problem will just occur again sometime later with a different node. The node it complains about is always a directory that someone else committed to recently, [...]
> >
> > Have you looked at these commits? Is there anything unusual in them,
> > e.g., do they involve renames?
>
> Nothing unusual; just modifying one file and adding another:
>
> https://github.com/macports/macports-ports/commit/2198debd25b02bcf7f6b17f79b415690449a85f0
Okay, so perhaps the problem has to do with revisions that add new
files.
> Somehow I don't think specific revisions are the problem. I mean, I
> can now check out a new working copy at revision 139307 and
> successfully update it to 139308. Meanwhile, on our server, once,
> back in August, it encountered a corruption already halfway through
> the process of checking out a new working copy. And on the next
> checkout attempt it worked.
Indeed, there is little we can do without knowledge of the bridge.
There is any number of ways in which these observations might be
confounders.
> > You might be able to reproduce the problem by backdating your working
> > copy, 'svn up -r N-1 && svn up -r N kde'.
>
> Inside the kde directory, I ran:
⋮
> $ svn up -r 139308
> Updating '.':
> UU kdepimlibs4/Portfile
> Skipped 'kdepimlibs4/files/patch-do_not_include_cpp.diff' -- An obstructing working copy was found
That's interesting; why would there be an obstruction?
Maybe a file by this name already existed at some point, and was not
removed cleanly?
Or perhaps github reported the 'add' to the client twice?
On your build server you can try recording the output of 'svn status
--no-ignore' before the update; that will show whether the file existed
before the update.
> Updated to revision 139308.
> Summary of conflicts:
> Skipped paths: 1
>
> $ svn status
>
> $ rm -rf kdepimlibs4/files/patch-do_not_include_cpp.diff
>
> $ svn up -r 139308
> Updating '.':
> Restored 'kdepimlibs4/files/patch-do_not_include_cpp.diff'
> A /path/to/macports-ports-trunk/patch-do_not_include_cpp.diff
> Updated to revision 139308.
>
> $ svn up
> Updating '.':
> At revision 139412.
>
>
> >> Thanks for any suggestions you might have! (Other than "use the git client instead"; I've tried that for months and it doesn't seem to be compatible with my brain.)
> >
> > If you're comfortable with rebuilding, you can try wrapping the "debug
> > editor" (svn_delta__get_debug_editor()) around the update editor
> > (svn_ra_do_update3()) to get a dump of the changes the server reports to
> > the client. That debug output would be at a higher level than a wire dump.
>
> I'd like to try this but I'm not very familiar with the svn code. In
> subversion/libsvn_client/update.c I replaced
> "SVN_ERR(svn_ra_do_update3(...))" with
> "SVN_ERR(svn_delta__get_debug_editor(svn_ra_do_update3(...)))" but now
> svn segfaults when I update, so that must not have been right.
That should have produced a compiler warning to the effect that
svn_ra_do_update3()'s return type is different to the type of the first
formal argument to svn_ra_do_update3().
> Could you give more guidance on how to do this properly?
Certainly. Try this:
[[[
--- subversion/libsvn_ra/ra_loader.c 2017-08-09 19:58:28.925998170 +0000
+++ - 2017-10-15 19:14:31.505187973 +0000
@@ -715,6 +715,9 @@ svn_ra_do_update3
{
SVN_ERR_ASSERT(svn_path_is_empty(update_target)
|| svn_path_is_single_path_component(update_target));
+ SVN_ERR(svn_delta__get_debug_editor(&update_editor, &update_baton,
+ update_editor, update_baton,
+ NULL, result_pool));
return session->vtable->do_update(session,
reporter, report_baton,
revision_to_update_to, update_target,
]]]
The "wrapping" here is that the update_editor->foo callbacks, after
the call, are functions that print to stderr and then invoke the
before-the-call update_editor->foo.
Sorry for the ambiguity.
Cheers,
Daniel
Re: GitHub svn bridge corrupting working copies
Posted by Ryan Schmidt <su...@ryandesign.com>.
Thanks for your suggestions!
On Oct 15, 2017, at 13:07, Daniel Shahaf wrote:
> Ryan Schmidt wrote on Sun, 15 Oct 2017 12:49 -0500:
>> And the problem will just occur again sometime later with a different node. The node it complains about is always a directory that someone else committed to recently, [...]
>
> Have you looked at these commits? Is there anything unusual in them,
> e.g., do they involve renames?
Nothing unusual; just modifying one file and adding another:
https://github.com/macports/macports-ports/commit/2198debd25b02bcf7f6b17f79b415690449a85f0
> Try 'svn log -qvr N URL' and 'svnrdump
> dump -r N --incremental' (or is it '-r N:N+1'?) on those commits,
> does either command error out?
svn log works fine:
$ svn log -qvr r139308 https://github.com/macports/macports-ports/trunk
------------------------------------------------------------------------
r139308 | nicolas.pavillon | 2017-10-13 08:22:43 -0500 (Fri, 13 Oct 2017)
Changed paths:
M /trunk/kde/kdepimlibs4/Portfile
A /trunk/kde/kdepimlibs4/files/patch-do_not_include_cpp.diff
------------------------------------------------------------------------
svnrdump errors:
$ svnrdump dump https://github.com/macports/macports-ports/trunk -r 139308 --incremental
SVN-fs-dump-format-version: 3
UUID: 0bf748bf-1d32-881d-ce5d-ce57b7db7bff
svnrdump: E200007: The requested report is unknown.
Same error on any revision I tried. Guess the bridge doesn't support svnrdump. (It doesn't support a lot of things, like blame.)
Somehow I don't think specific revisions are the problem. I mean, I can now check out a new working copy at revision 139307 and successfully update it to 139308. Meanwhile, on our server, once, back in August, it encountered a corruption already halfway through the process of checking out a new working copy. And on the next checkout attempt it worked.
>> I can "rm -rf kde && svn update kde" it, which restores it from the pristines, and then again fails to update it because of the corrupted node.
>
> Have you also tried 'svn up -r0 kde' and 'svn up --set-depth=exclude kde'?
I have not! I've fixed the working copy with one of your later suggestions; see below.
>> I haven't tried to capture the network traffic. I haven't used such tools before, so I'm unfamiliar with them, and I can't reproduce the problem on demand so it seems problematic to try capturing all network traffic for days until the problem happens to occur.
>
> Well, the next time you run into the problem, backup the working copy
> somewhere, so you'd be able to reproduce by restoring the backup and
> doing 'rm -rf kde && svn up' as you just mentioned.
I have several corrupted working copies saved from the server-side script that I can try things on later.
> You might be able to reproduce the problem by backdating your working
> copy, 'svn up -r N-1 && svn up -r N kde'.
Inside the kde directory, I ran:
$ svn up -r 139306
Updating '.':
At revision 139306.
$ svn up -r 139307
Updating '.':
At revision 139307.
$ svn up -r 139308
Updating '.':
UU kdepimlibs4/Portfile
Skipped 'kdepimlibs4/files/patch-do_not_include_cpp.diff' -- An obstructing working copy was found
Updated to revision 139308.
Summary of conflicts:
Skipped paths: 1
$ svn status
$ rm -rf kdepimlibs4/files/patch-do_not_include_cpp.diff
$ svn up -r 139308
Updating '.':
Restored 'kdepimlibs4/files/patch-do_not_include_cpp.diff'
A /path/to/macports-ports-trunk/patch-do_not_include_cpp.diff
Updated to revision 139308.
$ svn up
Updating '.':
At revision 139412.
So at this point, the working copy seems to be fixed.
>> Thanks for any suggestions you might have! (Other than "use the git client instead"; I've tried that for months and it doesn't seem to be compatible with my brain.)
>
> If you're comfortable with rebuilding, you can try wrapping the "debug
> editor" (svn_delta__get_debug_editor()) around the update editor
> (svn_ra_do_update3()) to get a dump of the changes the server reports to
> the client. That debug output would be at a higher level than a wire dump.
I'd like to try this but I'm not very familiar with the svn code. In subversion/libsvn_client/update.c I replaced "SVN_ERR(svn_ra_do_update3(...))" with "SVN_ERR(svn_delta__get_debug_editor(svn_ra_do_update3(...)))" but now svn segfaults when I update, so that must not have been right. Could you give more guidance on how to do this properly?
Re: GitHub svn bridge corrupting working copies
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Ryan Schmidt wrote on Sun, 15 Oct 2017 12:49 -0500:
> And the problem will just occur again sometime later with a different node. The node it complains about is always a directory that someone else committed to recently, [...]
Have you looked at these commits? Is there anything unusual in them,
e.g., do they involve renames? Try 'svn log -qvr N URL' and 'svnrdump
dump -r N --incremental' (or is it '-r N:N+1'?) on those commits,
does either command error out?
> I can "rm -rf kde && svn update kde" it, which restores it from the pristines, and then again fails to update it because of the corrupted node.
Have you also tried 'svn up -r0 kde' and 'svn up --set-depth=exclude kde'?
> I haven't tried to capture the network traffic. I haven't used such tools before, so I'm unfamiliar with them, and I can't reproduce the problem on demand so it seems problematic to try capturing all network traffic for days until the problem happens to occur.
Well, the next time you run into the problem, backup the working copy
somewhere, so you'd be able to reproduce by restoring the backup and
doing 'rm -rf kde && svn up' as you just mentioned.
You might be able to reproduce the problem by backdating your working
copy, 'svn up -r N-1 && svn up -r N kde'.
> Thanks for any suggestions you might have! (Other than "use the git client instead"; I've tried that for months and it doesn't seem to be compatible with my brain.)
If you're comfortable with rebuilding, you can try wrapping the "debug
editor" (svn_delta__get_debug_editor()) around the update editor
(svn_ra_do_update3()) to get a dump of the changes the server reports to
the client. That debug output would be at a higher level than a wire dump.
Cheers,
Daniel