You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Trent Fisher <tr...@oracle.com> on 2012/06/22 19:52:47 UTC
Possible svnsync data loss bug?
I just ran into a rather frightening problem. We do replications via
svnsync for backup purposes. I just found two of the replicated
repositories which had these entries for *every* version:
r20927 | (no author) | (no date) | 1 line
Changed paths:
And svnlook tree shows there is nothing in the repositories.
We have never seen any errors while running svnsync, however, these
replicas have been around so long that most of the logs are long since
gone, though I did find a recent log entry for the very version shown
above (note: the first and last lines are output by my scripts):
Running command: svnsync sync file:///svn/repos/blah-blah
--non-interactive
Committed revision 20927.
Copied properties for revision 20927.
No errors, not sending email
It appears that svnsync is failing somehow but never indicating that in
the output or exit status and, instead, creating empty revisions.
These two repositories are the only two we are replicating from this
host, and I have yet to find any other repositories in the same state
(though I am still looking). So, I guess it could be a bug on the
remote end, which svnsync is unable to detect. My machine is running
SVN 1.6.16 (on a Linux machine running our variant of RHEL 5.8). I
can't tell you much about the remote end as it is currently down ;(
I searched through the SVN bugdb but didn't see anything reminiscent of
this. Has anyone seen this behavior?
++thanks,
trent...
Re: Possible svnsync data loss bug?
Posted by Trent Fisher <tr...@oracle.com>.
On 06/22/2012 08:17 PM, Nico Kadel-Garcia wrote:
> On Fri, Jun 22, 2012 at 5:21 PM, Trent Fisher<tr...@oracle.com> wrote:
>> On 06/22/2012 01:59 PM, Mark Phippard wrote:
>>> On Fri, Jun 22, 2012 at 1:52 PM, Trent Fisher<tr...@oracle.com>
>>> wrote:
>>>> I just ran into a rather frightening problem. We do replications via
>>>> svnsync for backup purposes. I just found two of the replicated
>>>> repositories which had these entries for *every* version:
>>>>
>>>> r20927 | (no author) | (no date) | 1 line
>>>> Changed paths:
>>>>
>>>> And svnlook tree shows there is nothing in the repositories.
>>> I believe this matches the behavior you would get if you were using
>>> path-based permissions and the svnsync user did not have read
>>> permission to the paths in the revision. Could that be the reason?
>>> You could also get this if you did not specify the URL to the root of
>>> the repository. Any revision that does not touch the path you
>>> specified would just sync an empty revision.
>>
>> Duh! Of course. I just tested both of the possible reasons you gave and,
>> sure enough, both will produce revs like the one in my first message. But I
>> think the only way to get a replica, like mine, with *all* empty revs would
>> be via path-based permissions. If we were syncing only a subdirectory of a
>> repository which we, otherwise, had read access to, the commits outside of
>> the sync'ed url would have proper usernames and dates, but no files. (I
>> tested that too)
>>
>> I was about to ask if svnsync would get access denied errors... but I did
>> another test which, I think, answered my own question: If I run "svn log"
>> against a repository to which I have only access to part of, revs in the
>> areas I don't have access to will show up as "(no author) | (no date)".
>> Presumably svnsync is doing some variant of a "log" call and then getting
>> the contents of each rev and faithfully reproduces what seem to be empty
>> revs.
>>
>> So, I guess svnsync is arguably doing the right thing. But even if it were
>> declared to be incorrect, and a fix were checked in right now, it wouldn't
>> address any of my existing replicas. So, I'll have to develop some
>> auditing for these sorts of things. Which brings up another question: is
>> there any other reason you would ever see "(no date)" in a log entry?
> Ouch. svnsync failing to transfer due to permissions, but leaving
> empty logs, is...... That's a silent failure that's going to be really
> problematic in the field.
>
> What happens after the permissions are opened? Do the logs get
> recovere successfully?
I agree this is problematic. I might have a total loss on two
repositories due to this. I have no idea if or how this could be fixed,
I've never really delved into the guts of SVN, though maybe I will after
I take care of my current crisis.
As for later opening up permissions, I don't think that's going to help
at all with old revisions. Even though they are empty, svnsync will not
go back to reconsider them, only newly created revisions would be
affected. But if you created a new replica with svnsync and start over,
it should pick up the revisions correctly.
Re: Possible svnsync data loss bug?
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Fri, Jun 22, 2012 at 5:21 PM, Trent Fisher <tr...@oracle.com> wrote:
>
> On 06/22/2012 01:59 PM, Mark Phippard wrote:
>>
>> On Fri, Jun 22, 2012 at 1:52 PM, Trent Fisher<tr...@oracle.com>
>> wrote:
>>>
>>> I just ran into a rather frightening problem. We do replications via
>>> svnsync for backup purposes. I just found two of the replicated
>>> repositories which had these entries for *every* version:
>>>
>>> r20927 | (no author) | (no date) | 1 line
>>> Changed paths:
>>>
>>> And svnlook tree shows there is nothing in the repositories.
>>
>> I believe this matches the behavior you would get if you were using
>> path-based permissions and the svnsync user did not have read
>> permission to the paths in the revision. Could that be the reason?
>> You could also get this if you did not specify the URL to the root of
>> the repository. Any revision that does not touch the path you
>> specified would just sync an empty revision.
>
>
> Duh! Of course. I just tested both of the possible reasons you gave and,
> sure enough, both will produce revs like the one in my first message. But I
> think the only way to get a replica, like mine, with *all* empty revs would
> be via path-based permissions. If we were syncing only a subdirectory of a
> repository which we, otherwise, had read access to, the commits outside of
> the sync'ed url would have proper usernames and dates, but no files. (I
> tested that too)
>
> I was about to ask if svnsync would get access denied errors... but I did
> another test which, I think, answered my own question: If I run "svn log"
> against a repository to which I have only access to part of, revs in the
> areas I don't have access to will show up as "(no author) | (no date)".
> Presumably svnsync is doing some variant of a "log" call and then getting
> the contents of each rev and faithfully reproduces what seem to be empty
> revs.
>
> So, I guess svnsync is arguably doing the right thing. But even if it were
> declared to be incorrect, and a fix were checked in right now, it wouldn't
> address any of my existing replicas. So, I'll have to develop some
> auditing for these sorts of things. Which brings up another question: is
> there any other reason you would ever see "(no date)" in a log entry?
Ouch. svnsync failing to transfer due to permissions, but leaving
empty logs, is...... That's a silent failure that's going to be really
problematic in the field.
What happens after the permissions are opened? Do the logs get
recovere successfully?
Re: Possible svnsync data loss bug?
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Jun 27, 2012 at 11:55:33AM -0400, Trent Fisher wrote:
> On 06/26/2012 05:02 PM, Stefan Sperling wrote:
> >On Tue, Jun 26, 2012 at 10:13:50PM +0200, Johan Corveleyn wrote:
> >>On Tue, Jun 26, 2012 at 8:07 PM,<km...@rockwellcollins.com> wrote:
> >>>Any reason svnsync couldn't grow a "--fail-on-access-error" option?
> >>Sure, I think that could be useful (as long as it's not the default
> >>(backwards compat)).
> >Would printing a warning not be sufficient?
> That would be an improvement, but since it seems like sometimes this
> behavior may be desired or expected, it may be better to always
> print a warning, and bomb out given an extra option like that
> suggested by Kevin, above.
Yes, I agree that would make sense.
Re: Possible svnsync data loss bug?
Posted by Trent Fisher <tr...@oracle.com>.
On 06/26/2012 05:02 PM, Stefan Sperling wrote:
> On Tue, Jun 26, 2012 at 10:13:50PM +0200, Johan Corveleyn wrote:
>> On Tue, Jun 26, 2012 at 8:07 PM,<km...@rockwellcollins.com> wrote:
>>> Any reason svnsync couldn't grow a "--fail-on-access-error" option?
>> Sure, I think that could be useful (as long as it's not the default
>> (backwards compat)).
> Would printing a warning not be sufficient?
That would be an improvement, but since it seems like sometimes this
behavior may be desired or expected, it may be better to always print a
warning, and bomb out given an extra option like that suggested by
Kevin, above.
Re: Possible svnsync data loss bug?
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Jun 26, 2012 at 10:13:50PM +0200, Johan Corveleyn wrote:
> On Tue, Jun 26, 2012 at 8:07 PM, <km...@rockwellcollins.com> wrote:
> > Any reason svnsync couldn't grow a "--fail-on-access-error" option?
>
> Sure, I think that could be useful (as long as it's not the default
> (backwards compat)).
Would printing a warning not be sufficient?
RE: Possible svnsync data loss bug?
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
> Sent: dinsdag 26 juni 2012 22:14
> To: kmradke@rockwellcollins.com
> Cc: Mark Phippard; Trent Fisher; users@subversion.apache.org
> Subject: Re: Possible svnsync data loss bug?
>
> On Tue, Jun 26, 2012 at 8:07 PM, <km...@rockwellcollins.com> wrote:
> >> > I was about to ask if svnsync would get access denied errors... but I
> >> > did
> >> > another test which, I think, answered my own question: If I run "svn
> >> > log"
> >> > against a repository to which I have only access to part of, revs in
the
> >> > areas I don't have access to will show up as "(no author) | (no
date)".
> >> > Presumably svnsync is doing some variant of a "log" call and then
> >> > getting
> >> > the contents of each rev and faithfully reproduces what seem to be
> empty
> >> > revs.
> >> >
> >> > So, I guess svnsync is arguably doing the right thing. But even if
it
> >> > were
> >> > declared to be incorrect, and a fix were checked in right now, it
> >> > wouldn't
> >> > address any of my existing replicas. So, I'll have to develop some
> >> > auditing for these sorts of things. Which brings up another
question:
> >> > is
> >> > there any other reason you would ever see "(no date)" in a log entry?
> >>
> >> I cannot speak to all scenarios because I have not tested them. For
> >> example, if the svnsync user simply had no read access to anywhere in
> >> the repository it seems like it ought to error. But that scenario
> >> aside, the way that svnsync works is actually a huge feature. Imagine
> >> you have a remote team of contractors that you want to give a local
> >> replica of your repository. These contractors are not allowed access
> >> to parts of your repository. The contractors can use svnsync to crete
> >> a local replica using their credentials and their local repository
> >> will only contain the content they are allowed to access. The
> >> revisions that altered other paths will simply be empty commits.
> >>
> >> As I started, there may be other scenarios where this could be handled
> >> differently, but overall I think this is a great feature of svn.
> >
> > Both are valid use cases. I frequently use the current access behavior
> > to "filter" large repositories instead of using svndumpfilter which
> > would require terabytes of scratch space and days of downtime to filter.
>
> It's also sometimes used to recover from some forms of repository
> corruption (lacking proper backups), if the corruption is limited to a
> single file (or a set of files) in history. By configuring an authz
> policy that denies access to that file, one can svnsync to a new
> repository, which is then free of corruption (without the corrupted
> file of course).
>
> > Any reason svnsync couldn't grow a "--fail-on-access-error" option?
>
> Sure, I think that could be useful (as long as it's not the default
> (backwards compat)).
The problem here is that we don't get an access denied error. Svnsync just
notes an empty revision, which is somehow valid. (We try not to create empty
revisions, but there are quite a few repositories that have them via
different tricks or bugs).
So we should have a way to detect that this happens.
Bert
>
> --
> Johan
Re: Possible svnsync data loss bug?
Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, Jun 26, 2012 at 8:07 PM, <km...@rockwellcollins.com> wrote:
>> > I was about to ask if svnsync would get access denied errors... but I
>> > did
>> > another test which, I think, answered my own question: If I run "svn
>> > log"
>> > against a repository to which I have only access to part of, revs in the
>> > areas I don't have access to will show up as "(no author) | (no date)".
>> > Presumably svnsync is doing some variant of a "log" call and then
>> > getting
>> > the contents of each rev and faithfully reproduces what seem to be empty
>> > revs.
>> >
>> > So, I guess svnsync is arguably doing the right thing. But even if it
>> > were
>> > declared to be incorrect, and a fix were checked in right now, it
>> > wouldn't
>> > address any of my existing replicas. So, I'll have to develop some
>> > auditing for these sorts of things. Which brings up another question:
>> > is
>> > there any other reason you would ever see "(no date)" in a log entry?
>>
>> I cannot speak to all scenarios because I have not tested them. For
>> example, if the svnsync user simply had no read access to anywhere in
>> the repository it seems like it ought to error. But that scenario
>> aside, the way that svnsync works is actually a huge feature. Imagine
>> you have a remote team of contractors that you want to give a local
>> replica of your repository. These contractors are not allowed access
>> to parts of your repository. The contractors can use svnsync to crete
>> a local replica using their credentials and their local repository
>> will only contain the content they are allowed to access. The
>> revisions that altered other paths will simply be empty commits.
>>
>> As I started, there may be other scenarios where this could be handled
>> differently, but overall I think this is a great feature of svn.
>
> Both are valid use cases. I frequently use the current access behavior
> to "filter" large repositories instead of using svndumpfilter which
> would require terabytes of scratch space and days of downtime to filter.
It's also sometimes used to recover from some forms of repository
corruption (lacking proper backups), if the corruption is limited to a
single file (or a set of files) in history. By configuring an authz
policy that denies access to that file, one can svnsync to a new
repository, which is then free of corruption (without the corrupted
file of course).
> Any reason svnsync couldn't grow a "--fail-on-access-error" option?
Sure, I think that could be useful (as long as it's not the default
(backwards compat)).
--
Johan
Re: Possible svnsync data loss bug?
Posted by km...@rockwellcollins.com.
> > I was about to ask if svnsync would get access denied errors... but I
did
> > another test which, I think, answered my own question: If I run "svn
log"
> > against a repository to which I have only access to part of, revs in
the
> > areas I don't have access to will show up as "(no author) | (no
date)".
> > Presumably svnsync is doing some variant of a "log" call and then
getting
> > the contents of each rev and faithfully reproduces what seem to be
empty
> > revs.
> >
> > So, I guess svnsync is arguably doing the right thing. But even if it
were
> > declared to be incorrect, and a fix were checked in right now, it
wouldn't
> > address any of my existing replicas. So, I'll have to develop some
> > auditing for these sorts of things. Which brings up another question:
is
> > there any other reason you would ever see "(no date)" in a log entry?
>
> I cannot speak to all scenarios because I have not tested them. For
> example, if the svnsync user simply had no read access to anywhere in
> the repository it seems like it ought to error. But that scenario
> aside, the way that svnsync works is actually a huge feature. Imagine
> you have a remote team of contractors that you want to give a local
> replica of your repository. These contractors are not allowed access
> to parts of your repository. The contractors can use svnsync to crete
> a local replica using their credentials and their local repository
> will only contain the content they are allowed to access. The
> revisions that altered other paths will simply be empty commits.
>
> As I started, there may be other scenarios where this could be handled
> differently, but overall I think this is a great feature of svn.
Both are valid use cases. I frequently use the current access behavior
to "filter" large repositories instead of using svndumpfilter which
would require terabytes of scratch space and days of downtime to filter.
Any reason svnsync couldn't grow a "--fail-on-access-error" option?
Kevin R.
Re: Possible svnsync data loss bug?
Posted by Mark Phippard <ma...@gmail.com>.
On Jun 26, 2012, at 6:43 PM, Ryan Schmidt <su...@ryandesign.com> wrote:
>
> On Jun 26, 2012, at 12:43, Mark Phippard wrote:
>
>> I cannot speak to all scenarios because I have not tested them. For
>> example, if the svnsync user simply had no read access to anywhere in
>> the repository it seems like it ought to error. But that scenario
>> aside, the way that svnsync works is actually a huge feature. Imagine
>> you have a remote team of contractors that you want to give a local
>> replica of your repository. These contractors are not allowed access
>> to parts of your repository. The contractors can use svnsync to crete
>> a local replica using their credentials and their local repository
>> will only contain the content they are allowed to access. The
>> revisions that altered other paths will simply be empty commits.
>
> What happens if a single commit touches both paths that are allowed and paths that are prohibit by a particular svnsync user? Will the entire commit be ignored or will the parts of it that are in the allowed paths be included and the rest removed?
>
The latter will happen.
Mark
Re: Possible svnsync data loss bug?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jun 26, 2012, at 12:43, Mark Phippard wrote:
> I cannot speak to all scenarios because I have not tested them. For
> example, if the svnsync user simply had no read access to anywhere in
> the repository it seems like it ought to error. But that scenario
> aside, the way that svnsync works is actually a huge feature. Imagine
> you have a remote team of contractors that you want to give a local
> replica of your repository. These contractors are not allowed access
> to parts of your repository. The contractors can use svnsync to crete
> a local replica using their credentials and their local repository
> will only contain the content they are allowed to access. The
> revisions that altered other paths will simply be empty commits.
What happens if a single commit touches both paths that are allowed and paths that are prohibit by a particular svnsync user? Will the entire commit be ignored or will the parts of it that are in the allowed paths be included and the rest removed?
Re: Possible svnsync data loss bug?
Posted by Mark Phippard <ma...@gmail.com>.
On Fri, Jun 22, 2012 at 5:21 PM, Trent Fisher <tr...@oracle.com> wrote:
> I was about to ask if svnsync would get access denied errors... but I did
> another test which, I think, answered my own question: If I run "svn log"
> against a repository to which I have only access to part of, revs in the
> areas I don't have access to will show up as "(no author) | (no date)".
> Presumably svnsync is doing some variant of a "log" call and then getting
> the contents of each rev and faithfully reproduces what seem to be empty
> revs.
>
> So, I guess svnsync is arguably doing the right thing. But even if it were
> declared to be incorrect, and a fix were checked in right now, it wouldn't
> address any of my existing replicas. So, I'll have to develop some
> auditing for these sorts of things. Which brings up another question: is
> there any other reason you would ever see "(no date)" in a log entry?
I cannot speak to all scenarios because I have not tested them. For
example, if the svnsync user simply had no read access to anywhere in
the repository it seems like it ought to error. But that scenario
aside, the way that svnsync works is actually a huge feature. Imagine
you have a remote team of contractors that you want to give a local
replica of your repository. These contractors are not allowed access
to parts of your repository. The contractors can use svnsync to crete
a local replica using their credentials and their local repository
will only contain the content they are allowed to access. The
revisions that altered other paths will simply be empty commits.
As I started, there may be other scenarios where this could be handled
differently, but overall I think this is a great feature of svn.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Possible svnsync data loss bug?
Posted by Trent Fisher <tr...@oracle.com>.
On 06/22/2012 01:59 PM, Mark Phippard wrote:
> On Fri, Jun 22, 2012 at 1:52 PM, Trent Fisher<tr...@oracle.com> wrote:
>> I just ran into a rather frightening problem. We do replications via
>> svnsync for backup purposes. I just found two of the replicated
>> repositories which had these entries for *every* version:
>>
>> r20927 | (no author) | (no date) | 1 line
>> Changed paths:
>>
>> And svnlook tree shows there is nothing in the repositories.
> I believe this matches the behavior you would get if you were using
> path-based permissions and the svnsync user did not have read
> permission to the paths in the revision. Could that be the reason?
> You could also get this if you did not specify the URL to the root of
> the repository. Any revision that does not touch the path you
> specified would just sync an empty revision.
Duh! Of course. I just tested both of the possible reasons you gave
and, sure enough, both will produce revs like the one in my first
message. But I think the only way to get a replica, like mine, with
*all* empty revs would be via path-based permissions. If we were
syncing only a subdirectory of a repository which we, otherwise, had
read access to, the commits outside of the sync'ed url would have proper
usernames and dates, but no files. (I tested that too)
I was about to ask if svnsync would get access denied errors... but I
did another test which, I think, answered my own question: If I run
"svn log" against a repository to which I have only access to part of,
revs in the areas I don't have access to will show up as "(no author) |
(no date)". Presumably svnsync is doing some variant of a "log" call
and then getting the contents of each rev and faithfully reproduces what
seem to be empty revs.
So, I guess svnsync is arguably doing the right thing. But even if it
were declared to be incorrect, and a fix were checked in right now, it
wouldn't address any of my existing replicas. So, I'll have to develop
some auditing for these sorts of things. Which brings up another
question: is there any other reason you would ever see "(no date)" in a
log entry?
Thanks!
trent...
Re: Possible svnsync data loss bug?
Posted by Mark Phippard <ma...@gmail.com>.
On Fri, Jun 22, 2012 at 1:52 PM, Trent Fisher <tr...@oracle.com> wrote:
> I just ran into a rather frightening problem. We do replications via
> svnsync for backup purposes. I just found two of the replicated
> repositories which had these entries for *every* version:
>
> r20927 | (no author) | (no date) | 1 line
> Changed paths:
>
> And svnlook tree shows there is nothing in the repositories.
I believe this matches the behavior you would get if you were using
path-based permissions and the svnsync user did not have read
permission to the paths in the revision. Could that be the reason?
You could also get this if you did not specify the URL to the root of
the repository. Any revision that does not touch the path you
specified would just sync an empty revision.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/