You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Shahaf <d....@daniel.shahaf.name> on 2020/06/01 13:31:30 UTC

Re: [jira] [Commented] (SVN-4858) FSFS corruption - Found malformed header - Corrupt node-revision

Julian Foad (Jira) wrote on Mon, 01 Jun 2020 11:20 +0000:
> Two clues strongly suggest the corruption was originally caused by a bug rather than hardware corruption:
>  * the checksum on the node-revision must account for the corrupted data, otherwise a checksum error is thrown instead;

_Which_ checksum?  Do you mean the checksum of the directory
representation wherein the dangling (off-by-four) pointer was found?

>  * although an off-by-4 can sometimes be a 1-bit error, this particular one (41271 vs. 41275) is not a 1-bit error.

Is there any chance that this _was_ originally a 1-bit error and then
some offset got added/subtracted to both the id in the noderev header
and the id in the directory rep?

> I do not know which revision contains the bad reference nor what version of svn committed it.

Was CONFIG_OPTION_VERIFY_BEFORE_COMMIT enabled at the time the revision
containing the dangling pointer was committed?  (You may be able to
answer this even without running grep -aR to find the dangling pointer.)

>


It's curious that the wrong offset points directly to the value of the
first field.  However, having glanced at svn_fs_fs__write_noderev(),
I guess that's just a coincidence.

Assuming the node-rev headers _are_ synthesized by svn_fs_fs__write_noderev()
in this user's environment, that is.

Re: [jira] [Commented] (SVN-4858) FSFS corruption - Found malformed header - Corrupt node-revision

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Julian Foad wrote on Tue, 02 Jun 2020 13:05 +0100:
> Daniel Shahaf wrote:
> > Julian Foad (Jira) wrote on Mon, 01 Jun 2020 11:20 +0000:  
> >> Two clues strongly suggest the corruption was originally caused by a bug rather than hardware corruption:
⋮
> >>   * although an off-by-4 can sometimes be a 1-bit error, this particular one (41271 vs. 41275) is not a 1-bit error.  
> >>   * the checksum on the node-revision must account for the corrupted data, otherwise a checksum error is thrown instead;  

> > Is there any chance that this _was_ originally a 1-bit error and then
> > some offset got added/subtracted to both the id in the noderev header
> > and the id in the directory rep?  
> 
> Technically I expect that's possible but it seems less likely.

Does the rev file assembly code add/subtract offsets in such a manner?

> It would have to have happened at original commit time in the
> transaction construction and commit time, whereas I suspect (without
> research) that a bit error is much more likely to occur in data at
> rest on disk for years.

But as you said yourself, a bit error in data at rest wouldn't have
caused the dir rep's checksum to end up being correct.  Can't we rule
that scenario out, then?

I'm not sure what else might have caused this.  The only off-by-4 I can
think of is the magic bytes of svndiff streams (e.g.,
subversion/libsvn_fs_fs/cached_data.c:1597; «rs->current == 4» at that
point), but this could well be unrelated.

Cheers,

Daniel

Re: [jira] [Commented] (SVN-4858) FSFS corruption - Found malformed header - Corrupt node-revision

Posted by Julian Foad <ju...@apache.org>.
Daniel Shahaf wrote:
> Julian Foad (Jira) wrote on Mon, 01 Jun 2020 11:20 +0000:
>> Two clues strongly suggest the corruption was originally caused by a bug rather than hardware corruption:
>>   * the checksum on the node-revision must account for the corrupted data, otherwise a checksum error is thrown instead;
> 
> _Which_ checksum?  Do you mean the checksum of the directory
> representation wherein the dangling (off-by-four) pointer was found?

Yes: in my test an error is thrown for the checksum of that 
representation if that checksum isn't corrected or nulled.

>>   * although an off-by-4 can sometimes be a 1-bit error, this particular one (41271 vs. 41275) is not a 1-bit error.
> 
> Is there any chance that this _was_ originally a 1-bit error and then
> some offset got added/subtracted to both the id in the noderev header
> and the id in the directory rep?

Technically I expect that's possible but it seems less likely.  It would 
have to have happened at original commit time in the transaction 
construction and commit time, whereas I suspect (without research) that 
a bit error is much more likely to occur in data at rest on disk for years.

>> I do not know which revision contains the bad reference nor what version of svn committed it.
> 
> Was CONFIG_OPTION_VERIFY_BEFORE_COMMIT enabled at the time the revision
> containing the dangling pointer was committed?  (You may be able to
> answer this even without running grep -aR to find the dangling pointer.)

I very much expect not, because I am not aware of it being used in any 
production environments that I have been connected with.

> It's curious that the wrong offset points directly to the value of the
> first field.  However, having glanced at svn_fs_fs__write_noderev(),
> I guess that's just a coincidence.
> 
> Assuming the node-rev headers _are_ synthesized by svn_fs_fs__write_noderev()
> in this user's environment, that is.

I'm confident they were synthesized by standard FSFS code.

- Julian