You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Dave Jenkins <n1...@hotmail.com> on 2007/10/23 17:06:42 UTC

Corruption problems

We had a user experience a checksum error on a commit.  He tried again and 
with no error.
The next export (performed by the build process) got a checksum error.
svnadmin verify then reported a checksum error.

We restored the entire repository from a backup that was done just prior to 
the last successful export/build.

Now, after restore, about half way through the revision list svnadmin now 
reports: svnadmin: Invalid diff stream: insn 6174 cannot be decoded.

Any thoughts or suggestions?!?


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems - Crisis averted

Posted by Dave Jenkins <n1...@hotmail.com>.
We were ultimately able to get an older backup from off-site storage and 
restore rev 503...  along with at least 6 other old revs that subsequently 
reported checksum errors on verify.  None of the files that were restored 
from the older backup report any errors and the repository now verifies 
normally (as do all of our others).  That would certainly seem to suggest 
that hardware issues started to develope between the two weekly full 
backups.  And were likely the cause of the initial error that started this 
whole oddessey.  With the new hardware, mirrored drives, and latest updates 
we should be all set for now (knock on... particle board?).  Though I think 
I will still be re-assessing our back-up strategy taking some of Johnathan's 
sage council.

Many thanks to those who responded (Miha, Ryan, and Johnathan).
May I never need your help again!  :-)

Dave


----- Original Message ----- 
From: "Miha Vitorovic" <mv...@nil.si>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: "Ryan Schmidt" <su...@ryandesign.com>; 
<us...@subversion.tigris.org>
Sent: Friday, October 26, 2007 3:34 AM
Subject: Re: Corruption problems


"Dave Jenkins" <n1...@hotmail.com> wrote on 25.10.2007 17:24:11:

> Ok... we took the opportunity to replace the somewhat dated and
potentially
> suspect hardware.
> It's now a VM on a screaming shiney new box.
>
> I upgraded from subversion 1.2.3 to 1.4.5.
>
> I re-ran the svnadmin verify... same error.
>
> I installed Python and ran fsfsverify.py against the rev that was cited
by
> verify.
> After filtering out a ton of stuff the net result was::
>
> D:> fsfsverify.py db\revs\503
>         [...]
>         <Svndiff so: 7473866 ver: 0>
>
>         Error InvalidInstruction: Invalid instruction found at offset
> 7492432 (EC)
>         Try running with -f to fix the revision
>
> D:> fsfsverify.py -f db\revs\503
>         [...]
>         Copy 224819 bytes from offset 7473870
>         Write 224819 bytes at offset 7473866
>         Fixed? :-)  Re-run fsfsverify without the -f option
>
> D:>fsfsverify.py db\revs\503
>         [...]
>         Traceback (most recent call last):
>           File "fsfsverify.py", line 1126, in <module>
>             options.dumpWindows)
>           File "fsfsverify.py", line 565, in verify
>             svndiff = Svndiff(f, self.length)
>           File "fsfsverify.py", line 456, in __init__
>             (self.startingOffset)
>         __main__.InvalidSvndiffHeader: Invalid svndiff header at offset
> 7473866
>

Well, since you already tried everything else, I suggest you try to
contact the SVN developers. They usually monitor this list anyway, but
obviously haven't picked up on your problem. I also know they try to help
a user in a situation like this, since it helps them to tack down the bugs
as well.

Br,
---
  Miha Vitorovic
  Inženir v tehničnem področju
  Customer Support Engineer

   NIL Data Communications,  Tivolska cesta 48,  1000 Ljubljana,  Slovenia
   Phone +386 1 4746 500      Fax +386 1 4746 501     http://www.NIL.si

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems

Posted by Miha Vitorovic <mv...@nil.si>.
"Dave Jenkins" <n1...@hotmail.com> wrote on 25.10.2007 17:24:11:

> Ok... we took the opportunity to replace the somewhat dated and 
potentially 
> suspect hardware.
> It's now a VM on a screaming shiney new box.
> 
> I upgraded from subversion 1.2.3 to 1.4.5.
> 
> I re-ran the svnadmin verify... same error.
> 
> I installed Python and ran fsfsverify.py against the rev that was cited 
by 
> verify.
> After filtering out a ton of stuff the net result was::
> 
> D:> fsfsverify.py db\revs\503
>         [...]
>         <Svndiff so: 7473866 ver: 0>
> 
>         Error InvalidInstruction: Invalid instruction found at offset 
> 7492432 (EC)
>         Try running with -f to fix the revision
> 
> D:> fsfsverify.py -f db\revs\503
>         [...]
>         Copy 224819 bytes from offset 7473870
>         Write 224819 bytes at offset 7473866
>         Fixed? :-)  Re-run fsfsverify without the -f option
> 
> D:>fsfsverify.py db\revs\503
>         [...]
>         Traceback (most recent call last):
>           File "fsfsverify.py", line 1126, in <module>
>             options.dumpWindows)
>           File "fsfsverify.py", line 565, in verify
>             svndiff = Svndiff(f, self.length)
>           File "fsfsverify.py", line 456, in __init__
>             (self.startingOffset)
>         __main__.InvalidSvndiffHeader: Invalid svndiff header at offset 
> 7473866
> 

Well, since you already tried everything else, I suggest you try to 
contact the SVN developers. They usually monitor this list anyway, but 
obviously haven't picked up on your problem. I also know they try to help 
a user in a situation like this, since it helps them to tack down the bugs 
as well.

Br,
---
  Miha Vitorovic
  Inženir v tehničnem področju
  Customer Support Engineer

   NIL Data Communications,  Tivolska cesta 48,  1000 Ljubljana,  Slovenia
   Phone +386 1 4746 500      Fax +386 1 4746 501     http://www.NIL.si

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org


Re: Corruption problems

Posted by Dave Jenkins <n1...@hotmail.com>.
Oh, and, just in case it is relevant...

The revision that verify is choking on (that was part of what we restored 
from backup) is some 400 revisions ago.  So the notion that this file was in 
some state of flux at the time the backup was run on Friday seems unlikely 
at best.

Dave


----- Original Message ----- 
From: "Dave Jenkins" <n1...@hotmail.com>
To: "Ryan Schmidt" <su...@ryandesign.com>
Cc: <us...@subversion.tigris.org>
Sent: Thursday, October 25, 2007 11:24 AM
Subject: Re: Corruption problems


> Ok... we took the opportunity to replace the somewhat dated and 
> potentially suspect hardware.
> It's now a VM on a screaming shiney new box.
>
> I upgraded from subversion 1.2.3 to 1.4.5.
>
> I re-ran the svnadmin verify... same error.
>
> I installed Python and ran fsfsverify.py against the rev that was cited by 
> verify.
> After filtering out a ton of stuff the net result was::
>
> D:> fsfsverify.py db\revs\503
>        [...]
>        <Svndiff so: 7473866 ver: 0>
>
>        Error InvalidInstruction: Invalid instruction found at offset 
> 7492432 (EC)
>        Try running with -f to fix the revision
>
> D:> fsfsverify.py -f db\revs\503
>        [...]
>        Copy 224819 bytes from offset 7473870
>        Write 224819 bytes at offset 7473866
>        Fixed? :-)  Re-run fsfsverify without the -f option
>
> D:>fsfsverify.py db\revs\503
>        [...]
>        Traceback (most recent call last):
>          File "fsfsverify.py", line 1126, in <module>
>            options.dumpWindows)
>          File "fsfsverify.py", line 565, in verify
>            svndiff = Svndiff(f, self.length)
>          File "fsfsverify.py", line 456, in __init__
>            (self.startingOffset)
>        __main__.InvalidSvndiffHeader: Invalid svndiff header at offset 
> 7473866
>
> ----- Original Message ----- 
> From: "Ryan Schmidt" <su...@ryandesign.com>
> To: "Dave Jenkins" <n1...@hotmail.com>
> Cc: <us...@subversion.tigris.org>
> Sent: Tuesday, October 23, 2007 4:00 PM
> Subject: Re: Corruption problems
>
>
>> On Oct 23, 2007, at 14:23, Dave Jenkins wrote:
>>
>>>> Tell us more about your setup.
>>> Sure, Thanks!
>>>
>>> It has been working flawlessly for 2+ years so I have been loathe  to 
>>> update subversion.
>>>
>>>> OS
>>> Windows Server 2003
>>> Standard Edition
>>>
>>>> OS version
>>> Hmmm... upgraded to Service Pack 2... YESTERDAY.  Suspicious?!?
>>>
>>>> Subversion version,
>>> 1.2.3
>>>
>>>> what media is the repository on (local hard disk, local RAID,  NFS, 
>>>> SAN), could you have hardware problems (have you run some  kind of 
>>>> hardware test), etc.
>>> Local NTFS hard drive.  (Backed up nightly.)
>>
>> You may want to try accessing a copy of your repository using  Subversion 
>> 1.4.5, see if that works better. Subversion 1.2.x and  earlier aren't 
>> supported anymore.
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems

Posted by Dave Jenkins <n1...@hotmail.com>.
Ok... we took the opportunity to replace the somewhat dated and potentially 
suspect hardware.
It's now a VM on a screaming shiney new box.

I upgraded from subversion 1.2.3 to 1.4.5.

I re-ran the svnadmin verify... same error.

I installed Python and ran fsfsverify.py against the rev that was cited by 
verify.
After filtering out a ton of stuff the net result was::

D:> fsfsverify.py db\revs\503
        [...]
        <Svndiff so: 7473866 ver: 0>

        Error InvalidInstruction: Invalid instruction found at offset 
7492432 (EC)
        Try running with -f to fix the revision

D:> fsfsverify.py -f db\revs\503
        [...]
        Copy 224819 bytes from offset 7473870
        Write 224819 bytes at offset 7473866
        Fixed? :-)  Re-run fsfsverify without the -f option

D:>fsfsverify.py db\revs\503
        [...]
        Traceback (most recent call last):
          File "fsfsverify.py", line 1126, in <module>
            options.dumpWindows)
          File "fsfsverify.py", line 565, in verify
            svndiff = Svndiff(f, self.length)
          File "fsfsverify.py", line 456, in __init__
            (self.startingOffset)
        __main__.InvalidSvndiffHeader: Invalid svndiff header at offset 
7473866

----- Original Message ----- 
From: "Ryan Schmidt" <su...@ryandesign.com>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: <us...@subversion.tigris.org>
Sent: Tuesday, October 23, 2007 4:00 PM
Subject: Re: Corruption problems


> On Oct 23, 2007, at 14:23, Dave Jenkins wrote:
>
>>> Tell us more about your setup.
>> Sure, Thanks!
>>
>> It has been working flawlessly for 2+ years so I have been loathe  to 
>> update subversion.
>>
>>> OS
>> Windows Server 2003
>> Standard Edition
>>
>>> OS version
>> Hmmm... upgraded to Service Pack 2... YESTERDAY.  Suspicious?!?
>>
>>> Subversion version,
>> 1.2.3
>>
>>> what media is the repository on (local hard disk, local RAID,  NFS, 
>>> SAN), could you have hardware problems (have you run some  kind of 
>>> hardware test), etc.
>> Local NTFS hard drive.  (Backed up nightly.)
>
> You may want to try accessing a copy of your repository using  Subversion 
> 1.4.5, see if that works better. Subversion 1.2.x and  earlier aren't 
> supported anymore.
>
>
>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems

Posted by Miha Vitorovic <mv...@nil.si>.
Dave,

If it's a FSFS repository you might try running 
http://svn.collab.net/viewvc/svn/trunk/contrib/server-side/fsfsverify.py?view=log 
. Do a full backup prior to that...

Br,
---
  Miha Vitorovic
  Inženir v tehničnem področju
  Customer Support Engineer

   NIL Data Communications,  Tivolska cesta 48,  1000 Ljubljana,  Slovenia
   Phone +386 1 4746 500      Fax +386 1 4746 501     http://www.NIL.si

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org


Re: Corruption problems

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 23, 2007, at 14:23, Dave Jenkins wrote:

>> Tell us more about your setup.
> Sure, Thanks!
>
> It has been working flawlessly for 2+ years so I have been loathe  
> to update subversion.
>
>> OS
> Windows Server 2003
> Standard Edition
>
>> OS version
> Hmmm... upgraded to Service Pack 2... YESTERDAY.  Suspicious?!?
>
>> Subversion version,
> 1.2.3
>
>> what media is the repository on (local hard disk, local RAID,  
>> NFS,  SAN), could you have hardware problems (have you run some  
>> kind of  hardware test), etc.
> Local NTFS hard drive.  (Backed up nightly.)

You may want to try accessing a copy of your repository using  
Subversion 1.4.5, see if that works better. Subversion 1.2.x and  
earlier aren't supported anymore.



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems

Posted by Dave Jenkins <n1...@hotmail.com>.
> Tell us more about your setup.
Sure, Thanks!

It has been working flawlessly for 2+ years so I have been loathe to update 
subversion.

> OS
Windows Server 2003
Standard Edition

> OS version
Hmmm... upgraded to Service Pack 2... YESTERDAY.  Suspicious?!?

> Subversion version,
1.2.3

> what media is the repository on (local hard disk, local RAID, NFS,  SAN), 
> could you have hardware problems (have you run some kind of  hardware 
> test), etc.
Local NTFS hard drive.  (Backed up nightly.)



----- Original Message ----- 
From: "Ryan Schmidt" <su...@ryandesign.com>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: <us...@subversion.tigris.org>
Sent: Tuesday, October 23, 2007 2:42 PM
Subject: Re: Corruption problems


>
> On Oct 23, 2007, at 12:06, Dave Jenkins wrote:
>
>> We had a user experience a checksum error on a commit.  He tried  again 
>> and with no error.
>> The next export (performed by the build process) got a checksum error.
>> svnadmin verify then reported a checksum error.
>>
>> We restored the entire repository from a backup that was done just  prior 
>> to the last successful export/build.
>>
>> Now, after restore, about half way through the revision list  svnadmin 
>> now reports: svnadmin: Invalid diff stream: insn 6174  cannot be decoded.
>>
>> Any thoughts or suggestions?!?
>
> Tell us more about your setup. OS, OS version, Subversion version,  what 
> media is the repository on (local hard disk, local RAID, NFS,  SAN), could 
> you have hardware problems (have you run some kind of  hardware test), 
> etc.
>
>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Corruption problems

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 23, 2007, at 12:06, Dave Jenkins wrote:

> We had a user experience a checksum error on a commit.  He tried  
> again and with no error.
> The next export (performed by the build process) got a checksum error.
> svnadmin verify then reported a checksum error.
>
> We restored the entire repository from a backup that was done just  
> prior to the last successful export/build.
>
> Now, after restore, about half way through the revision list  
> svnadmin now reports: svnadmin: Invalid diff stream: insn 6174  
> cannot be decoded.
>
> Any thoughts or suggestions?!?

Tell us more about your setup. OS, OS version, Subversion version,  
what media is the repository on (local hard disk, local RAID, NFS,  
SAN), could you have hardware problems (have you run some kind of  
hardware test), etc.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org