You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Dave Jenkins <n1...@hotmail.com> on 2007/10/23 17:06:42 UTC
Corruption problems
We had a user experience a checksum error on a commit. He tried again and
with no error.
The next export (performed by the build process) got a checksum error.
svnadmin verify then reported a checksum error.
We restored the entire repository from a backup that was done just prior to
the last successful export/build.
Now, after restore, about half way through the revision list svnadmin now
reports: svnadmin: Invalid diff stream: insn 6174 cannot be decoded.
Any thoughts or suggestions?!?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems - Crisis averted
Posted by Dave Jenkins <n1...@hotmail.com>.
We were ultimately able to get an older backup from off-site storage and
restore rev 503... along with at least 6 other old revs that subsequently
reported checksum errors on verify. None of the files that were restored
from the older backup report any errors and the repository now verifies
normally (as do all of our others). That would certainly seem to suggest
that hardware issues started to develope between the two weekly full
backups. And were likely the cause of the initial error that started this
whole oddessey. With the new hardware, mirrored drives, and latest updates
we should be all set for now (knock on... particle board?). Though I think
I will still be re-assessing our back-up strategy taking some of Johnathan's
sage council.
Many thanks to those who responded (Miha, Ryan, and Johnathan).
May I never need your help again! :-)
Dave
----- Original Message -----
From: "Miha Vitorovic" <mv...@nil.si>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: "Ryan Schmidt" <su...@ryandesign.com>;
<us...@subversion.tigris.org>
Sent: Friday, October 26, 2007 3:34 AM
Subject: Re: Corruption problems
"Dave Jenkins" <n1...@hotmail.com> wrote on 25.10.2007 17:24:11:
> Ok... we took the opportunity to replace the somewhat dated and
potentially
> suspect hardware.
> It's now a VM on a screaming shiney new box.
>
> I upgraded from subversion 1.2.3 to 1.4.5.
>
> I re-ran the svnadmin verify... same error.
>
> I installed Python and ran fsfsverify.py against the rev that was cited
by
> verify.
> After filtering out a ton of stuff the net result was::
>
> D:> fsfsverify.py db\revs\503
> [...]
> <Svndiff so: 7473866 ver: 0>
>
> Error InvalidInstruction: Invalid instruction found at offset
> 7492432 (EC)
> Try running with -f to fix the revision
>
> D:> fsfsverify.py -f db\revs\503
> [...]
> Copy 224819 bytes from offset 7473870
> Write 224819 bytes at offset 7473866
> Fixed? :-) Re-run fsfsverify without the -f option
>
> D:>fsfsverify.py db\revs\503
> [...]
> Traceback (most recent call last):
> File "fsfsverify.py", line 1126, in <module>
> options.dumpWindows)
> File "fsfsverify.py", line 565, in verify
> svndiff = Svndiff(f, self.length)
> File "fsfsverify.py", line 456, in __init__
> (self.startingOffset)
> __main__.InvalidSvndiffHeader: Invalid svndiff header at offset
> 7473866
>
Well, since you already tried everything else, I suggest you try to
contact the SVN developers. They usually monitor this list anyway, but
obviously haven't picked up on your problem. I also know they try to help
a user in a situation like this, since it helps them to tack down the bugs
as well.
Br,
---
Miha Vitorovic
Inženir v tehničnem področju
Customer Support Engineer
NIL Data Communications, Tivolska cesta 48, 1000 Ljubljana, Slovenia
Phone +386 1 4746 500 Fax +386 1 4746 501 http://www.NIL.si
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Miha Vitorovic <mv...@nil.si>.
"Dave Jenkins" <n1...@hotmail.com> wrote on 25.10.2007 17:24:11:
> Ok... we took the opportunity to replace the somewhat dated and
potentially
> suspect hardware.
> It's now a VM on a screaming shiney new box.
>
> I upgraded from subversion 1.2.3 to 1.4.5.
>
> I re-ran the svnadmin verify... same error.
>
> I installed Python and ran fsfsverify.py against the rev that was cited
by
> verify.
> After filtering out a ton of stuff the net result was::
>
> D:> fsfsverify.py db\revs\503
> [...]
> <Svndiff so: 7473866 ver: 0>
>
> Error InvalidInstruction: Invalid instruction found at offset
> 7492432 (EC)
> Try running with -f to fix the revision
>
> D:> fsfsverify.py -f db\revs\503
> [...]
> Copy 224819 bytes from offset 7473870
> Write 224819 bytes at offset 7473866
> Fixed? :-) Re-run fsfsverify without the -f option
>
> D:>fsfsverify.py db\revs\503
> [...]
> Traceback (most recent call last):
> File "fsfsverify.py", line 1126, in <module>
> options.dumpWindows)
> File "fsfsverify.py", line 565, in verify
> svndiff = Svndiff(f, self.length)
> File "fsfsverify.py", line 456, in __init__
> (self.startingOffset)
> __main__.InvalidSvndiffHeader: Invalid svndiff header at offset
> 7473866
>
Well, since you already tried everything else, I suggest you try to
contact the SVN developers. They usually monitor this list anyway, but
obviously haven't picked up on your problem. I also know they try to help
a user in a situation like this, since it helps them to tack down the bugs
as well.
Br,
---
Miha Vitorovic
Inženir v tehničnem področju
Customer Support Engineer
NIL Data Communications, Tivolska cesta 48, 1000 Ljubljana, Slovenia
Phone +386 1 4746 500 Fax +386 1 4746 501 http://www.NIL.si
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Dave Jenkins <n1...@hotmail.com>.
Oh, and, just in case it is relevant...
The revision that verify is choking on (that was part of what we restored
from backup) is some 400 revisions ago. So the notion that this file was in
some state of flux at the time the backup was run on Friday seems unlikely
at best.
Dave
----- Original Message -----
From: "Dave Jenkins" <n1...@hotmail.com>
To: "Ryan Schmidt" <su...@ryandesign.com>
Cc: <us...@subversion.tigris.org>
Sent: Thursday, October 25, 2007 11:24 AM
Subject: Re: Corruption problems
> Ok... we took the opportunity to replace the somewhat dated and
> potentially suspect hardware.
> It's now a VM on a screaming shiney new box.
>
> I upgraded from subversion 1.2.3 to 1.4.5.
>
> I re-ran the svnadmin verify... same error.
>
> I installed Python and ran fsfsverify.py against the rev that was cited by
> verify.
> After filtering out a ton of stuff the net result was::
>
> D:> fsfsverify.py db\revs\503
> [...]
> <Svndiff so: 7473866 ver: 0>
>
> Error InvalidInstruction: Invalid instruction found at offset
> 7492432 (EC)
> Try running with -f to fix the revision
>
> D:> fsfsverify.py -f db\revs\503
> [...]
> Copy 224819 bytes from offset 7473870
> Write 224819 bytes at offset 7473866
> Fixed? :-) Re-run fsfsverify without the -f option
>
> D:>fsfsverify.py db\revs\503
> [...]
> Traceback (most recent call last):
> File "fsfsverify.py", line 1126, in <module>
> options.dumpWindows)
> File "fsfsverify.py", line 565, in verify
> svndiff = Svndiff(f, self.length)
> File "fsfsverify.py", line 456, in __init__
> (self.startingOffset)
> __main__.InvalidSvndiffHeader: Invalid svndiff header at offset
> 7473866
>
> ----- Original Message -----
> From: "Ryan Schmidt" <su...@ryandesign.com>
> To: "Dave Jenkins" <n1...@hotmail.com>
> Cc: <us...@subversion.tigris.org>
> Sent: Tuesday, October 23, 2007 4:00 PM
> Subject: Re: Corruption problems
>
>
>> On Oct 23, 2007, at 14:23, Dave Jenkins wrote:
>>
>>>> Tell us more about your setup.
>>> Sure, Thanks!
>>>
>>> It has been working flawlessly for 2+ years so I have been loathe to
>>> update subversion.
>>>
>>>> OS
>>> Windows Server 2003
>>> Standard Edition
>>>
>>>> OS version
>>> Hmmm... upgraded to Service Pack 2... YESTERDAY. Suspicious?!?
>>>
>>>> Subversion version,
>>> 1.2.3
>>>
>>>> what media is the repository on (local hard disk, local RAID, NFS,
>>>> SAN), could you have hardware problems (have you run some kind of
>>>> hardware test), etc.
>>> Local NTFS hard drive. (Backed up nightly.)
>>
>> You may want to try accessing a copy of your repository using Subversion
>> 1.4.5, see if that works better. Subversion 1.2.x and earlier aren't
>> supported anymore.
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Dave Jenkins <n1...@hotmail.com>.
Ok... we took the opportunity to replace the somewhat dated and potentially
suspect hardware.
It's now a VM on a screaming shiney new box.
I upgraded from subversion 1.2.3 to 1.4.5.
I re-ran the svnadmin verify... same error.
I installed Python and ran fsfsverify.py against the rev that was cited by
verify.
After filtering out a ton of stuff the net result was::
D:> fsfsverify.py db\revs\503
[...]
<Svndiff so: 7473866 ver: 0>
Error InvalidInstruction: Invalid instruction found at offset
7492432 (EC)
Try running with -f to fix the revision
D:> fsfsverify.py -f db\revs\503
[...]
Copy 224819 bytes from offset 7473870
Write 224819 bytes at offset 7473866
Fixed? :-) Re-run fsfsverify without the -f option
D:>fsfsverify.py db\revs\503
[...]
Traceback (most recent call last):
File "fsfsverify.py", line 1126, in <module>
options.dumpWindows)
File "fsfsverify.py", line 565, in verify
svndiff = Svndiff(f, self.length)
File "fsfsverify.py", line 456, in __init__
(self.startingOffset)
__main__.InvalidSvndiffHeader: Invalid svndiff header at offset
7473866
----- Original Message -----
From: "Ryan Schmidt" <su...@ryandesign.com>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: <us...@subversion.tigris.org>
Sent: Tuesday, October 23, 2007 4:00 PM
Subject: Re: Corruption problems
> On Oct 23, 2007, at 14:23, Dave Jenkins wrote:
>
>>> Tell us more about your setup.
>> Sure, Thanks!
>>
>> It has been working flawlessly for 2+ years so I have been loathe to
>> update subversion.
>>
>>> OS
>> Windows Server 2003
>> Standard Edition
>>
>>> OS version
>> Hmmm... upgraded to Service Pack 2... YESTERDAY. Suspicious?!?
>>
>>> Subversion version,
>> 1.2.3
>>
>>> what media is the repository on (local hard disk, local RAID, NFS,
>>> SAN), could you have hardware problems (have you run some kind of
>>> hardware test), etc.
>> Local NTFS hard drive. (Backed up nightly.)
>
> You may want to try accessing a copy of your repository using Subversion
> 1.4.5, see if that works better. Subversion 1.2.x and earlier aren't
> supported anymore.
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Miha Vitorovic <mv...@nil.si>.
Dave,
If it's a FSFS repository you might try running
http://svn.collab.net/viewvc/svn/trunk/contrib/server-side/fsfsverify.py?view=log
. Do a full backup prior to that...
Br,
---
Miha Vitorovic
Inženir v tehničnem področju
Customer Support Engineer
NIL Data Communications, Tivolska cesta 48, 1000 Ljubljana, Slovenia
Phone +386 1 4746 500 Fax +386 1 4746 501 http://www.NIL.si
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 23, 2007, at 14:23, Dave Jenkins wrote:
>> Tell us more about your setup.
> Sure, Thanks!
>
> It has been working flawlessly for 2+ years so I have been loathe
> to update subversion.
>
>> OS
> Windows Server 2003
> Standard Edition
>
>> OS version
> Hmmm... upgraded to Service Pack 2... YESTERDAY. Suspicious?!?
>
>> Subversion version,
> 1.2.3
>
>> what media is the repository on (local hard disk, local RAID,
>> NFS, SAN), could you have hardware problems (have you run some
>> kind of hardware test), etc.
> Local NTFS hard drive. (Backed up nightly.)
You may want to try accessing a copy of your repository using
Subversion 1.4.5, see if that works better. Subversion 1.2.x and
earlier aren't supported anymore.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Dave Jenkins <n1...@hotmail.com>.
> Tell us more about your setup.
Sure, Thanks!
It has been working flawlessly for 2+ years so I have been loathe to update
subversion.
> OS
Windows Server 2003
Standard Edition
> OS version
Hmmm... upgraded to Service Pack 2... YESTERDAY. Suspicious?!?
> Subversion version,
1.2.3
> what media is the repository on (local hard disk, local RAID, NFS, SAN),
> could you have hardware problems (have you run some kind of hardware
> test), etc.
Local NTFS hard drive. (Backed up nightly.)
----- Original Message -----
From: "Ryan Schmidt" <su...@ryandesign.com>
To: "Dave Jenkins" <n1...@hotmail.com>
Cc: <us...@subversion.tigris.org>
Sent: Tuesday, October 23, 2007 2:42 PM
Subject: Re: Corruption problems
>
> On Oct 23, 2007, at 12:06, Dave Jenkins wrote:
>
>> We had a user experience a checksum error on a commit. He tried again
>> and with no error.
>> The next export (performed by the build process) got a checksum error.
>> svnadmin verify then reported a checksum error.
>>
>> We restored the entire repository from a backup that was done just prior
>> to the last successful export/build.
>>
>> Now, after restore, about half way through the revision list svnadmin
>> now reports: svnadmin: Invalid diff stream: insn 6174 cannot be decoded.
>>
>> Any thoughts or suggestions?!?
>
> Tell us more about your setup. OS, OS version, Subversion version, what
> media is the repository on (local hard disk, local RAID, NFS, SAN), could
> you have hardware problems (have you run some kind of hardware test),
> etc.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Corruption problems
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 23, 2007, at 12:06, Dave Jenkins wrote:
> We had a user experience a checksum error on a commit. He tried
> again and with no error.
> The next export (performed by the build process) got a checksum error.
> svnadmin verify then reported a checksum error.
>
> We restored the entire repository from a backup that was done just
> prior to the last successful export/build.
>
> Now, after restore, about half way through the revision list
> svnadmin now reports: svnadmin: Invalid diff stream: insn 6174
> cannot be decoded.
>
> Any thoughts or suggestions?!?
Tell us more about your setup. OS, OS version, Subversion version,
what media is the repository on (local hard disk, local RAID, NFS,
SAN), could you have hardware problems (have you run some kind of
hardware test), etc.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org