Re: Berkeley DB corruption - 'changes' table (WORSE)

["Paul L. Suh" <ps...@apple.com>]

Folks,

OK, the repository is seriously toast. After restoring from a backup,  
we tried to check in additional changes and ran into tons of problems.  
Individual files keep causing repository deaths with errors like:

subversion/libsvn_fs/bdb/strings-table.c:101: (apr_err=160010)
svn: Filesystem has no such string
svn: locate_key: no such string `h6w'

Worse, it's inconsistent throughout the repository. Most files work.  
Some files don't, consistently. Occasionally, if we delete a file from  
the head of the tree, add back a small file (like only a few bytes),  
then replace it in the wc with the real (~1.5-2 MB file) and do a  
check-in it works. Other times it doesn't. The error code is  
consistent, but the 3 letters `h6w' are not -- other times they are  
`f8r' or other letter-number-letter combos. The repository dies the  
same way when I run 'svnadmin dump' on it, even after running 'svnadmin  
restore'.

Our previous backup to that was a few days back thanks to a messed up  
tape rotation. :-P At least that one seems clean.

BTW is there an Issue Tracker item on this? Or should I open one?


--Paul


On Oct 2, 2003, at 2:35 PM, Paul L. Suh wrote:

> Karl,
>
> Yes, the admin did turn off httpd before doing the recover.
>
>
> --Paul
>
>
> On Oct 2, 2003, at 1:36 PM, kfogel@collab.net wrote:
>
>> When the person ran 'svnadmin recover', did they observe the cautions
>> mentioned in
>>
>>    http://subversion.tigris.org/project_faq.html#wedged-repos
>>
>> ... especially the part about shutting off Apache and any other
>> process that might conceivably access the repository?
>>
>> -Karl
>>
>> "Paul L. Suh" <ps...@apple.com> writes:
>>> Folks,
>>>
>>> Subversion 0.24.2 (Yes I know it's old, we're planning on updating to
>>> 0.30 in 2 days, just as soon as the current project phase
>>> finishes. Curse you, Murphy!) We're getting the error:
>>>
>>> "Berkeley DB error while opening `copies' table for filesystem
>>> /Library/Repository/MacOSXTroubleshooting/db: Invalid argument "
>>>
>>> Three questions:
>>>
>>> 1) Is there any way to recover? (Not a screaming necessity since we
>>> have daily backups, but there were a goodly number of commits
>>> yesterday since we're getting close to a release.)
>>>
>>> 2) Have there been any changes in the BDB table structure in more
>>> recent versions that would make this problem unlikely to happen?
>>>
>>> 3) Is there an Issue Tracker entry for this? I looked but couldn't
>>> find one.
>>>
>>> Additional detail follows.
>>>
>>>
>>> --Paul
>>>
>>> Paul L. Suh
>>> Trainer/Curriculum Developer
>>> Apple Computer
>>> 1892 Preston White Drive
>>> Reston, VA 20191
>>>
>>> (703) 264-3216
>>> http://train.apple.com/
>>>
>>>
>>> Original error e-mail from user:
>>>
>>>> Hi,
>>>>
>>>> Any idea how to fix this?
>>>>
>>>> A little while ago, svn failed due to a full disk on
>>>> my machine:
>>>>
>>>> [da0703a-dhcp110:Current/Production_Materials/Guide]
>>>> ellend% svn update 09_FileInternetShareing.fm
>>>> Restored 09_FileInternetShareing.fm
>>>> subversion/libsvn_wc/log.c:294: (apr_err=155009)
>>>> svn: Problem running log
>>>> svn: in directory ''
>>>> subversion/libsvn_subr/io.c:216: (apr_err=28)
>>>> svn: No space left on device
>>>> svn: svn_io_copy_file: error copying
>>>> '.svn/tmp/text-base/09_FileInternetShareing.fm.svn-base'
>>>> to '09_FileInternetShareing.fm.tmp'
>>>>
>>>> I emptied my trash to free up space.
>>>>
>>>> Now, when I try to check in a file I get this:
>>>>
>>>> [da0703a-dhcp110:HDEssentials/Current/Production_Materials]
>>>> ellend% svn ci Guide/09_FileInternetShareing.fm
>>>> subversion/libsvn_wc/lock.c:116: (apr_err=155004)
>>>> svn: Attempted to lock an already-locked dir
>>>> svn: working copy locked:
>>>> /Users/ellend/Desktop/HDEssentials/Current/Production_Materials/ 
>>>> Guide
>>>> svn: run 'svn cleanup' to remove locks (type 'svn help
>>>> cleanup' for details)
>>>>
>>>> So then I try to run svn cleanup, and I get this:
>>>>
>>>> [da0703a-dhcp110:HDEssentials/Current/Production_Materials]
>>>> ellend% svn cleanup
>>>> subversion/libsvn_wc/log.c:294: (apr_err=155009)
>>>> svn: Problem running log
>>>> svn: in directory 'Guide'
>>>> subversion/libsvn_wc/log.c:1145: (apr_err=155009)
>>>> svn: start_handler: error processing command
>>>> 'modify-entry' in 'Guide'
>>>> subversion/libsvn_wc/log.c:487: (apr_err=155009)
>>>> svn: error getting file affected time on
>>>> `Guide/.svn/props/09_FileInternetShareing.fm'
>>>
>>>
>>> svnadmin recover produces:
>>>
>>> Acquiring exclusive lock on repository db.
>>> Recovery is running, please stand by...
>>> Recovery completed.
>>> subversion/libsvn_fs/bdb/bdb-err.c:61: (apr_err=160029)
>>> svn: Berkeley DB error
>>> svn: Berkeley DB error while opening `copies' table for filesystem
>>> /Volumes/data/Repository/MacOSXTroubleshooting/db:
>>> Invalid argument
>>>
>>> Doing a direct db_recover results in:
>>>
>>> sudo /opt/BerkeleyDB.4.1/bin/db_recover -v
>>> db_recover: Finding last valid log LSN: file: 4311 offset 581049
>>> db_recover: Recovery starting from [4311][578974]
>>> db_recover: Recovery complete at Thu Oct  2 08:30:57 2003
>>> db_recover: Maximum transaction ID 8000000b Recovery checkpoint
>>> [4311][581049]
>>> db_recover: Recovery complete at Thu Oct  2 08:30:57 2003
>>> db_recover: Maximum transaction id 80000000 Recovery checkpoint
>>> [4311][581049]
>>>
>>> But svnadmin recover after that still produces the same problem.
>>>
>>> Systems involved are Mac OS X 10.2.6 on client and server, I built  
>>> the
>>> entire 0.24.2 tarball from source, Berkeley DB 4.1.25, httpd 2.0.46
>>> custom built from source, all access via http.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>>> For additional commands, e-mail: users-help@subversion.tigris.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: users-help@subversion.tigris.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Berkeley DB corruption - 'changes' table (WORSE)

[kf...@collab.net]

"Paul L. Suh" <ps...@apple.com> writes:
> OK, the repository is seriously toast. After restoring from a backup,
> we tried to check in additional changes and ran into tons of problems.
> Individual files keep causing repository deaths with errors like:
> 
> subversion/libsvn_fs/bdb/strings-table.c:101: (apr_err=160010)
> svn: Filesystem has no such string
> svn: locate_key: no such string `h6w'
> 
> Worse, it's inconsistent throughout the repository. Most files work.
> Some files don't, consistently. Occasionally, if we delete a file from
> the head of the tree, add back a small file (like only a few bytes),
> then replace it in the wc with the real (~1.5-2 MB file) and do a
> check-in it works. Other times it doesn't. The error code is
> consistent, but the 3 letters `h6w' are not -- other times they are
> `f8r' or other letter-number-letter combos. The repository dies the
> same way when I run 'svnadmin dump' on it, even after running
> 'svnadmin  restore'.
> 
> Our previous backup to that was a few days back thanks to a messed up
> tape rotation. :-P At least that one seems clean.
> 
> BTW is there an Issue Tracker item on this? Or should I open one?

Wow.  Paul, I'm really sorry this is happening to you... It's worth
opening an Issue Tracker item when we have a reproduction recipe.
Until then, though, the issue doesn't do a whole lot of good.  The
issue will just sit there, gathering dust, until someday someone
closes it saying "unable to reproduce".

Maybe if you post the whole history of your repository, with tons of
details, we can figure out a reproduction recipe.  (Or if you did
already, just post the archive url.)

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org