You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Dave Camp <da...@thinbits.com> on 2004/10/12 17:59:00 UTC
Database corrupted after power failure
The UPS on our server failed last night and we seem to have experienced
some damage our the database. I've reviewed the docs and tried the
following:
[absinthe:/svn/foo] dave% svnadmin lstxns .
svn: Berkeley DB error while checkpointing after Berkeley DB
transaction for filesystem db:
Invalid argument
svn: bdb: Ignoring log file: db/log.0000000002: magic number 0, not
40988
svn: bdb: DB_ENV->log_put: 2: Invalid argument
svn: bdb: txn_checkpoint: failed to flush the buffer cache Invalid
argument
svn: Berkeley DB error while closing 'nodes' database for filesystem db:
Invalid argument
svn: bdb: Ignoring log file: db/log.0000000002: magic number 0, not
40988
svn: bdb: DB_ENV->log_put: 2: Invalid argument
svn: bdb: Ignoring log file: db/log.0000000002: magic number 0, not
40988
svn: bdb: DB_ENV->log_put: 2: Invalid argument
[absinthe:/svn] dave% sudo svnadmin recover foo/
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Ignoring log file: foo/db/log.0000000002: magic number 0, not
40988
svn: bdb: Invalid log file: log.0000000002: Invalid argument
svn: bdb: First log record not found
svn: bdb: PANIC: Invalid argument
What do I do now? Neither of the suggested commands seems to get very
far?
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by "C. Michael Pilato" <cm...@collab.net>.
Dave Camp <da...@thinbits.com> writes:
> -rw-r--r-- 1 dave dave 1738 11 Oct 13:11 DB_CONFIG
> -rw-r--r-- 1 dave dave 61440 11 Oct 18:05 changes
> -rw-r--r-- 1 dave dave 8192 11 Oct 19:57 copies
> -rw-r--r-- 1 dave dave 4 11 Oct 13:11 fs-type
> -rw-r--r-- 1 dave dave 65502 11 Oct 19:57 log.0000000002
> -rw-r--r-- 1 dave dave 1047376 11 Oct 18:05 log.0000000005
> -rw-r--r-- 1 dave dave 98238 11 Oct 18:05 log.0000000006
> -rw-r--r-- 1 dave dave 40960 11 Oct 18:05 nodes
> -rw-r--r-- 1 dave dave 45056 11 Oct 18:05 representations
> -rw-r--r-- 1 dave dave 8192 11 Oct 19:57 revisions
> -rw-r--r-- 1 dave dave 4300800 11 Oct 18:05 strings
> -rw-r--r-- 1 dave dave 8192 11 Oct 19:57 transactions
> -rw-r--r-- 1 dave dave 8192 11 Oct 19:57 uuids
There's a noticeable gap in your log file set. Try this:
cp -a repos repos.backup
rm repos/db/log.0000000002
db_recover -vech repos/db
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Dave Camp <da...@thinbits.com>.
On Oct 13, 2004, at 7:22 AM, Tobias Ringström wrote:
> Dave Camp wrote:
>
>> -rw-r--r-- 1 dave dave 65502 11 Oct 19:57 log.0000000002
>> -rw-r--r-- 1 dave dave 1047376 11 Oct 18:05 log.0000000005
>> -rw-r--r-- 1 dave dave 98238 11 Oct 18:05 log.0000000006
>
> This is very odd. What is log.*2 doing there, and why is it newer than
> 5 and 6? Do you recognize the times? When did the crash occur?
The UPS failed somewhere between 18:15 and 18:30 I'd guess. I don't
think anyone was using the repository when it failed (I was the last
one in the building). I have no idea what the file from 19:57 is.
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Tobias Ringström <to...@ringstrom.mine.nu>.
Dave Camp wrote:
> -rw-r--r-- 1 dave dave 65502 11 Oct 19:57 log.0000000002
> -rw-r--r-- 1 dave dave 1047376 11 Oct 18:05 log.0000000005
> -rw-r--r-- 1 dave dave 98238 11 Oct 18:05 log.0000000006
This is very odd. What is log.*2 doing there, and why is it newer than 5
and 6? Do you recognize the times? When did the crash occur?
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Dave Camp <da...@thinbits.com>.
On Oct 12, 2004, at 9:24 PM, kfogel@collab.net wrote:
> Dave Camp <da...@thinbits.com> writes:
>> # cd foo/db
>> # /usr/local/BerkeleyDB.4.2/bin/db_recover -c
>> db_recover: Ignoring log file: log.0000000002: magic number 0, not
>> 40988
>> db_recover: Invalid log file: log.0000000002: Invalid argument
>> db_recover: First log record not found
>> db_recover: PANIC: Invalid argument
>> db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
>> recovery
>>
>> At this point, I've given up. The repository was brand new (we just
>> switched to subversion on Monday) so starting over wasn't a big
>> deal. However, it is somewhat distressing that the database can't be
>> fixed. I was under the impression that the journalling was supposed to
>> prevent that... especially considering that the file system it's on is
>> journaled as well (Mac OS X Server with HFS+).
>
> Yuck.
>
> Do you have an 'ls -lR' of the repository?
-rw-r--r-- 1 dave dave 1738 11 Oct 13:11 DB_CONFIG
-rw-r--r-- 1 dave dave 61440 11 Oct 18:05 changes
-rw-r--r-- 1 dave dave 8192 11 Oct 19:57 copies
-rw-r--r-- 1 dave dave 4 11 Oct 13:11 fs-type
-rw-r--r-- 1 dave dave 65502 11 Oct 19:57 log.0000000002
-rw-r--r-- 1 dave dave 1047376 11 Oct 18:05 log.0000000005
-rw-r--r-- 1 dave dave 98238 11 Oct 18:05 log.0000000006
-rw-r--r-- 1 dave dave 40960 11 Oct 18:05 nodes
-rw-r--r-- 1 dave dave 45056 11 Oct 18:05 representations
-rw-r--r-- 1 dave dave 8192 11 Oct 19:57 revisions
-rw-r--r-- 1 dave dave 4300800 11 Oct 18:05 strings
-rw-r--r-- 1 dave dave 8192 11 Oct 19:57 transactions
-rw-r--r-- 1 dave dave 8192 11 Oct 19:57 uuids
Thanks,
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Scott Palmer wrote:
>
> On Oct 13, 2004, at 8:05 PM, Greg Hudson wrote:
>
>>> They both compress. The reasons BDB is bigger are complex, and have
>>> to do with how databases allocate storage.
>>
>>
>> Er, no, BDB stores the head revision in uncompressed plaintext,
>> whereas FSFS stores all file revisions as deltas against something.
>> (The first rev of a file is stored as a delta against empty.)
>>
>
> How does FSFS store the HEAD revision then? If it needs to process
> deltas to get the HEAD revision that would lead to performance problems
> would it not? I was under the impression that the usually way to store
> versions is to use reverse deltas from the HEAD revision to get the
> older revision, so generally the HEAD revision is stored verbatim (as
> you say it is for BDB).
It uses skip deltas, so it only needs to apply at most log(n) deltas to
get to any given revision. It's a little bit slower, but not noticable
in my experience.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Scott Palmer <sc...@2connected.org>.
On Oct 13, 2004, at 8:05 PM, Greg Hudson wrote:
>> They both compress. The reasons BDB is bigger are complex, and have
>> to do with how databases allocate storage.
>
> Er, no, BDB stores the head revision in uncompressed plaintext,
> whereas FSFS stores all file revisions as deltas against something.
> (The first rev of a file is stored as a delta against empty.)
>
How does FSFS store the HEAD revision then? If it needs to process
deltas to get the HEAD revision that would lead to performance problems
would it not? I was under the impression that the usually way to store
versions is to use reverse deltas from the HEAD revision to get the
older revision, so generally the HEAD revision is stored verbatim (as
you say it is for BDB).
Scott
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Mark Phippard <Ma...@softlanding.com>.
Greg Hudson <gh...@MIT.EDU> wrote on 10/14/2004 11:55:21 AM:
> > I believe Greg's point is that the presence of these full-texts,
> > especially if you have a lot of branches, accounts for a lot of the
> > difference in the repository sizes. Also, if I understand Greg, even
the
> > head revision in fsfs undergoes some degree of compression, such as
> > stripping out whitespace etc...
>
> Stripping out whitespace? No.
>
> In FSFS, the first rev of a file is stored as a delta against empty,
> which is a form of compression comparable to gzip (though not quite as
> good). Later revs of the file are stored as deltas against earlier
> revs--but not always the most immediate earlier rev. See
> http://svn.collab.net/repos/svn/trunk/notes/skip-deltas for details.
To us mortals, a delta against empty would imply the same end result as a
full text. I know you didn't mean that, so I was suggesting that there is
some degree of compression even in that scenario.
What I meant by stripping out whitespace, is that if I had 100 consecutive
spaces the algorithm would probably compress that down to something far
less than 100 bytes. I have no idea what the delta algorithm, or gzip,
actually does, I just thought that might be something that us lay persons
might understand.
Thanks
Mark
_____________________________________________________________________________
Scanned for SoftLanding Systems, Inc. by IBM Email Security Management Services powered by MessageLabs.
_____________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2004-10-14 at 10:34, Mark Phippard wrote:
> kfogel@newton.ch.collab.net wrote on 10/14/2004 08:37:08 AM:
> > Sure, the head revision is uncompressed. But the vast majority of
> > data in the repository is compressed, and therefore I think it's fair,
> > when speaking in general terms, to say that the repository is
> > compressed. (Unless the question was specifically about head, which I
> > may have missed.)
The historical data in the repository may not be "the vast majority of
the data" in some repositories. For instance, my work repository is
primarily an integration repository--imports of external source code
with a few local mods. Although we do import new versions of the
external source code periodically (thus creating history), the ratio of
head data to history is quite large.
> I believe Greg's point is that the presence of these full-texts,
> especially if you have a lot of branches, accounts for a lot of the
> difference in the repository sizes. Also, if I understand Greg, even the
> head revision in fsfs undergoes some degree of compression, such as
> stripping out whitespace etc...
Stripping out whitespace? No.
In FSFS, the first rev of a file is stored as a delta against empty,
which is a form of compression comparable to gzip (though not quite as
good). Later revs of the file are stored as deltas against earlier
revs--but not always the most immediate earlier rev. See
http://svn.collab.net/repos/svn/trunk/notes/skip-deltas for details.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Mark Phippard <Ma...@softlanding.com>.
kfogel@newton.ch.collab.net wrote on 10/14/2004 08:37:08 AM:
> Greg Hudson <gh...@MIT.EDU> writes:
> > > They both compress. The reasons BDB is bigger are complex, and have
> > > to do with how databases allocate storage.
> >
> > Er, no, BDB stores the head revision in uncompressed plaintext,
> > whereas FSFS stores all file revisions as deltas against something.
> > (The first rev of a file is stored as a delta against empty.)
>
> Sure, the head revision is uncompressed. But the vast majority of
> data in the repository is compressed, and therefore I think it's fair,
> when speaking in general terms, to say that the repository is
> compressed. (Unless the question was specifically about head, which I
> may have missed.)
I believe Greg's point is that the presence of these full-texts,
especially if you have a lot of branches, accounts for a lot of the
difference in the repository sizes. Also, if I understand Greg, even the
head revision in fsfs undergoes some degree of compression, such as
stripping out whitespace etc...
Mark
_____________________________________________________________________________
Scanned for SoftLanding Systems, Inc. by IBM Email Security Management Services powered by MessageLabs.
_____________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by kf...@collab.net.
Greg Hudson <gh...@MIT.EDU> writes:
> > They both compress. The reasons BDB is bigger are complex, and have
> > to do with how databases allocate storage.
>
> Er, no, BDB stores the head revision in uncompressed plaintext,
> whereas FSFS stores all file revisions as deltas against something.
> (The first rev of a file is stored as a delta against empty.)
Sure, the head revision is uncompressed. But the vast majority of
data in the repository is compressed, and therefore I think it's fair,
when speaking in general terms, to say that the repository is
compressed. (Unless the question was specifically about head, which I
may have missed.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by Greg Hudson <gh...@MIT.EDU>.
> They both compress. The reasons BDB is bigger are complex, and have
> to do with how databases allocate storage.
Er, no, BDB stores the head revision in uncompressed plaintext,
whereas FSFS stores all file revisions as deltas against something.
(The first rev of a file is stored as a delta against empty.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Is FSFS compressing stored data?
Posted by kf...@collab.net.
quastst@in-euro.de writes:
> I tested a bit the differences between FSFS-repos and BDB-repos. I
> found that FSFS-repos must be compressed. Is this true? I found no
> information about it in chapter 5 of the book. Btw FSFS-repos where
> slight slower than BDB-repos in my tests. For example a small project
> only 8.5 MB was imported. As result I got 6s by BDB and 13 by FSFS,
> but the size was 5.8 MB for FSFS and 13 MB for BDB. The project was a
> mixture of dirs, sources, binaries and tars. If FSFS compresses
> files, is it possible to make it in BDB to?
They both compress. The reasons BDB is bigger are complex, and have
to do with how databases allocate storage.
By the way, when you make a new post to the mailing list, please don't
do it by hitting Reply to an existing post and then changing the
subject line. The mailer still remembers what thread you replied to,
and so your post shows up as being part of that thread.
http://subversion.tigris.org/mailing-list-guidelines.html#fresh-post
has more on this.
Thanks,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Is FSFS compressing stored data?
Posted by qu...@in-euro.de.
Hello,
I tested a bit the differences between FSFS-repos and BDB-repos. I found that
FSFS-repos must be compressed. Is this true?
I found no information about it in chapter 5 of the book. Btw FSFS-repos where
slight slower than BDB-repos in my tests. For example a small project only 8.5
MB was imported. As result I got 6s by BDB and 13 by FSFS, but the size was 5.8
MB for FSFS and 13 MB for BDB. The project was a mixture of dirs, sources,
binaries and tars.
If FSFS compresses files, is it possible to make it in BDB to?
Greets
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by kf...@collab.net.
Dave Camp <da...@thinbits.com> writes:
> # cd foo/db
> # /usr/local/BerkeleyDB.4.2/bin/db_recover -c
> db_recover: Ignoring log file: log.0000000002: magic number 0, not 40988
> db_recover: Invalid log file: log.0000000002: Invalid argument
> db_recover: First log record not found
> db_recover: PANIC: Invalid argument
> db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
> recovery
>
> At this point, I've given up. The repository was brand new (we just
> switched to subversion on Monday) so starting over wasn't a big
> deal. However, it is somewhat distressing that the database can't be
> fixed. I was under the impression that the journalling was supposed to
> prevent that... especially considering that the file system it's on is
> journaled as well (Mac OS X Server with HFS+).
Yuck.
Do you have an 'ls -lR' of the repository?
Obviously, if the OS filesystem or disk itself lost data, there's
nothing Subversion can do about it. That seems unlikely, though. An
'ls -lR' might help us detect that.
See also:
http://www.contactor.se/~dast/svnusers/archive-2004-03/0893.shtml
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Dave Camp <da...@thinbits.com>.
On Oct 13, 2004, at 4:29 AM, Tobias Ringström wrote:
> Dave Camp wrote:
>
>> At this point, I've given up. The repository was brand new (we just
>> switched to subversion on Monday) so starting over wasn't a big deal.
>> However, it is somewhat distressing that the database can't be fixed.
>> I was under the impression that the journalling was supposed to
>> prevent that... especially considering that the file system it's on
>> is journaled as well (Mac OS X Server with HFS+).
>
> I've heard about almost exactly this problem several times recently,
> and the common denominator is OS X. Two people even claimed that the
> last access to the repository was an hour before the crash, and the
> repository still became corrupt. I would say that such a thing is
> impossible, but when it has happened twice I'm not so sure...
We discovered yesterday (after a number of non-fatal database wedges)
that the umask for bash and tcsh were not correct. We have fixed that
(set to 002 now) and we'll see how day 3 on subversion goes... :-)
It would be nice if subversion could give more useful error messages in
this case.
> When I see problems such as this one, I usually fix them by removing
> the __db.* and log.* files in the db directory of the repository and
> then dump and load the old repository into a new one and copy the
> contents of conf/ and hooks/. The dump+load is *probably* not
> necessary, but it's safer.
Thanks for the tip. The next task on my list is a daily cron job for
hotbackup.py to guarantee we don't ever lose more than a day's worth of
changes.
> Try to upgrade to 1.1.0 and use FSFS (svnadmin create --fs-type fsfs
> repos) if at all possible.
We are already using 1.1.0 (we waited for it to ship before switching
to subversion).
It's not immediately obvious from what I've read that FSFS is "worthy"
of storing my source yet (i.e. has it been proven to be completely
stable) or how it performs better in failure cases. But that could just
be my ignorance. I'll have to go read more about FSFS I guess... it's
been a while since I first looked at it.
Thanks,
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Tobias Ringström <to...@ringstrom.mine.nu>.
Dave Camp wrote:
> I tried the following:
>
> # cd foo/db
> # /usr/local/BerkeleyDB.4.2/bin/db_recover -c
> db_recover: Ignoring log file: log.0000000002: magic number 0, not 40988
> db_recover: Invalid log file: log.0000000002: Invalid argument
> db_recover: First log record not found
> db_recover: PANIC: Invalid argument
> db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
> recovery
>
> At this point, I've given up. The repository was brand new (we just
> switched to subversion on Monday) so starting over wasn't a big deal.
> However, it is somewhat distressing that the database can't be fixed.
> I was under the impression that the journalling was supposed to
> prevent that... especially considering that the file system it's on is
> journaled as well (Mac OS X Server with HFS+).
I've heard about almost exactly this problem several times recently, and
the common denominator is OS X. Two people even claimed that the last
access to the repository was an hour before the crash, and the
repository still became corrupt. I would say that such a thing is
impossible, but when it has happened twice I'm not so sure...
When I see problems such as this one, I usually fix them by removing the
__db.* and log.* files in the db directory of the repository and then
dump and load the old repository into a new one and copy the contents of
conf/ and hooks/. The dump+load is *probably* not necessary, but it's safer.
Try to upgrade to 1.1.0 and use FSFS (svnadmin create --fs-type fsfs
repos) if at all possible.
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by Dave Camp <da...@thinbits.com>.
On Oct 12, 2004, at 11:21 AM, kfogel@collab.net wrote:
> Dave Camp <da...@thinbits.com> writes:
>> What do I do now? Neither of the suggested commands seems to get very
>> far?
>
> What happens if you try '/path/to/Berkeley/bin/db_recover -c'? (We
> really ought to have a "catastrophic" recover switch available to
> 'svnadmin recover', but apparently we don't, sigh.)
I tried the following:
# cd foo/db
# /usr/local/BerkeleyDB.4.2/bin/db_recover -c
db_recover: Ignoring log file: log.0000000002: magic number 0, not 40988
db_recover: Invalid log file: log.0000000002: Invalid argument
db_recover: First log record not found
db_recover: PANIC: Invalid argument
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
recovery
At this point, I've given up. The repository was brand new (we just
switched to subversion on Monday) so starting over wasn't a big deal.
However, it is somewhat distressing that the database can't be fixed. I
was under the impression that the journalling was supposed to prevent
that... especially considering that the file system it's on is
journaled as well (Mac OS X Server with HFS+).
Thanks,
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Database corrupted after power failure
Posted by kf...@collab.net.
Dave Camp <da...@thinbits.com> writes:
> What do I do now? Neither of the suggested commands seems to get very
> far?
What happens if you try '/path/to/Berkeley/bin/db_recover -c'? (We
really ought to have a "catastrophic" recover switch available to
'svnadmin recover', but apparently we don't, sigh.)
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org