You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Evan Driscoll <dr...@cs.wisc.edu> on 2012/01/26 22:04:43 UTC
"database disk image is malformed"
[There's a lot of background information; I have some questions near the
end.]
When I try to commit, I see the following:
Sending <file>
...
Sending <file>
Transmitting file data ............svn: Commit failed (details follow):
svn: database disk image is malformed
svn: database disk image is malformed
Vital statistics:
Repository format: FSFS stored on AFS
Access via: svn+ssh
Subversion version: 1.6.0
The subversion FAQ [1] states:
If you are using the FSFS repository back end, then storing the
repository on a modern NFS server (i.e., one that supports locking)
should be fine.
Here's what one site [2] says about file locking on AFS.
The file locking mechanism in AFS does not really follow POSIX6
semantics. There are a few issues to mention:
- Files may only be locked as a whole; regions of a file may not be
locked.
- File locking only works properly and reliably from a single system.
If a file is locked from one client and an attempt is made to
access the file from another client, the error EWOULDBLOCK is
returned.
- There is no deadlock prevention in AFS, so deadlock situations can
occur with file locking.
- Any program that attempts to use byte-range file locking in AFS
will get a message from the cache manager warning that other users
may be accessing the same file. Usually these messages can be
safely ignored.
Generally we don't recommend including applications that depend on
file locking in the AFS file space.
From the second point there it sounds to me like AFS provides the
environment that Subversion needs, but I'm no expert.
I found an old thread [3] that seems to suggest dumping and reloading
the repository should fix the problem, but also suggests removing the
db/rep-cache.db file and perhaps disabling rep-sharing. I did the
latter, and it cleared up the problem.
My questions:
1. Did removing rep-cache.db fix it, or is there still a potential for
some latent repository corruption? Is there any way to check a
repository for something like that?
2. Regarding dumping and reloading vs deleting rep-cache.db, what are
the tradeoffs? How do I decide which one to do? The ~15% savings is
a little nice but not even remotely essential. Does that mean that
deleting rep-sharing.db is more attractive since it's less work?
3. What caused this problem in the first place? Was it putting the
repository on AFS? Is there any *safe* way to do that with using the
svn+ssh protocol, which is attractive because it requires no setup
on my part, no worries about security of access, etc.?
4. A later reply [4] to the aforementioned thread states that
Subversion 1.7+ uses BDB with revprops.db. What is in this file? Is
it something which is really important (e.g. commit logs and users)
or just something inessential like rep-sharing.db?
If it's essential, does that mean that whatever caused corruption to
our rep-sharing.dp could, if we upgrade, cause actual problems and
made us restore from backup?
Evan Driscoll
[1] http://subversion.apache.org/faq.html#nfs
[2] http://computing.fnal.gov/unixatfermilab/html/afs.html#57212
[3]
http://mail-archives.apache.org/mod_mbox/subversion-users/201012.mbox/%3C201012161703.01954.rdorsch@web.de%3E
[4]
http://mail-archives.apache.org/mod_mbox/subversion-users/201012.mbox/%3C20101218184143.GA8544@daniel3.local%3E
Re: "database disk image is malformed"
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evan Driscoll wrote on Fri, Jan 27, 2012 at 10:31:32 -0600:
> On 1/27/2012 3:41, Daniel Shahaf wrote:
> > Evan Driscoll wrote on Thu, Jan 26, 2012 at 15:04:43 -0600:
> >> 1. Did removing rep-cache.db fix it, or is there still a potential for
> >> some latent repository corruption?
> >
> > Yes. rep-cache.db is used by the commit process. If that file was
> > silently corrupted, it's possible that the text: and props:
> > lines on node-revisions point to invalid data. (In English: versioned
> > properties, file contents, and symlinks targets may be affected.)
> >
> > I think the list archives and the developer docs explain this, so I'm
> > not going into detail here. But feel free to follow up (on list) if you
> > have questions that those don't cover.
>
> I ran 'svnadmin verify' (as Mark Cooke suggested) and it did not
> indicate any problems. Does that mean there aren't any (or that the
> probability is vanishingly small)?
No. If SQLite returned wrong answers when queried (as opposed to
declaring itself corrupt), it's possible that a commit process would
have set the contents of a committed revision to the contents of some
old file rather than to the newly-submitted contents.
'svnadmin verify' cannot identify such a corruption.
How likely it is --- judge for yourself.
>
> Evan
>
>
Re: "database disk image is malformed"
Posted by Evan Driscoll <dr...@cs.wisc.edu>.
On 1/27/2012 3:41, Daniel Shahaf wrote:
> Evan Driscoll wrote on Thu, Jan 26, 2012 at 15:04:43 -0600:
>> 1. Did removing rep-cache.db fix it, or is there still a potential for
>> some latent repository corruption?
>
> Yes. rep-cache.db is used by the commit process. If that file was
> silently corrupted, it's possible that the text: and props:
> lines on node-revisions point to invalid data. (In English: versioned
> properties, file contents, and symlinks targets may be affected.)
>
> I think the list archives and the developer docs explain this, so I'm
> not going into detail here. But feel free to follow up (on list) if you
> have questions that those don't cover.
I ran 'svnadmin verify' (as Mark Cooke suggested) and it did not
indicate any problems. Does that mean there aren't any (or that the
probability is vanishingly small)?
Evan
Re: "database disk image is malformed"
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Evan Driscoll wrote on Thu, Jan 26, 2012 at 15:04:43 -0600:
> 1. Did removing rep-cache.db fix it, or is there still a potential for
> some latent repository corruption?
Yes. rep-cache.db is used by the commit process. If that file was
silently corrupted, it's possible that the text: and props:
lines on node-revisions point to invalid data. (In English: versioned
properties, file contents, and symlinks targets may be affected.)
I think the list archives and the developer docs explain this, so I'm
not going into detail here. But feel free to follow up (on list) if you
have questions that those don't cover.
RE: "database disk image is malformed"
Posted by "Cooke, Mark" <ma...@siemens.com>.
> -----Original Message-----
> From: Evan Driscoll [mailto:driscoll@cs.wisc.edu]
> Sent: 26 January 2012 21:05
> Subject: "database disk image is malformed"
>
> [There's a lot of background information; I have some
> questions near the end.]
>
>
> When I try to commit, I see the following:
>
> Sending <file>
> ...
> Sending <file>
> Transmitting file data ............svn: Commit failed
> (details follow):
> svn: database disk image is malformed
> svn: database disk image is malformed
>
>
> Vital statistics:
>
> Repository format: FSFS stored on AFS
> Access via: svn+ssh
> Subversion version: 1.6.0
[Knee jerk:] 1.6.0 is quite old now (march 2009). Any chance of upgrading?
> The subversion FAQ [1] states:
>
> If you are using the FSFS repository back end, then storing the
> repository on a modern NFS server (i.e., one that supports locking)
> should be fine.
>
> Here's what one site [2] says about file locking on AFS.
>
> The file locking mechanism in AFS does not really follow POSIX6
> semantics. There are a few issues to mention:
>
> - Files may only be locked as a whole; regions of a file may not be
> locked.
> - File locking only works properly and reliably from a
> single system.
> If a file is locked from one client and an attempt is made to
> access the file from another client, the error EWOULDBLOCK is
> returned.
> - There is no deadlock prevention in AFS, so deadlock
> situations can occur with file locking.
> - Any program that attempts to use byte-range file locking in AFS
> will get a message from the cache manager warning that other
> users may be accessing the same file. Usually these messages
> can be safely ignored.
>
> Generally we don't recommend including applications that depend on
> file locking in the AFS file space.
>
> From the second point there it sounds to me like AFS provides the
> environment that Subversion needs, but I'm no expert.
>
>
> I found an old thread [3] that seems to suggest dumping and reloading
> the repository should fix the problem, but also suggests removing the
> db/rep-cache.db file and perhaps disabling rep-sharing. I did the
> latter, and it cleared up the problem.
>
>
> My questions:
>
> 1. Did removing rep-cache.db fix it, or is there still a potential for
> some latent repository corruption? Is there any way to check a
> repository for something like that?
Check out svnadmin verify:-
http://svnbook.red-bean.com/en/1.6/svn.ref.svnadmin.c.verify.html
> 2. Regarding dumping and reloading vs deleting rep-cache.db, what are
> the tradeoffs? How do I decide which one to do? The ~15%
> savings is a little nice but not even remotely essential.
> Does that mean that deleting rep-sharing.db is more attractive
> since it's less work?
>
> 3. What caused this problem in the first place? Was it putting the
> repository on AFS? Is there any *safe* way to do that
> with using the svn+ssh protocol, which is attractive because
> it requires no setup on my part, no worries about security
> of access, etc.?
>
> 4. A later reply [4] to the aforementioned thread states that
> Subversion 1.7+ uses BDB with revprops.db. What is in
> this file? Is it something which is really important
> (e.g. commit logs and users) or just something inessential
> like rep-sharing.db?
I believe that this uses SQLite, not BDB ?
> If it's essential, does that mean that whatever caused
> corruption to our rep-sharing.dp could, if we upgrade,
> cause actual problems and made us restore from backup?
>
> Evan Driscoll
>
>
>
> [1] http://subversion.apache.org/faq.html#nfs
> [2] http://computing.fnal.gov/unixatfermilab/html/afs.html#57212
> [3]
> http://mail-archives.apache.org/mod_mbox/subversion-users/201012.mbox/%3C201012161703.01954.rdorsch@web.de%3E
> [4]
> http://mail-archives.apache.org/mod_mbox/subversion-users/201012.mbox/%3C20101218184143.GA8544@daniel3.local%3E
>