You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Malcolm Rowe <ma...@farside.org.uk> on 2007/03/07 09:35:14 UTC

Testers wanted for FSFS svnadmin recover feature (r23595)

On Wed, Mar 07, 2007 at 01:12:46AM -0800, malcolm@tigris.org wrote:
> Implement 'svnadmin recover' for FSFS, recreating the db/current file
> from information in the rev/ files.
> 

I _have_ tested this, and the algorithm is pretty simple --- But:

I'd really appreciate some level of confidence testing on a few more
repositories, particularly those that are particularly large or
particularly weird (anything converted from another vcs, perhaps).

By definition, this code is only going to be run when it absolutely
needs to work, and probably isn't going to be exercised at all
otherwise, so current test coverage isn't great.

If anyone has a repository that they don't mind running development code
on to test this feature, this is what you need to do:

1. Make a backup of your repository.  If you _really_ trust me, just make
   a backup of db/current.
2. Optionally remove db/current to prove that we're recreating it properly.
   (This will obviously make the repository unusable to any clients).
3. Run "svnadmin recover repo/".
4. Wait, as svnadmin reads all of the revision files.  Any writers are
   blocked for the duration of recovery.
   (It'd be nice to get some timing data here as well.  My tests had too
   much variability, but it looked like a couple of minutes to recover
   against [a mirror of] the main Subversion repository).
5. Check that the new db/current is identical to the old db/current.
   Scream if not.

That is all.

Many thanks,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Daniel Rall <dl...@collab.net>.
On Thu, 08 Mar 2007, Malcolm Rowe wrote:

> On Wed, Mar 07, 2007 at 05:06:32PM -0800, Daniel Rall wrote:
> > > I think that it would be useful, especially since there's nothing that
> > > gets broken by an early cancellation.
> > 
> > +1.  Since we're already changed the FS-layer API, might as well add
> > the parameters necessary for cancellation while we're at it.
> > 
> 
> r23732, thanks.

Looks good!

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Mar 07, 2007 at 05:06:32PM -0800, Daniel Rall wrote:
> > I think that it would be useful, especially since there's nothing that
> > gets broken by an early cancellation.
> 
> +1.  Since we're already changed the FS-layer API, might as well add
> the parameters necessary for cancellation while we're at it.
> 

r23732, thanks.

Regards,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Daniel Rall <dl...@collab.net>.
On Wed, 07 Mar 2007, David Glasser wrote:

> On 3/7/07, Daniel Rall <dl...@collab.net> wrote:
> >> Oh, I see.  svn_repos_recover2 opens the repository, and in BDB only
> >> it takes the lock at this time.  It then calls back to svnadmin, which
> >> prints out "Repository lock acquired." even if the lock was not asked
> >> for (ie, FSFS).  It then gets to svn_fs_fs__recover... at which point
> >> the second svnadmin blocks.  Heh.
> >
> >You're saying the "lock acquired" message is appropriate even for FSFS?
> 
> My impression is that printing a "lock acquired" message is
> appropriate for FSFS, but the time it is printed is too early.  (The
> current svn_fs_fs__recover implementation wraps most of the
> calculation in a write lock on the repository, but this isn't called
> until *after* svn_svn_recover2 calls the callback which prints the
> message.)

Gotcha.  So moving the locking down into libsvn_fs_base would not fix
this problem -- we'd have to change the message printed by the
svnadmin callback to "Acquiring repository lock..." or some such, or
punch the invocation of the callback down into the FS layer.

> >> Does anyone know what the reason for special-casing BDB locking in
> >> libsvn_repos/repos.c was (instead of moving that behavior into
> >> libsvn_fs_base)?  Is it possible for us to fix this incongruity?
> >
> >Ah, in repos.c:lock_repos() (called by get_repos(), called by
> >svn_repos_recover2()).  The doc string added back in r16241 says:
> >
> >/* There is, at present, nothing within the direct responsibility
> >   of libsvn_repos which requires locking.  For historical compatibility
> >   reasons, the BDB libsvn_fs backend does not do its own locking, 
> >   expecting
> >   libsvn_repos to do the locking for it.  Here we take care of that
> >   backend-specific requirement.
> >   The kind of lock is controlled by EXCLUSIVE and NONBLOCKING.
> >   The lock is scoped to POOL.  */
> >
> >The BDB check was added in r16297, with the log message:
> >
> >------------------------------------------------------------------------
> >r16297 | maxb | 2005-09-27 03:44:44 -0700 (Tue, 27 Sep 2005) | 12 lines
> >
> >Teach libsvn_repos to use svn_fs_type() to know the kind of filesystem it
> >contains, and only apply BDB-specific locking requirements to BDB FSes.
> >This change *might* (untested) allow the hosting of FSFS repositories on 
> >Win9x.
> >
> >* subversion/libsvn_repos/repos.h (svn_repos_t): New field fs_type.
> >* subversion/libsvn_repos/repos.c (create_svn_repos_t): Adjust 
> >documentation
> >    to reference new fs_type struct member.
> >  (lock_repos): Do locking only for BDB filesystems.
> >  (svn_repos_create): Initialize new svn_repos_t fs_type field.
> >  (get_repos): Initialize new svn_repos_t fs_type field.
> >  (svn_repos_hotcopy): Initialize new svn_repos_t fs_type field.
> 
> OK, so it appears that some repos things require locking on BDB but
> not FSFS, and this is special-cased inside libsvn_repos instead of
> just letting libsvn_fs_base do the locking.  Is there a good reason to
> not move the locking into libsvn_fs_base?

Just guessing here, but perhaps there are some BDB operations which
don't require a repository-global lock?

It still seems like the FS layer should be managing the lock...

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Mar 07, 2007 at 11:31:21PM -0500, David Glasser wrote:
> >You're saying the "lock acquired" message is appropriate even for FSFS?
> 
> My impression is that printing a "lock acquired" message is
> appropriate for FSFS, but the time it is printed is too early.  (The
> current svn_fs_fs__recover implementation wraps most of the
> calculation in a write lock on the repository, but this isn't called
> until *after* svn_svn_recover2 calls the callback which prints the
> message.)
> 

This is because the repository layer is reponsible for doing the locking
for a filesystem-layer function.  Fixing that would be nice, since it's
only the filesystem that's in a position to know what locking is necessary.

Probably hard to do though.

Regards,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by David Glasser <gl...@mit.edu>.
On 3/7/07, Daniel Rall <dl...@collab.net> wrote:
> > Oh, I see.  svn_repos_recover2 opens the repository, and in BDB only
> > it takes the lock at this time.  It then calls back to svnadmin, which
> > prints out "Repository lock acquired." even if the lock was not asked
> > for (ie, FSFS).  It then gets to svn_fs_fs__recover... at which point
> > the second svnadmin blocks.  Heh.
>
> You're saying the "lock acquired" message is appropriate even for FSFS?

My impression is that printing a "lock acquired" message is
appropriate for FSFS, but the time it is printed is too early.  (The
current svn_fs_fs__recover implementation wraps most of the
calculation in a write lock on the repository, but this isn't called
until *after* svn_svn_recover2 calls the callback which prints the
message.)

> > Does anyone know what the reason for special-casing BDB locking in
> > libsvn_repos/repos.c was (instead of moving that behavior into
> > libsvn_fs_base)?  Is it possible for us to fix this incongruity?
>
> Ah, in repos.c:lock_repos() (called by get_repos(), called by
> svn_repos_recover2()).  The doc string added back in r16241 says:
>
> /* There is, at present, nothing within the direct responsibility
>    of libsvn_repos which requires locking.  For historical compatibility
>    reasons, the BDB libsvn_fs backend does not do its own locking, expecting
>    libsvn_repos to do the locking for it.  Here we take care of that
>    backend-specific requirement.
>    The kind of lock is controlled by EXCLUSIVE and NONBLOCKING.
>    The lock is scoped to POOL.  */
>
> The BDB check was added in r16297, with the log message:
>
> ------------------------------------------------------------------------
> r16297 | maxb | 2005-09-27 03:44:44 -0700 (Tue, 27 Sep 2005) | 12 lines
>
> Teach libsvn_repos to use svn_fs_type() to know the kind of filesystem it
> contains, and only apply BDB-specific locking requirements to BDB FSes.
> This change *might* (untested) allow the hosting of FSFS repositories on Win9x.
>
> * subversion/libsvn_repos/repos.h (svn_repos_t): New field fs_type.
> * subversion/libsvn_repos/repos.c (create_svn_repos_t): Adjust documentation
>     to reference new fs_type struct member.
>   (lock_repos): Do locking only for BDB filesystems.
>   (svn_repos_create): Initialize new svn_repos_t fs_type field.
>   (get_repos): Initialize new svn_repos_t fs_type field.
>   (svn_repos_hotcopy): Initialize new svn_repos_t fs_type field.

OK, so it appears that some repos things require locking on BDB but
not FSFS, and this is special-cased inside libsvn_repos instead of
just letting libsvn_fs_base do the locking.  Is there a good reason to
not move the locking into libsvn_fs_base?

--dave

-- 
David Glasser | glasser@mit.edu | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Daniel Rall <dl...@collab.net>.
On Wed, 07 Mar 2007, David Glasser wrote:

> On 3/7/07, Malcolm Rowe <ma...@farside.org.uk> wrote:
> >On Wed, Mar 07, 2007 at 01:43:46PM -0500, David Glasser wrote:
> >> I started to run this on my 114285-revision svk depot.  The first
> >> thing I noticed was that it seems to not want to be canceled; it seems
> >> to ignore SIGINTS, and I had to kill -9 it to cancel it.  I assume
> >> this is intentional?
> >
> >No, it's not intentional as such, but it's the same behaviour as BDB
> >recovery, I believe.  Unfortunately the current recovery API doesn't
> >provide for cancellation or feedback, though it should be pretty trivial
> >to extend the new API to include at least cancellation (for FSFS, anyway)
> >- does that sound like a good idea?
> 
> I think that it would be useful, especially since there's nothing that
> gets broken by an early cancellation.

+1.  Since we're already changed the FS-layer API, might as well add
the parameters necessary for cancellation while we're at it.

> >> More disturbingly, despite the message "Repository lock acquired.", I
> >> was able to run multiple copies of svnadmin recover on the repository
> >> at the same time.  When I noticed that, I freaked out and killed them
> >> both; I'll run it for real once I do an actual backup of my computer
> >> at home sometime soon.
> >>
> >
> >Heh.  FSFS recovery doesn't require an exclusive lock, so this is actually
> >okay.  That said, the user experience is certainly sub-optimal, but it's
> >not entirely clear how we can improve it (the locking is semi-hardcoded
> >in svnadmin and libsvn_repos).  Suggestions welcome.
> 
> Hmm.  But svn_fs_fs__recover does take out a write lock on the
> repository (not an exclusive lock, of course)...
> 
> Oh, I see.  svn_repos_recover2 opens the repository, and in BDB only
> it takes the lock at this time.  It then calls back to svnadmin, which
> prints out "Repository lock acquired." even if the lock was not asked
> for (ie, FSFS).  It then gets to svn_fs_fs__recover... at which point
> the second svnadmin blocks.  Heh.

You're saying the "lock acquired" message is appropriate even for FSFS?

> Does anyone know what the reason for special-casing BDB locking in
> libsvn_repos/repos.c was (instead of moving that behavior into
> libsvn_fs_base)?  Is it possible for us to fix this incongruity?

Ah, in repos.c:lock_repos() (called by get_repos(), called by
svn_repos_recover2()).  The doc string added back in r16241 says:

/* There is, at present, nothing within the direct responsibility
   of libsvn_repos which requires locking.  For historical compatibility
   reasons, the BDB libsvn_fs backend does not do its own locking, expecting
   libsvn_repos to do the locking for it.  Here we take care of that
   backend-specific requirement. 
   The kind of lock is controlled by EXCLUSIVE and NONBLOCKING.
   The lock is scoped to POOL.  */

The BDB check was added in r16297, with the log message:

------------------------------------------------------------------------
r16297 | maxb | 2005-09-27 03:44:44 -0700 (Tue, 27 Sep 2005) | 12 lines

Teach libsvn_repos to use svn_fs_type() to know the kind of filesystem it
contains, and only apply BDB-specific locking requirements to BDB FSes.
This change *might* (untested) allow the hosting of FSFS repositories on Win9x.

* subversion/libsvn_repos/repos.h (svn_repos_t): New field fs_type.
* subversion/libsvn_repos/repos.c (create_svn_repos_t): Adjust documentation
    to reference new fs_type struct member.
  (lock_repos): Do locking only for BDB filesystems.
  (svn_repos_create): Initialize new svn_repos_t fs_type field.
  (get_repos): Initialize new svn_repos_t fs_type field.
  (svn_repos_hotcopy): Initialize new svn_repos_t fs_type field.

------------------------------------------------------------------------


Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by David Glasser <gl...@mit.edu>.
On 3/7/07, Malcolm Rowe <ma...@farside.org.uk> wrote:
> On Wed, Mar 07, 2007 at 01:43:46PM -0500, David Glasser wrote:
> > I started to run this on my 114285-revision svk depot.  The first
> > thing I noticed was that it seems to not want to be canceled; it seems
> > to ignore SIGINTS, and I had to kill -9 it to cancel it.  I assume
> > this is intentional?
>
> No, it's not intentional as such, but it's the same behaviour as BDB
> recovery, I believe.  Unfortunately the current recovery API doesn't
> provide for cancellation or feedback, though it should be pretty trivial
> to extend the new API to include at least cancellation (for FSFS, anyway)
> - does that sound like a good idea?

I think that it would be useful, especially since there's nothing that
gets broken by an early cancellation.

> > More disturbingly, despite the message "Repository lock acquired.", I
> > was able to run multiple copies of svnadmin recover on the repository
> > at the same time.  When I noticed that, I freaked out and killed them
> > both; I'll run it for real once I do an actual backup of my computer
> > at home sometime soon.
> >
>
> Heh.  FSFS recovery doesn't require an exclusive lock, so this is actually
> okay.  That said, the user experience is certainly sub-optimal, but it's
> not entirely clear how we can improve it (the locking is semi-hardcoded
> in svnadmin and libsvn_repos).  Suggestions welcome.

Hmm.  But svn_fs_fs__recover does take out a write lock on the
repository (not an exclusive lock, of course)...

Oh, I see.  svn_repos_recover2 opens the repository, and in BDB only
it takes the lock at this time.  It then calls back to svnadmin, which
prints out "Repository lock acquired." even if the lock was not asked
for (ie, FSFS).  It then gets to svn_fs_fs__recover... at which point
the second svnadmin blocks.  Heh.

Does anyone know what the reason for special-casing BDB locking in
libsvn_repos/repos.c was (instead of moving that behavior into
libsvn_fs_base)?  Is it possible for us to fix this incongruity?

--dave

-- 
David Glasser | glasser@mit.edu | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Mar 07, 2007 at 01:43:46PM -0500, David Glasser wrote:
> I started to run this on my 114285-revision svk depot.  The first
> thing I noticed was that it seems to not want to be canceled; it seems
> to ignore SIGINTS, and I had to kill -9 it to cancel it.  I assume
> this is intentional?
> 

No, it's not intentional as such, but it's the same behaviour as BDB
recovery, I believe.  Unfortunately the current recovery API doesn't
provide for cancellation or feedback, though it should be pretty trivial
to extend the new API to include at least cancellation (for FSFS, anyway)
- does that sound like a good idea?

> More disturbingly, despite the message "Repository lock acquired.", I
> was able to run multiple copies of svnadmin recover on the repository
> at the same time.  When I noticed that, I freaked out and killed them
> both; I'll run it for real once I do an actual backup of my computer
> at home sometime soon.
> 

Heh.  FSFS recovery doesn't require an exclusive lock, so this is actually
okay.  That said, the user experience is certainly sub-optimal, but it's
not entirely clear how we can improve it (the locking is semi-hardcoded
in svnadmin and libsvn_repos).  Suggestions welcome.

Regards,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by David Glasser <gl...@mit.edu>.
On 3/7/07, Malcolm Rowe <ma...@farside.org.uk> wrote:
> 1. Make a backup of your repository.  If you _really_ trust me, just make
>    a backup of db/current.
> 2. Optionally remove db/current to prove that we're recreating it properly.
>    (This will obviously make the repository unusable to any clients).
> 3. Run "svnadmin recover repo/".
> 4. Wait, as svnadmin reads all of the revision files.  Any writers are
>    blocked for the duration of recovery.
>    (It'd be nice to get some timing data here as well.  My tests had too
>    much variability, but it looked like a couple of minutes to recover
>    against [a mirror of] the main Subversion repository).
> 5. Check that the new db/current is identical to the old db/current.
>    Scream if not.

I started to run this on my 114285-revision svk depot.  The first
thing I noticed was that it seems to not want to be canceled; it seems
to ignore SIGINTS, and I had to kill -9 it to cancel it.  I assume
this is intentional?

More disturbingly, despite the message "Repository lock acquired.", I
was able to run multiple copies of svnadmin recover on the repository
at the same time.  When I noticed that, I freaked out and killed them
both; I'll run it for real once I do an actual backup of my computer
at home sometime soon.

--dave

-- 
David Glasser | glasser@mit.edu | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by John Szakmeister <jo...@szakmeister.net>.
Malcolm Rowe wrote:
> On Tue, Mar 13, 2007 at 03:17:41PM -0400, Greg Hudson wrote:
>> On Tue, 2007-03-13 at 11:51 +0000, Malcolm Rowe wrote:
>>> Justin was kind enough to lend me the ASF's repository - all 515,882
>>> revisions of it :-)  It took me just over an hour to run recovery on
>>> this repository on my extremely underpowered fileserver, but it
>>> completed without problems.
>> It occurs to me that if we stored the next-node-id and next-copy-id in
>> each revision file in a well-defined place, "recovery" would be a much
>> more trivial operation; just find the highest revision and pluck the
>> information from it.
>>
> 
> That sounds like a good idea.  Can you think of an easy way to do it in
> a backward-compatible manner?  (Perhaps we could stash them at the tail
> of the rev file - would that work?).

I think just after the changed path information in the rev file would do 
the trick.

> 
> Oh, actually, we could just store them in the revprops file - that might
> make more sense.

I think of revprops as storing user accessible data, not for storing 
direct information about revisions.  I much prefer to keep it in the rev 
file.

-John

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Tue, Mar 13, 2007 at 03:17:41PM -0400, Greg Hudson wrote:
> On Tue, 2007-03-13 at 11:51 +0000, Malcolm Rowe wrote:
> > Justin was kind enough to lend me the ASF's repository - all 515,882
> > revisions of it :-)  It took me just over an hour to run recovery on
> > this repository on my extremely underpowered fileserver, but it
> > completed without problems.
> 
> It occurs to me that if we stored the next-node-id and next-copy-id in
> each revision file in a well-defined place, "recovery" would be a much
> more trivial operation; just find the highest revision and pluck the
> information from it.
> 

That sounds like a good idea.  Can you think of an easy way to do it in
a backward-compatible manner?  (Perhaps we could stash them at the tail
of the rev file - would that work?).

Oh, actually, we could just store them in the revprops file - that might
make more sense.

Regards,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Greg Hudson <gh...@MIT.EDU>.
On Tue, 2007-03-13 at 11:51 +0000, Malcolm Rowe wrote:
> Justin was kind enough to lend me the ASF's repository - all 515,882
> revisions of it :-)  It took me just over an hour to run recovery on
> this repository on my extremely underpowered fileserver, but it
> completed without problems.

It occurs to me that if we stored the next-node-id and next-copy-id in
each revision file in a well-defined place, "recovery" would be a much
more trivial operation; just find the highest revision and pluck the
information from it.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Mar 07, 2007 at 11:58:30AM -0800, Justin Erenkrantz wrote:
> On 3/7/07, Malcolm Rowe <ma...@farside.org.uk> wrote:
> >I'd really appreciate some level of confidence testing on a few more
> >repositories, particularly those that are particularly large or
> >particularly weird (anything converted from another vcs, perhaps).
> 
> I can facilitate access to Apache's fsfs repository for testing.  It's
> currently 20GB.  Contact me off-list and I can rsync over it to you
> somewhere if you want it.  -- justin
> 

Justin was kind enough to lend me the ASF's repository - all 515,882
revisions of it :-)  It took me just over an hour to run recovery on
this repository on my extremely underpowered fileserver, but it
completed without problems.

Regards,
Malcolm

Re: Testers wanted for FSFS svnadmin recover feature (r23595)

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 3/7/07, Malcolm Rowe <ma...@farside.org.uk> wrote:
> I'd really appreciate some level of confidence testing on a few more
> repositories, particularly those that are particularly large or
> particularly weird (anything converted from another vcs, perhaps).

I can facilitate access to Apache's fsfs repository for testing.  It's
currently 20GB.  Contact me off-list and I can rsync over it to you
somewhere if you want it.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org