You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ben Collins-Sussman <su...@collab.net> on 2001/08/01 21:55:39 UTC
M3 ahead...
Here's a little status mail about exactly how we're closing in on M3,
which is currently slated for 2 weeks from today. Overall things look
good. Our hope is that after August 15th, we'll be running a
dedicated Apache/SVN server for this project. That means all
developers will need to use SVN to contribute. :-)
* World of DAV (gstein)
Greg is going to bring the DAV layer back up to speed. This
includes:
* using the new authentication system to get user/password
* passing this info to Apache, and then into the new write-hooks
* making properties work -- without leaving RA-specific ones
in the repository.
(This is the semi-large project. It may require some
large changes to mod_dav itself, not just mod_dav_svn.
* modifying python tests to run over RA_DAV (sussman will do.)
* Hooks: (cmpilato, kfogel -- working, nearly done)
The write-hooks are essentially working now, and have an
excellent 'svnlook' tool for assisting hook scripts. We
might attempt to get "reading sentinels" working if time
permits.
* Authentication: (sussman -- working, nearly done)
* ra_local just gets a username.
* ra_dav will get username/password, store them in working
copy, and allow Apache to authenticate.
Usernames are stored automatically as properties on transaction
and revision roots, along with log messages and datestamps.
* Authorization: finished -- simple for now. (ACL's after 1.0.)
* for ra_local, the Unix filesystem is all the security there
can be. The user running the svn client app either does or
doesn't have permission to read/write the Berkeley DB files.
* for ra_dav, Apache does the authorization by looking at a
list of committers in a .conf file.
* more complex authorization (such as per-directory rights)
can be done using hook scripts.
* Mime-types: (cmpilato -- working)
Working on heuristic to determine if an added file is binary
or not. This will (later) affect keyword substitution and
line endings. It will also affect whether we can safely run
'diff' and 'patch' under the hood when receiving an update to
the file.
* DB_RUN_RECOVERY: (unassigned)
Still need to write cross-process locking function so that the
repository can repair itself when this error is returned. We
believe APR now has the means to help us with this.
* Filesystem deltification optimization (unassigned)
Karl is worried about speed of our deltification right now; we
may optimize it, if time permits.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: DB_RUN_RECOVERY (was: Re: M3 ahead...)
Posted by kf...@collab.net.
It may help to point out that the word "transaction" is being used for
two unrelated things, here.
Berkeley transactions are what you think they are -- they mean the
same thing in Berkeley DB-speak that they usually mean in database
contexts.
Subversion transactions (or "txns") are mutable trees in which a
client makes changes, either to commit or to update against. In the
case of a commit, a txn is started as a copy of the revision tree that
is the base of the commit, then the client's modifications are made,
then any changes between base and head are merged into the txn (don't
worry, this is not a server-side merge in the bad sense), and then the
txn tree becomes the next revision tree. In the course of doing all
that, *many* Berkeley transactions will be used.
Hope that clarifies things a bit.
-K
Greg Stein <gs...@lyra.org> writes:
> On Sat, Aug 04, 2001 at 01:30:58AM -0500, subversion-dev@thewrittenword.com wrote:
> > On Fri, Aug 03, 2001 at 11:36:04PM -0500, Jim Blandy wrote:
> > > subversion-dev@thewrittenword.com writes:
> > > > Unless you're doing locking, I think the chances for encountering
> > > > DB_RUN_RECOVERY are minimal. What does subversion do if a
> > > > transaction has begun (txn_begin) and a signal interrupts the server
> > > > before txn_commit() or txn_abort()?
> > >
> > > Whatever resources were held by that transaction hang around, locked,
> > > indefinitely, until they're cleaned up. Since Berkeley DB has no
> > > reliable way to know when a database client has exited, it can't clean
> > > up after it. The parent process (Apache) needs to notice when
> > > children die ungracefully, and clean up.
>
> A single child dying does not imply that a transaction needs to be cleaned
> up. A given transaction could be performed across multiple children over a
> period of time. Moreover, if the child dies (and the connection aborts),
> then the client "should" potentially retry the operation, meaning that you
> want that transaction left alone.
>
> > And how does Apache find out about the transaction ID that is needed
> > by txn_abort()? Does mod_dav_svn have this info?
>
> Apache will not worry about "garbage" transactions. svnadmin will be used
> periodically to find "old" transactions and delete them.
> svn_fs_list_transactions() can be used by svnadmin to get the transaction
> information.
>
> There is a bit more to clearing old transactions, but the main point is that
> it will not be handled by Apache.
>
>
> Now... back to the original question: DB_RUN_RECOVERY. If that error ever
> appears during our operation, then we will simply bail out, as usual. That
> particular sequence of operations (e.g. the HTTP request) will terminate,
> needing to be restarted by the client (at its discretion).
>
> Instead, we handle DB_RUN_RECOVERY at the point where the DB is opened. If
> we see that error, then we take out a lock, run the recovery process, and
> release the lock. The lock is used to shut out users of the DB while the
> recovery is occurring. Note that this implies that we always go for a
> "reader" lock every time we open the DB. Multiple readers, single writer.
>
> This is all presumed on the idea that we will see DB_RUN_RECOVERY when we
> attempt to open the DB. If the error only occurs on *operations*, then we
> would need to place the writer-lock-then-cleanup operation on the error exit
> codepaths.
>
> Cheers,
> -g
>
> --
> Greg Stein, http://www.lyra.org/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: DB_RUN_RECOVERY (was: Re: M3 ahead...)
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Greg Stein <gs...@lyra.org> writes:
> On Sat, Aug 04, 2001 at 01:30:58AM -0500, subversion-dev@thewrittenword.com wrote:
> > On Fri, Aug 03, 2001 at 11:36:04PM -0500, Jim Blandy wrote:
> > > subversion-dev@thewrittenword.com writes:
> > > > Unless you're doing locking, I think the chances for encountering
> > > > DB_RUN_RECOVERY are minimal. What does subversion do if a
> > > > transaction has begun (txn_begin) and a signal interrupts the server
> > > > before txn_commit() or txn_abort()?
> > >
> > > Whatever resources were held by that transaction hang around, locked,
> > > indefinitely, until they're cleaned up. Since Berkeley DB has no
> > > reliable way to know when a database client has exited, it can't clean
> > > up after it. The parent process (Apache) needs to notice when
> > > children die ungracefully, and clean up.
>
> A single child dying does not imply that a transaction needs to be cleaned
> up. A given transaction could be performed across multiple children over a
> period of time. Moreover, if the child dies (and the connection aborts),
> then the client "should" potentially retry the operation, meaning that you
> want that transaction left alone.
We were talking about Berkeley DB transactions, not Subversion
transactions.
Berkeley DB transactions can (I think) be used by more than one
process, but that would require them to be a visible piece of the FS
interface, which is bogus. Berkeley DB should be an implementation
detail of the FS --- even if we have to make weird recovery procedures
part of the published interface. :(
> Now... back to the original question: DB_RUN_RECOVERY. If that error ever
> appears during our operation, then we will simply bail out, as usual. That
> particular sequence of operations (e.g. the HTTP request) will terminate,
> needing to be restarted by the client (at its discretion).
>
> Instead, we handle DB_RUN_RECOVERY at the point where the DB is opened. If
> we see that error, then we take out a lock, run the recovery process, and
> release the lock. The lock is used to shut out users of the DB while the
> recovery is occurring. Note that this implies that we always go for a
> "reader" lock every time we open the DB. Multiple readers, single writer.
>
> This is all presumed on the idea that we will see DB_RUN_RECOVERY when we
> attempt to open the DB. If the error only occurs on *operations*, then we
> would need to place the writer-lock-then-cleanup operation on the error exit
> codepaths.
I think we'll see it when we open the DB. We'll play it by ear.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
DB_RUN_RECOVERY (was: Re: M3 ahead...)
Posted by Greg Stein <gs...@lyra.org>.
On Sat, Aug 04, 2001 at 01:30:58AM -0500, subversion-dev@thewrittenword.com wrote:
> On Fri, Aug 03, 2001 at 11:36:04PM -0500, Jim Blandy wrote:
> > subversion-dev@thewrittenword.com writes:
> > > Unless you're doing locking, I think the chances for encountering
> > > DB_RUN_RECOVERY are minimal. What does subversion do if a
> > > transaction has begun (txn_begin) and a signal interrupts the server
> > > before txn_commit() or txn_abort()?
> >
> > Whatever resources were held by that transaction hang around, locked,
> > indefinitely, until they're cleaned up. Since Berkeley DB has no
> > reliable way to know when a database client has exited, it can't clean
> > up after it. The parent process (Apache) needs to notice when
> > children die ungracefully, and clean up.
A single child dying does not imply that a transaction needs to be cleaned
up. A given transaction could be performed across multiple children over a
period of time. Moreover, if the child dies (and the connection aborts),
then the client "should" potentially retry the operation, meaning that you
want that transaction left alone.
> And how does Apache find out about the transaction ID that is needed
> by txn_abort()? Does mod_dav_svn have this info?
Apache will not worry about "garbage" transactions. svnadmin will be used
periodically to find "old" transactions and delete them.
svn_fs_list_transactions() can be used by svnadmin to get the transaction
information.
There is a bit more to clearing old transactions, but the main point is that
it will not be handled by Apache.
Now... back to the original question: DB_RUN_RECOVERY. If that error ever
appears during our operation, then we will simply bail out, as usual. That
particular sequence of operations (e.g. the HTTP request) will terminate,
needing to be restarted by the client (at its discretion).
Instead, we handle DB_RUN_RECOVERY at the point where the DB is opened. If
we see that error, then we take out a lock, run the recovery process, and
release the lock. The lock is used to shut out users of the DB while the
recovery is occurring. Note that this implies that we always go for a
"reader" lock every time we open the DB. Multiple readers, single writer.
This is all presumed on the idea that we will see DB_RUN_RECOVERY when we
attempt to open the DB. If the error only occurs on *operations*, then we
would need to place the writer-lock-then-cleanup operation on the error exit
codepaths.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by Greg Stein <gs...@lyra.org>.
I've been reading "txn_abort" as SVN's abort function. Sorry.
Otherwise, I know the difference... just not the function name :-)
Cheers,
-g
On Tue, Aug 07, 2001 at 06:44:14PM -0500, Jim Blandy wrote:
>
> Greg Stein <gs...@lyra.org> writes:
> > On Tue, Aug 07, 2001 at 11:08:47AM -0500, Jim Blandy wrote:
> > >
> > > subversion-dev@thewrittenword.com writes:
> > > > And how does Apache find out about the transaction ID that is needed
> > > > by txn_abort()? Does mod_dav_svn have this info?
> > >
> > > Ah, a careful reader! I don't know. :(
> >
> > As I mentioned before, Apache will never call txn_abort unless the *client*
> > sends a request to do that (e.g. DELETE the activity).
>
> I can't be sure, but it really sounds like you are confusing Berkeley
> DB transactions with Subversion transactions. txn_abort aborts a
> Berkeley DB transaction; it has nothing to do with deleting a DAV
> activity.
>
> If I understand Berkeley DB correctly, when a process working on a
> DB exits unexpectedly, it will leave locks in the shared memory area.
> These locks will need to be cleared out.
>
> Now that I think about it, it should be safe for
> svn_fs_berkeley_recover to simply abort all pending Berkeley DB
> transactions, since (at least in theory) it knows it is called when
> nobody else is writing to the database. So it's not necessary to
> recover the Berkeley DB transaction ID's.
>
> > There is no way for Apache to know that an SVN transaction needs to be
> > aborted. The death of a child process/thread is an insufficient trigger.
>
> For an SVN transaction, this is correct. For a Berkeley DB
> transaction (which is what txn_abort works with), however, if a child
> dies, then its Berkeley DB locks can certainly be disposed of --- and
> must be, I think.
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Greg Stein <gs...@lyra.org> writes:
> On Tue, Aug 07, 2001 at 11:08:47AM -0500, Jim Blandy wrote:
> >
> > subversion-dev@thewrittenword.com writes:
> > > And how does Apache find out about the transaction ID that is needed
> > > by txn_abort()? Does mod_dav_svn have this info?
> >
> > Ah, a careful reader! I don't know. :(
>
> As I mentioned before, Apache will never call txn_abort unless the *client*
> sends a request to do that (e.g. DELETE the activity).
I can't be sure, but it really sounds like you are confusing Berkeley
DB transactions with Subversion transactions. txn_abort aborts a
Berkeley DB transaction; it has nothing to do with deleting a DAV
activity.
If I understand Berkeley DB correctly, when a process working on a
DB exits unexpectedly, it will leave locks in the shared memory area.
These locks will need to be cleared out.
Now that I think about it, it should be safe for
svn_fs_berkeley_recover to simply abort all pending Berkeley DB
transactions, since (at least in theory) it knows it is called when
nobody else is writing to the database. So it's not necessary to
recover the Berkeley DB transaction ID's.
> There is no way for Apache to know that an SVN transaction needs to be
> aborted. The death of a child process/thread is an insufficient trigger.
For an SVN transaction, this is correct. For a Berkeley DB
transaction (which is what txn_abort works with), however, if a child
dies, then its Berkeley DB locks can certainly be disposed of --- and
must be, I think.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Aug 07, 2001 at 11:08:47AM -0500, Jim Blandy wrote:
>
> subversion-dev@thewrittenword.com writes:
> > And how does Apache find out about the transaction ID that is needed
> > by txn_abort()? Does mod_dav_svn have this info?
>
> Ah, a careful reader! I don't know. :(
As I mentioned before, Apache will never call txn_abort unless the *client*
sends a request to do that (e.g. DELETE the activity).
There is no way for Apache to know that an SVN transaction needs to be
aborted. The death of a child process/thread is an insufficient trigger.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
subversion-dev@thewrittenword.com writes:
> And how does Apache find out about the transaction ID that is needed
> by txn_abort()? Does mod_dav_svn have this info?
Ah, a careful reader! I don't know. :(
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by su...@thewrittenword.com.
On Fri, Aug 03, 2001 at 11:36:04PM -0500, Jim Blandy wrote:
> subversion-dev@thewrittenword.com writes:
> > Unless you're doing locking, I think the chances for encountering
> > DB_RUN_RECOVERY are minimal. What does subversion do if a
> > transaction has begun (txn_begin) and a signal interrupts the server
> > before txn_commit() or txn_abort()?
>
> Whatever resources were held by that transaction hang around, locked,
> indefinitely, until they're cleaned up. Since Berkeley DB has no
> reliable way to know when a database client has exited, it can't clean
> up after it. The parent process (Apache) needs to notice when
> children die ungracefully, and clean up.
And how does Apache find out about the transaction ID that is needed
by txn_abort()? Does mod_dav_svn have this info?
--
albert chin (china@thewrittenword.com)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
subversion-dev@thewrittenword.com writes:
> Taking a brief look at the subversion code, you use Berkeley DB
> transactions though I don't see any evidence of Berkeley DB locks in
> libsvn_fs/.
Transactions use locks implicitly.
> Unless you're doing locking, I think the chances for encountering
> DB_RUN_RECOVERY are minimal. What does subversion do if a
> transaction has begun (txn_begin) and a signal interrupts the server
> before txn_commit() or txn_abort()?
Whatever resources were held by that transaction hang around, locked,
indefinitely, until they're cleaned up. Since Berkeley DB has no
reliable way to know when a database client has exited, it can't clean
up after it. The parent process (Apache) needs to notice when
children die ungracefully, and clean up.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: M3 ahead...
Posted by su...@thewrittenword.com.
On Wed, Aug 01, 2001 at 04:55:39PM -0500, Ben Collins-Sussman wrote:
> Here's a little status mail about exactly how we're closing in on M3,
> which is currently slated for 2 weeks from today. Overall things look
> good. Our hope is that after August 15th, we'll be running a
> dedicated Apache/SVN server for this project. That means all
> developers will need to use SVN to contribute. :-)
>
> * DB_RUN_RECOVERY: (unassigned)
>
> Still need to write cross-process locking function so that the
> repository can repair itself when this error is returned. We
> believe APR now has the means to help us with this.
Are there any plans for how you plan to solve this?
Taking a brief look at the subversion code, you use Berkeley DB
transactions though I don't see any evidence of Berkeley DB locks in
libsvn_fs/. Unless you're doing locking, I think the chances for
encountering DB_RUN_RECOVERY are minimal. What does subversion do if a
transaction has begun (txn_begin) and a signal interrupts the server
before txn_commit() or txn_abort()?
When we cannot get a lock on an object, we try a few times then error
out with a message to the user that they must run a program that opens
the database in DB_RECOVER mode. The database code ensures that the
program running with DB_RECOVER is the *only* process which has the
database open (we use fcntl to synchronize access to the database
for all programs that perform RW operations).
--
albert chin (china@thewrittenword.com)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org