backend databases (was: Re: The /. article)

[Greg Stein <gs...@lyra.org>]

On Tue, Feb 05, 2002 at 05:57:17PM -0800, Sean Russell wrote:
> On Tuesday 05 February 2002 17:17, Karl wrote:
> > This is technically true, but that's a pretty big change.  I think
> > it's quite fair of Tom to simply say "Subversion uses Berkeley DB for
> > its repository" and leave it at that.  Any system can be changed,
> > including Arch; the SQL backend claim amounts to saying "Subversion
> > has clean internal interfaces", which for all we know, Arch does too.
> 
> I suppose I was thinking (a) subversion was specifically /designed/ to have 
> pluggable backends, and (b) there's a strong faction of SQLers in the group.  
> Adding backends isn't a priority, but I got the impression that it was pretty 
> high on the list for post-1.0.

The Subversion FS is implemented to use Berkeley DB right now, and that is
pretty well built in. The overall design and the APIs do not require it,
however. But to change that implementation... it's a bit of work.

(we made an initial design decision to code for BDB directly, rather than to
try and craft an intermediate API; there were a lot of benefits to that
initial approach, which I won't go into here)

All that said: Subversion 2.0 will have SQL backend support, if at all
possible. I don't think anybody disagrees with the notion, but nobody is
immediately jumping at the bit to do it immediately :-)

(my other main focus for 2.0 will be increased DAV interop)

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: backend databases

[Greg Stein <gs...@lyra.org>]

On Wed, Feb 06, 2002 at 09:21:36AM -0500, Mark C. Chu-Carroll wrote:
> On Tue, 2002-02-05 at 21:07, Greg Stein wrote:
>...
> > All that said: Subversion 2.0 will have SQL backend support, if at all
> > possible. I don't think anybody disagrees with the notion, but nobody is
> > immediately jumping at the bit to do it immediately :-)
> 
> I'd also encourage people to wait a bit on doing a SQL backend. For
> Stellation (the project formerly known as Synedra or Coven), we built
> our backend using SQL (with PostgreSQL as our primary target.... Anyone
> who is considering using MySQL for a reliable storage backend is simply
> nuts. Without solid transactionality, which MySQL doesn't really support
> yet, it's just not trustworthy enough...)

MySQL has data *integrity*. Its operations on the underlying data store (the
filesystem) is transacted. What it lacks are SQL-level transactions.
However, Subversion does not require SQL-level transactions. Each SVN-level
transaction is used by a single thread/client, and all revision trees are
immutable. Thus, the only point where a problem occurs is promoting an
SVN-level transaction tree into a revision tree. And that is simply an
INSERT statement into a table mapping revision numbers to root nodes.
Individual SQL statements are always transacted, so we'd be quite fine
without a SQL-level transaction capability.

[ more specifically, we'd do something like:

    INSERT into revisions values (expected-rev, root-node-id)

  if somebody beat us to expected-rev, then the statement errors due to a
  unique key constraint, so we re-merge and try again with the next revision
  number.
 ]

> There are huge advantages to a
> SQL backend vs BDB, but it's far from trivial to get it right. Our code

We've got some good database people on this list (Bill Tutt comes to mind).
I'm actually not too worried about our database design.

> will be going open shortly (if the damned lawyers would just get out of
> our way), and  we've got a well-tested SQL storage backend that would
> be, at the very least, highly instructional for anyone building a SQL
> backend for SubVersion. (There's a good chance that design could be
> lifted; since we're in Java, code can be adapted, but not directly
> copied.)

But this would definitely be helpful. Thanks :-)

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: backend databases (was: Re: The /. article)

["Mark C. Chu-Carroll" <mc...@watson.ibm.com>]

On Tue, 2002-02-05 at 21:07, Greg Stein wrote:
> On Tue, Feb 05, 2002 at 05:57:17PM -0800, Sean Russell wrote:
> > On Tuesday 05 February 2002 17:17, Karl wrote:
> > > This is technically true, but that's a pretty big change.  I think
> > > it's quite fair of Tom to simply say "Subversion uses Berkeley DB for
> > > its repository" and leave it at that.  Any system can be changed,
> > > including Arch; the SQL backend claim amounts to saying "Subversion
> > > has clean internal interfaces", which for all we know, Arch does too.
> > 
> > I suppose I was thinking (a) subversion was specifically /designed/ to have 
> > pluggable backends, and (b) there's a strong faction of SQLers in the group.  
> > Adding backends isn't a priority, but I got the impression that it was pretty 
> > high on the list for post-1.0.
> 
> The Subversion FS is implemented to use Berkeley DB right now, and that is
> pretty well built in. The overall design and the APIs do not require it,
> however. But to change that implementation... it's a bit of work.
> 
> (we made an initial design decision to code for BDB directly, rather than to
> try and craft an intermediate API; there were a lot of benefits to that
> initial approach, which I won't go into here)
> 
> All that said: Subversion 2.0 will have SQL backend support, if at all
> possible. I don't think anybody disagrees with the notion, but nobody is
> immediately jumping at the bit to do it immediately :-)

I'd also encourage people to wait a bit on doing a SQL backend. For
Stellation (the project formerly known as Synedra or Coven), we built
our backend using SQL (with PostgreSQL as our primary target.... Anyone
who is considering using MySQL for a reliable storage backend is simply
nuts. Without solid transactionality, which MySQL doesn't really support
yet, it's just not trustworthy enough...) There are huge advantages to a
SQL backend vs BDB, but it's far from trivial to get it right. Our code
will be going open shortly (if the damned lawyers would just get out of
our way), and  we've got a well-tested SQL storage backend that would
be, at the very least, highly instructional for anyone building a SQL
backend for SubVersion. (There's a good chance that design could be
lifted; since we're in Java, code can be adapted, but not directly
copied.)

	-Mark

-- 
Mark Craig Chu-Carroll,  IBM T.J. Watson Research Center 
<mc...@watson.ibm.com>
*** The (recently renamed) Stellation project:
***		http://domino.research.ibm.com/synedra/synedra.nsf
*** GPG Public key available at keyserver.net