You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Greg Stein <gs...@lyra.org> on 2000/12/09 13:44:35 UTC
couple issues
A couple high-level questions:
1) there is a need for some server-side tools:
a) create a repository
b) optimize/garbage-collect an existing repository
c) print/purge pending/discarded transactions
d) etc
I'd like to create subversion/server/ to hold these tools. I'm not sure
if we would have one, or whether we'd have a subcommand like the client.
But the main point is a directory to hold the tool(s).
2) APRUTIL is being put together as a sister project to APR. I'd like to
propose using it for the following features: MD5/SHA1 hashing (MD5 is
currently in APR itself, but it will probably move), some XML utility
functions (and I'd like to shift some of ours over to APRUTIL), the DBM
facilities, and possibly the URI stuff.
The base64 encoding/decoding stuff is there, but it wasn't applicable to
our needs. I'd like to look into resolving some of that (upgrading the
base64 support in APRUTIL and/or shifting some of our support there).
Strictly speaking, our dependency on it would be pretty light.
mod_dav_svn can just use APRUTIL via Apache itself. However, the FS and
the server tool(s) will need MD5 hashing and DBM support (the latter
because mod_dav_svn will use DBM for some stuff, and the tool will need
to deal with those).
[ if Berkeley DB utils were exported from FS, then I'd probably use that;
even better is to add DB3 support to APRUTIL's DBM support. ]
I'm a bit light on rationale for this, but figured it would be good to
bring up now.
Input/thoughts appreciated...
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: tools location? (was: Re: couple issues)
Posted by Karl Fogel <kf...@galois.collab.net>.
Greg Stein <gs...@lyra.org> writes:
> Sure, but my divining rod is indicating somewhere under subversion/ rather
> than tools/ :-)
>
> [ okay, that was my last try :-) ... name a place for the admin tool, Karl,
> and I'll put it there. ]
Personally, I'm happy with whatever you choose, Greg. :-)
(Seriously.)
-K
Re: tools location? (was: Re: couple issues)
Posted by Greg Stein <gs...@lyra.org>.
On Mon, Dec 11, 2000 at 01:41:59PM -0600, Karl Fogel wrote:
> Greg Stein <gs...@lyra.org> writes:
> > I didn't suggest the tools directory because I seem to recall that as being
> > the "supportive" or non-core pieces of SVN. i.e. we don't put any source
> > code in there.
>
> The distinction seems more political than technical.
Yup.
> Non-core pieces
> have source code too, after all -- you even put some Python code in
> there yourself. :-)
Sorry, I meant "source for the 'core' pieces of SVN". The server tool that I
mentioned *is* a core piece -- you can't even get started without it. (it is
used to create a DB3 environment for the repository)
So I saw that tool as a core part of the server. But it isn't libsvn_fs.
Therefore, the suggestion for subversion/server/
Maybe subversion/admin/ ?
> I'll bet the distinction between core and non-core is something we'll
> have a hard time predicting correctly. Let's just put things that are
> neither the client nor the server into tools/, and let people figure
> out which ones they need.
Sure, but my divining rod is indicating somewhere under subversion/ rather
than tools/ :-)
[ okay, that was my last try :-) ... name a place for the admin tool, Karl,
and I'll put it there. ]
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: tools location? (was: Re: couple issues)
Posted by Karl Fogel <kf...@galois.collab.net>.
Greg Stein <gs...@lyra.org> writes:
> I didn't suggest the tools directory because I seem to recall that as being
> the "supportive" or non-core pieces of SVN. i.e. we don't put any source
> code in there.
The distinction seems more political than technical. Non-core pieces
have source code too, after all -- you even put some Python code in
there yourself. :-)
I'll bet the distinction between core and non-core is something we'll
have a hard time predicting correctly. Let's just put things that are
neither the client nor the server into tools/, and let people figure
out which ones they need.
> GregH built one that operates in a streamy fashion. The APR version does
> not (yet).
Then I like GregH's (yet). :-)
-K
tools location? (was: Re: couple issues)
Posted by Greg Stein <gs...@lyra.org>.
On Sun, Dec 10, 2000 at 09:26:04PM -0600, Karl Fogel wrote:
> Branko =?ISO-8859-2?Q?=C8ibej?= <br...@xbc.nu> writes:
> > We already have a tools directory, off of the top level. I'd rather put
> > that in tools/server than cerate a new directory for such things.
>
> +1
I didn't suggest the tools directory because I seem to recall that as being
the "supportive" or non-core pieces of SVN. i.e. we don't put any source
code in there.
> > We have too many base64 implementations now. There's one in Subversion,
> > one in neon and one in APRUTIL ... I'd go for shifting whatever is
> > needed to support SVN into APRUTIL and throw our implemetation away.
Agreed.
> > (No, GregH, I'm not implying you did a bad job of it, just that we don't
> > want to have too many copies of that stuff. :-) )
>
> +1, assuming we don't lose any functionality.
GregH built one that operates in a streamy fashion. The APR version does
not (yet).
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: couple issues
Posted by Branko Čibej <br...@xbc.nu>.
Jim Blandy wrote:
> It's nice to have someone else thinking about it.
Yeah, we've all Been Bitten By Broken Backups Before, haven't we? :-)
--
Brane �ibej
home: <br...@xbc.nu> http://www.xbc.nu/brane/
work: <br...@hermes.si> http://www.hermes-softlab.com/
ACM: <br...@acm.org> http://www.acm.org/
Re: couple issues
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
Branko =?ISO-8859-2?Q?=C8ibej?= <br...@xbc.nu> writes:
>
> Jim Blandy wrote:
>
> >> Incremental database backups these days are done by having the
> >> /database/ tell the backup app what and when to copy out; after all, the
> >> database knows best what's ready and when.
> >
> >
> > Right. That's what Berkeley DB does.
> >
> Great, Then I'm getting off this topic. Looks like I've outsmarted
> myself again.
It's nice to have someone else thinking about it.
Re: couple issues
Posted by Branko Čibej <br...@xbc.nu>.
Jim Blandy wrote:
>> Incremental database backups these days are done by having the
>> /database/ tell the backup app what and when to copy out; after all, the
>> database knows best what's ready and when.
>
>
> Right. That's what Berkeley DB does.
>
Great, Then I'm getting off this topic. Looks like I've outsmarted
myself again.
--
Brane �ibej
home: <br...@xbc.nu> http://www.xbc.nu/brane/
work: <br...@hermes.si> http://www.hermes-softlab.com/
ACM: <br...@acm.org> http://www.acm.org/
Re: couple issues
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
> Incremental database backups these days are done by having the
> /database/ tell the backup app what and when to copy out; after all, the
> database knows best what's ready and when.
Right. That's what Berkeley DB does.
Re: couple issues
Posted by Branko Čibej <br...@xbc.nu>.
I wrote:
> I'll ask one of our horde of backup and storage management experts
> for an opinion.
I did, and in short: The days when it was considered a Good Thing for
backup tools to know about the semantics of a database are long gone. If
your backup app is running locally on the server, it's much faster to
copy whole files or even whole volumes. Anything else just burns resources.
Incremental database backups these days are done by having the
/database/ tell the backup app what and when to copy out; after all, the
database knows best what's ready and when. Some especially smart backup
agents can pull data directly out of the database's shared memory cache,
thereby reducing the number of copy ops even more.
Given that, let's worry about online incremental backups once we need
subversion on high-availability servers. We'd probably have to implement
the filesystem on somethig other than DB3 by then, too.
--
Brane �ibej
home: <br...@xbc.nu> http://www.xbc.nu/brane/
work: <br...@hermes.si> http://www.hermes-softlab.com/
ACM: <br...@acm.org> http://www.acm.org/
Re: couple issues
Posted by Branko Čibej <br...@xbc.nu>.
Karl Fogel wrote:
> Branko =?ISO-8859-2?Q?=C8ibej?= <br...@xbc.nu> writes:
>
>> in the "etc." category, it would be nice to have a tool that could
>> atomically pull out a set of deltas between two repository revisions.
>> Then you could do incremental, history-preserving backups, which is a
>> Good Thing.
>
>
> Hmm. That sounds like an svn client to me. :-)
Maybe, except that you'd want to do backups on the server side; maybe
useing svn_ra_local, maybe not. It's possible that the filesytesm has to
know it's doing a backup, something the svn_ra_* layers don't need to
know how to tell it.
I'll ask one of our horde of backup and storage management experts for
an opinion.
--
Brane �ibej
home: <br...@xbc.nu> http://www.xbc.nu/brane/
work: <br...@hermes.si> http://www.hermes-softlab.com/
ACM: <br...@acm.org> http://www.acm.org/
Re: couple issues
Posted by Karl Fogel <kf...@galois.collab.net>.
Branko =?ISO-8859-2?Q?=C8ibej?= <br...@xbc.nu> writes:
> We already have a tools directory, off of the top level. I'd rather put
> that in tools/server than cerate a new directory for such things.
+1
> We have too many base64 implementations now. There's one in Subversion,
> one in neon and one in APRUTIL ... I'd go for shifting whatever is
> needed to support SVN into APRUTIL and throw our implemetation away.
>
> (No, GregH, I'm not implying you did a bad job of it, just that we don't
> want to have too many copies of that stuff. :-) )
+1, assuming we don't lose any functionality.
> in the "etc." category, it would be nice to have a tool that could
> atomically pull out a set of deltas between two repository revisions.
> Then you could do incremental, history-preserving backups, which is a
> Good Thing.
Hmm. That sounds like an svn client to me. :-)
-K
Re: couple issues
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
> in the "etc." category, it would be nice to have a tool that could
> atomically pull out a set of deltas between two repository revisions.
> Then you could do incremental, history-preserving backups, which is a
> Good Thing.
Berkeley DB provides incremental, history-preserving on-line backups.
There are arguments for providing some interface to this based on the
FS interface:
- the backups' structure would have some SVN-related meaning, not just a
random set of database log files
- it's not dependent on Berkeley DB
But the first option is done and ready to use today. I'm not sure
which is better.
Re: couple issues
Posted by Branko Čibej <br...@xbc.nu>.
Greg Stein wrote:
> A couple high-level questions:
>
> 1) there is a need for some server-side tools:
> a) create a repository
> b) optimize/garbage-collect an existing repository
> c) print/purge pending/discarded transactions
> d) etc
>
> I'd like to create subversion/server/ to hold these tools. I'm not sure
> if we would have one, or whether we'd have a subcommand like the client.
> But the main point is a directory to hold the tool(s).
We already have a tools directory, off of the top level. I'd rather put
that in tools/server than cerate a new directory for such things.
> 2) APRUTIL is being put together as a sister project to APR. I'd like to
> propose using it for the following features: MD5/SHA1 hashing (MD5 is
> currently in APR itself, but it will probably move), some XML utility
> functions (and I'd like to shift some of ours over to APRUTIL), the DBM
> facilities, and possibly the URI stuff.
>
> The base64 encoding/decoding stuff is there, but it wasn't applicable to
> our needs. I'd like to look into resolving some of that (upgrading the
> base64 support in APRUTIL and/or shifting some of our support there).
We have too many base64 implementations now. There's one in Subversion,
one in neon and one in APRUTIL ... I'd go for shifting whatever is
needed to support SVN into APRUTIL and throw our implemetation away.
(No, GregH, I'm not implying you did a bad job of it, just that we don't
want to have too many copies of that stuff. :-) )
I'm getting strong urges to take neon apart and put it together again on
top of APR, but I've got other things to do right now. If Joe agrees to
that, we'll need a volunteer ...
Oh BTW, Greg, isn't there going to be some XML stuff in APRUTIL, too? I
think I saw some discussion about that on the APR list. More duplication.
> Strictly speaking, our dependency on it would be pretty light.
> mod_dav_svn can just use APRUTIL via Apache itself.
Yes, but we need base64 in the delta encoder, and md5 in the filesystem.
> However, the FS and
> the server tool(s) will need MD5 hashing and DBM support (the latter
> because mod_dav_svn will use DBM for some stuff, and the tool will need
> to deal with those).
> [ if Berkeley DB utils were exported from FS, then I'd probably use that;
> even better is to add DB3 support to APRUTIL's DBM support. ]
Definitely +1 on that last. The other option is to simply use DB3
directly; after all, it will always be distributed with the server.
Wouldn't it look weird if we used two different dbm implementations in
Subversion?
> I'm a bit light on rationale for this, but figured it would be good to
> bring up now.
>
>
> Input/thoughts appreciated...
Well, it's definitely a good thing to have stand-alone administrative
tools for managing the repository. I think these tools would use
libsvn_fs, like the command line client uses libsvn_client. In fact that
would be a good thing: the filesystem must already know how to do a good
bit of that administration, and if it learned to do the rest, you could
administer it while online, without having to lock the repository or
block readers.
in the "etc." category, it would be nice to have a tool that could
atomically pull out a set of deltas between two repository revisions.
Then you could do incremental, history-preserving backups, which is a
Good Thing.
--
Brane �ibej
home: <br...@xbc.nu> http://www.xbc.nu/brane/
work: <br...@hermes.si> http://www.hermes-softlab.com/
ACM: <br...@acm.org> http://www.acm.org/
Re: couple issues
Posted by Karl Fogel <kf...@galois.collab.net>.
RADICS Peter <mi...@lbcons.net> writes:
> It just triggered an old marker in my brains: I would really like an
> easy way to create a "local repository" from a base repository, commit
> my changes to this local repo, and from time to time merge these changes
> to the base repository.
This is one specific case of the general cross-repository branching
and merging problem. Subversion will support this, though not 1.0, as
you pointed out. Interface? I have no idea. :-)
> I know it's possible to do this with CVS
> somehow, but it's definitely not "easy".
It is? I can't think of how to do it with CVS, not in any reasonably
automated fashion at least.
Re: couple issues
Posted by RADICS Peter <mi...@lbcons.net>.
On Sat, Dec 09, 2000 at 05:44:35AM -0800, Greg Stein wrote:
> A couple high-level questions:
>
> 1) there is a need for some server-side tools:
> a) create a repository
It just triggered an old marker in my brains: I would really like an
easy way to create a "local repository" from a base repository, commit
my changes to this local repo, and from time to time merge these changes
to the base repository. I know it's possible to do this with CVS
somehow, but it's definitely not "easy". It should work without any
additional magic and HOWTO reading: should be as easy as working with
the base repository directly. Any thoughts on an intuitive interface
for that? OK, I know it's post 1.0, but I'd really welcome it in 2.0 :)
[SNIP]
cheers,
mitch
--
// RADICS Peter <mi...@lbcons.net> (http://lbcons.net)
//
// "If human beings don't keep exercising their lips,
// their brains start working." -- Ford Prefect
Re: couple issues
Posted by Greg Stein <gs...@lyra.org>.
On Tue, Dec 12, 2000 at 09:13:19AM -0500, Jim Blandy wrote:
>
> > Strictly speaking, our dependency on it would be pretty light.
> > mod_dav_svn can just use APRUTIL via Apache itself. However, the FS and
> > the server tool(s) will need MD5 hashing and DBM support (the latter
> > because mod_dav_svn will use DBM for some stuff, and the tool will need
> > to deal with those).
> > [ if Berkeley DB utils were exported from FS, then I'd probably use that;
> > even better is to add DB3 support to APRUTIL's DBM support. ]
>
> Fitting Berkeley DB into some generic DBM interface sounds pretty
> doable. I'd also be happy to export some generic "utility" functions
> from svn_fs.h, which might allow tighter integration with a given FS
> object.
I just coded a whopper patch into the "apr_dbm" feature of APRUTIL. The
interface now talks to: SDBM, GDBM, DB1, DB_185, DB2, and DB3. Whee!
[ the DB* part is new. still have some configure magic and testing, but the
code is fully written and compiles against everything ]
Background: mod_dav_svn has to store mappings of DAV Activity names to FS
Transaction names. This will be a little bitty table manipulated thru
apr_dbm (rather than coding against a specific DBM interface).
The server admin tool would also use apr_dbm to read mod_dav_svn's tables.
We do have a slight gotcha in case Apache is built with an APRUTIL that uses
a different backend DBM than our server admin tool. I believe we'll be okay,
though, as long as both Apache and SVN allow configure to just default both
of them. We can also have SVN's configure examine the Apache installation to
see what it used and use the same setting for SVN's copy of APRUTIL.
[ and one day, we'll have aprutil as a shared lib and it won't matter ]
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: couple issues
Posted by Jim Blandy <ji...@zwingli.cygnus.com>.
> Strictly speaking, our dependency on it would be pretty light.
> mod_dav_svn can just use APRUTIL via Apache itself. However, the FS and
> the server tool(s) will need MD5 hashing and DBM support (the latter
> because mod_dav_svn will use DBM for some stuff, and the tool will need
> to deal with those).
> [ if Berkeley DB utils were exported from FS, then I'd probably use that;
> even better is to add DB3 support to APRUTIL's DBM support. ]
Fitting Berkeley DB into some generic DBM interface sounds pretty
doable. I'd also be happy to export some generic "utility" functions
from svn_fs.h, which might allow tighter integration with a given FS
object.