You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by John Peacock <jp...@rowman.com> on 2003/06/05 13:46:53 UTC
Re: Which Berkeley version to use? NOT 4.1.x!!!
As a late response to a thread from Feb 2003, I can personally confirm that
although Berkeley 4.1.x _will_ work with the current subversion, it is rather
massively slower and much more demanding of drive space.
I attempted to load the APC (All Perl Changes - all Perl patches since 5.004)
first using Berkeley DB 4.1.24 and subversion 0.22. It ran for ~36 hours before
I finally gave up; it barely got through 2000 revisions and I had to delete the
old log files twice (I only had 12GB free on that partition!).
Then I rebuilt using Berkeley DB 4.0.14 and subversion 0.23 and started from
scratch. The import took ~18 hours (~16000 revisions) and generated only ~10GB
of db log files total. Granted, I did also go from 128MB to 512MB on that
system, but I doubt that would account for all of the speed difference, or any
of the logging difference.
Either 4.1.x is much more paranoid about checkpoints or there is something wrong
with the way svn is interacting with BDB, I think. I'll have to defer to
someone who actually knows the code to fix it.
Hope this helps the next person trying to make this decision...
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by Branko Čibej <br...@xbc.nu>.
Ben Collins-Sussman wrote:
>John Peacock <jp...@rowman.com> writes:
>
>
>
>>Either 4.1.x is much more paranoid about checkpoints or there is
>>something wrong with the way svn is interacting with BDB, I think.
>>I'll have to defer to someone who actually knows the code to fix it.
>>
>>
>
>I think most of the developers here are certain that libsvn_fs needs
>to be doing a whole lot less BDB checkpointing, and create many fewer
>BDB transactions. BDB 4.1 seems to be more "sensitive" to our
>mis-usage of this features. :-)
>
>
Yeah... I've been looking at that on and off for several months, and the
conclusion I always come up with is that the trail mechanism is totally
bogus, and that our FS implementation layer separation (tree vs. dag vs.
BDB backend) leaves much to be desired. I'm sort of hoping that GAT's FS
refactoring effort will make it easier to get this fixed.
What we really have to do is use BDB's implicit locking for read-only
operations, and small, short transactions for writes. There are probably
even some operations that don't require transactions even for writes.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by Ben Collins-Sussman <su...@collab.net>.
John Peacock <jp...@rowman.com> writes:
> Either 4.1.x is much more paranoid about checkpoints or there is
> something wrong with the way svn is interacting with BDB, I think.
> I'll have to defer to someone who actually knows the code to fix it.
I think most of the developers here are certain that libsvn_fs needs
to be doing a whole lot less BDB checkpointing, and create many fewer
BDB transactions. BDB 4.1 seems to be more "sensitive" to our
mis-usage of this features. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by Ben Collins <bc...@debian.org>.
On Thu, Jun 05, 2003 at 11:13:05AM -0400, John Peacock wrote:
> Ben Collins wrote:
> >I would go back and compare 0.23 against 0.23 just to be sure the
> >logging problem isn't the result of changes in SVN though.
> >
>
> Sure, it's only disk space and cpu. It's not like I have to do any work...
> ;~)
Don't blame this on me...you suckered yourself into testing :)
--
Debian - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo - http://www.deqo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by John Peacock <jp...@rowman.com>.
Ben Collins wrote:
> I would go back and compare 0.23 against 0.23 just to be sure the
> logging problem isn't the result of changes in SVN though.
>
Sure, it's only disk space and cpu. It's not like I have to do any work... ;~)
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by Jani Monoses <ja...@iv.ro>.
On Thu, 5 Jun 2003 10:04:51 -0400
Ben Collins <bc...@debian.org> wrote:
> On Thu, Jun 05, 2003 at 11:02:48AM -0400, John Peacock wrote:
> > David Waite wrote:
> > >Not saying whether or not 4.1.x is this much slower or not, but
> > >this doesn't seem like a very fair test - 0.22 vs 0.23, 128 MB vs
> > >512 MB.
> >
> > I did want to add that caveat as a possible factor, but low memory
> > shouldn't cause extra logging, should it? The system was pretty
> > much unloaded other than this import, so any thrashing was just the
> > svn server processes themselves.
>
> I agree. I would say the logging is the cause of the slowdown.
>
> I would go back and compare 0.23 against 0.23 just to be sure the
> logging problem isn't the result of changes in SVN though.
FWIW
I recall seeing 4.1.24 eat a lot of memory until svn was OOM killed
back at 0.18. The same svn 0.18 worked just fine with BDB 4.0.14
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by Ben Collins <bc...@debian.org>.
On Thu, Jun 05, 2003 at 11:02:48AM -0400, John Peacock wrote:
> David Waite wrote:
> >Not saying whether or not 4.1.x is this much slower or not, but this
> >doesn't seem like a very fair test - 0.22 vs 0.23, 128 MB vs 512 MB.
>
> I did want to add that caveat as a possible factor, but low memory
> shouldn't cause extra logging, should it? The system was pretty much
> unloaded other than this import, so any thrashing was just the svn server
> processes themselves.
I agree. I would say the logging is the cause of the slowdown.
I would go back and compare 0.23 against 0.23 just to be sure the
logging problem isn't the result of changes in SVN though.
--
Debian - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo - http://www.deqo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by John Peacock <jp...@rowman.com>.
David Waite wrote:
> Not saying whether or not 4.1.x is this much slower or not, but this
> doesn't seem like a very fair test - 0.22 vs 0.23, 128 MB vs 512 MB.
I did want to add that caveat as a possible factor, but low memory shouldn't
cause extra logging, should it? The system was pretty much unloaded other than
this import, so any thrashing was just the svn server processes themselves.
I'm getting fairly good at recompiling Apache/apr/neon/subversion ;~), so if you
_really_ want, I can try and do an apples-to-apples test next week. I'll have
to tweak apc2svn slightly to delete the log files at intervals to keep myself
from running out of disk.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Which Berkeley version to use? NOT 4.1.x!!!
Posted by David Waite <ma...@akuma.org>.
Not saying whether or not 4.1.x is this much slower or not, but this
doesn't seem like a very fair test - 0.22 vs 0.23, 128 MB vs 512 MB.
-David Waite
John Peacock wrote:
> As a late response to a thread from Feb 2003, I can personally confirm
> that although Berkeley 4.1.x _will_ work with the current subversion,
> it is rather massively slower and much more demanding of drive space.
>
> I attempted to load the APC (All Perl Changes - all Perl patches since
> 5.004) first using Berkeley DB 4.1.24 and subversion 0.22. It ran for
> ~36 hours before I finally gave up; it barely got through 2000
> revisions and I had to delete the old log files twice (I only had 12GB
> free on that partition!).
>
> Then I rebuilt using Berkeley DB 4.0.14 and subversion 0.23 and
> started from scratch. The import took ~18 hours (~16000 revisions)
> and generated only ~10GB of db log files total. Granted, I did also
> go from 128MB to 512MB on that system, but I doubt that would account
> for all of the speed difference, or any of the logging difference.
>
> Either 4.1.x is much more paranoid about checkpoints or there is
> something wrong with the way svn is interacting with BDB, I think.
> I'll have to defer to someone who actually knows the code to fix it.
>
> Hope this helps the next person trying to make this decision...
>
> John
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org