You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Neil Schemenauer na...@mems-exchange.org on 2003/01/23 17:14:37 UTC
A performance tip
Before running svnadmin load or svnadmin recover, add set_flags
DB_TXN_NOSYNC in the DB_CONFIG file. It gives a massive speed
increase. Remember to remove it after since I'm not sure if it's safe.
Could svnadmin set the flag itself? It stupid for it to do millions of
fsync calls while running instead a handful when it completes. Try
running strace on svnserve, svnadmin or the apache. The number of fsync
calls made is truly staggering.
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: A performance tip
Posted by mark benedetto king mb...@boredom.org.
On Thu, Jan 23, 2003 at 10:49:07AM -0800, Bill Tutt wrote:
The smaller our BDB transactions are the better off our datastore will
be for scalability. As far as I know, BDB doesn't let us get away with
doing dirty reads (no read lock was acquired) on known non-changeable
data. BDB also doesn't do lock escalation. I don't recall whether or not
BDB supports row locks or not. It might only support page locks. If
that's the case the smaller our BDB transactions the less time we hold
read locks open that prevent writing threads from getting about their
business of creating new Subversion transactions (or whatever).
IANA BDB person, but db_open supports:
DB_DIRTY_READ Support dirty reads; that is, transactions, cursors and read operations in the database may request the return of modified but not yet committed data.
and db_get has:
DB_DIRTY_READ Read modified but not yet committed data. Silently ignored if the DB_DIRTY_READ flag was not specified when the underlying database was opened.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: A performance tip
Posted by Bill Tutt ra...@lyra.org.
From: Branko Cibej [mailto:brane@xbc.nu]
Somebody please tell me I'm totally mad...
No, you're not mad. I don't know if BDB lets you do this. It's certainly
worth a try.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: A performance tip
Posted by Branko Čibej br...@xbc.nu.
Bill Tutt wrote:
The smaller our BDB transactions are the better off our datastore will
be for scalability. As far as I know, BDB doesn't let us get away with
doing dirty reads (no read lock was acquired) on known non-changeable
data. BDB also doesn't do lock escalation. I don't recall whether or not
BDB supports row locks or not. It might only support page locks. If
that's the case the smaller our BDB transactions the less time we hold
read locks open that prevent writing threads from getting about their
business of creating new Subversion transactions (or whatever).
I started wondering yesterday if we couldn't simple blow away most of
our transactions anyway.
Most of the data in the filesystem is read-only. The only times we
really need transactions are
* acquiring new unique keys in the tables that use the next-id hack
* commits of mutable txns
* changes to the representation skels during
deltification/undeltification
We don't even have to make a mutable node within a transaction, since
the source node is read-only and the target is within a txn that we know
is being modified serially, by only one client.
For things like reading string contents, it would be much better to use
locks rather than transactions. That doesn't involve the logging and
fsync overhead. It's really stupid that the BDB log files keep growing
even when people are only reading from the repository.
Somebody please tell me I'm totally mad...
--
Brane Čibej brane@xbc.nu http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: A performance tip
Posted by Bill Tutt ra...@lyra.org.
The smaller our BDB transactions are the better off our datastore will
be for scalability. As far as I know, BDB doesn't let us get away with
doing dirty reads (no read lock was acquired) on known non-changeable
data. BDB also doesn't do lock escalation. I don't recall whether or not
BDB supports row locks or not. It might only support page locks. If
that's the case the smaller our BDB transactions the less time we hold
read locks open that prevent writing threads from getting about their
business of creating new Subversion transactions (or whatever).
FYI,
Bill
From: cmpilato@collab.net [mailto:cmpilato@collab.net]
Justin Erenkrantz jerenkrantz@apache.org writes:
I'm starting to think some of our transactions are a bit too
fine-grained and we're getting killed by the fact that they are
ridiculously expensive in BDB.
C-Mike? -- justin
I don't think we're getting killed by anything, or that
transactions are ridiculously expensive. :-)
Yeah, we make heavy use of BDB transactions, and we make no
attempt to group operations into a single transaction. But
I'm not thrilled about the alternatives. With ra-dav opening
and closing the database once-per-operation (practically), we
would have to expose the database transaction layer through
the libsvn_fs interface. I suppose we could open a master
transaction at svn_fs_open() time, and attach it to the
svn_fs_t * object, then register a pool cleanup function that
closes the environment.
But that's starting to look a little yucky, and I'm not sure
how it affects our movement into different database backends.
Also, I'd like to know if there are operations in the FS
that depend on that granularity.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: A performance tip
Posted by cm...@collab.net.
Justin Erenkrantz jerenkrantz@apache.org writes:
I'm starting to think some of our transactions are a bit too
fine-grained and we're getting killed by the fact that they are
ridiculously expensive in BDB.
C-Mike? -- justin
I don't think we're getting killed by anything, or that transactions
are ridiculously expensive. :-)
Yeah, we make heavy use of BDB transactions, and we make no attempt to
group operations into a single transaction. But I'm not thrilled
about the alternatives. With ra-dav opening and closing the database
once-per-operation (practically), we would have to expose the database
transaction layer through the libsvn_fs interface. I suppose we could
open a master transaction at svn_fs_open() time, and attach it to the
svn_fs_t * object, then register a pool cleanup function that closes
the environment.
But that's starting to look a little yucky, and I'm not sure how it
affects our movement into different database backends. Also, I'd like
to know if there are operations in the FS that depend on that
granularity.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: A performance tip
Posted by Justin Erenkrantz je...@apache.org.
--On Thursday, January 23, 2003 12:14 PM -0500 Neil Schemenauer
nas@mems-exchange.org wrote:
Before running svnadmin load or svnadmin recover, add set_flags
DB_TXN_NOSYNC in the DB_CONFIG file. It gives a massive speed
increase. Remember to remove it after since I'm not sure if it's
safe.
Could svnadmin set the flag itself? It stupid for it to do
millions of fsync calls while running instead a handful when it
completes. Try running strace on svnserve, svnadmin or the apache.
The number of fsync calls made is truly staggering.
Yeah, the fact that BDB does an fsync() after every transaction is a
massive performance problem overall. (Consider we do a transaction
for every value retrieved, etc.)
I think the better/best thing to do would be to do this recovery in
one transaction. I have a hunch that this isn't possible given the
current state of libsvn_fs. What we'd potentially lose in atomicity
(by having larger-grained txn's), we'll gain in speed.
I'm starting to think some of our transactions are a bit too
fine-grained and we're getting killed by the fact that they are
ridiculously expensive in BDB.
C-Mike? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org