You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Karl Fogel <kf...@newton.ch.collab.net> on 2003/02/21 02:20:44 UTC
When to use Berkeley transactions.
This is the other half of the Discussion Formerly Known As
"Transcript of chat between me and Sleeepycat", which was
"Checkpoint less frequently", which used to be called
"Still hang on svn 4951 RedHat 7.3 SMP"
At this point, I think I'll not bother to include former subject lines
with "(Was: ...)" :-)
Brane hasn't described his vision for replacing Berkeley transactions
with locking yet, so it's possible that what I write here will be
superseded by his proposal. However, I'm *pretty* confident that
Subversion's current use of transactions is necessary and appropriate,
and will try to explain why we shouldn't reduce their use.
What is a Berkeley transaction?
A transaction is effectively a private copy of the entire database.
Anything you do in a transaction is invisible to those outside the
transaction; and anything done outside a transaction is invisible to
those inside it. When a transaction is committed, it either entirely
succeeds, or entirely fails. (If this sounds familiar, it's only
because Berkeley and Subversion have some of the same needs; I'm
describing Berkeley transactions here, not Subversion txns :-) ).
Sometimes, in order to maintain these guarantees, the transaction must
lock records, causing other readers or writers to block until the
transaction completes. (The Berkeley docs have some good examples and
discussion at http://www.sleepycat.com/docs/ref/transapp/inc.html).
Subversion uses Berkeley transactions for almost everything, because
it needs to maintain semantic integrity across tables. For example,
imagine a checkout without transactions:
1. During the checkout, the fs code reads a representation into
memory.
2. Now some commit causes that node to be deltified. The
representation's key does not change, but the value at that key
does, because the rep now points to a new (svndiff) string in
the `strings' table. The old, fulltext string is removed.
3. Now the checkout thread uses its in-memory rep to fetch the
node's data. Whups! No string there, sorry.
With transactions, there's no problem: step 2 just blocks until the
checkout's transaction closes.
Of course, we could do it without transactions, if we implemented our
own locking scheme. And that locking scheme would be equivalent in
power and scope to... Berkeley transactions!
I suspect this is how transactions got invented. People realized that
individual locking schemes could be abstracted and implemented inside
the database itself, and then they'd never have to worry about it
again.
I'm not saying it wouldn't be more efficient to have our own locking
scheme. It probably would. We know the data intimately; we can take
shortcuts that Berkeley would never dare. But I don't think the gain
would be very large (after all, Sleepcat works helluva hard to
implement transactions as efficiently as they can), and it would come
at a high cost in complexity and new bugs.
So, barring a surprisingly brilliant insight from Brane (I'm not
saying it can't happen...), my feeling is that we should leave
Subversion's use of BDB transactions alone, and concentrate on
changing our checkpointing and other things.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
"Glenn A. Thompson" <gt...@cdr.net> writes:
> I thought the reps table was there to provide for other forms of data
> representation.
> In a SQL world, I think I can represent collections of properties and
> directories in a way that allows for more adhoc query
> capability. Currently it involves 4 tables not 2 :-)
Yes, that's also true.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>>
>>
>
>We didn't. We started out with one table, and added the layer of
>indirection because it was easier for us to wrap our brains around
>that way. (See changes around 2001-06-26 in the old CVS ChangeLog).
>
>
>
>
I thought the reps table was there to provide for other forms of data
representation.
In a SQL world, I think I can represent collections of properties and
directories in a way that allows for more adhoc query capability.
Currently it involves 4 tables not 2 :-)
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Greg Hudson <gh...@MIT.EDU> writes:
> I don't think the current situation is very clean. It presents the
> possiblities of representations pointing to nonexistent strings and
> strings with no representations pointing to them. It means an extra
> step to remove the old string when the representation is updated. It's
> an extra step on every access.
>
> (I admit that the transition from two tables to one would be pretty
> harsh. I'm just not sure why we designed it this way in the first
> place.)
We didn't. We started out with one table, and added the layer of
indirection because it was easier for us to wrap our brains around
that way. (See changes around 2001-06-26 in the old CVS ChangeLog).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2003-02-21 at 02:19, Branko Čibej wrote:
>> But if we're already locking the representation (using transactions or
>> not), why can't we just rewrite the representation--without changing its
>> key, of course--in multiple steps?
> It would also make moving to locks for representation reads much more
> complicated.
I don't understand why. Either way you have to lock around the
representation before reading it or rewriting it.
> And given caching, I don't think the extra indirection is all that
> important.
My intuition says that caching probably isn't helping us here, since the
important operations iterate through data sets larger than the cache.
> Let's just keep things clean, the way they are now.
I don't think the current situation is very clean. It presents the
possiblities of representations pointing to nonexistent strings and
strings with no representations pointing to them. It means an extra
step to remove the old string when the representation is updated. It's
an extra step on every access.
(I admit that the transition from two tables to one would be pretty
harsh. I'm just not sure why we designed it this way in the first
place.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Greg Hudson wrote:
>On Fri, 2003-02-21 at 02:02, Branko Čibej wrote:
>
>
>>Because we want to isolate node records from deltification. When we
>>deltify a file's contents, we create a completely new string (with a new
>>key) and delete the old one, but only modify the representation record
>>without changing its key.
>>
>>
>
>But if we're already locking the representation (using transactions or
>not), why can't we just rewrite the representation--without changing its
>key, of course--in multiple steps? (Unless there's some complication
>with doing that in the presence of duplicate keys.)
>
>(That might mean keeping representations locked for a longer period of
>time during redeltification. But it would mean one less layer of
>indirection every time we access a representation, which has to make
>more of a difference.)
>
>
It would also make moving to locks for representation reads much more
complicated. And given caching, I don't think the extra indirection is
all that important.
Let's just keep things clean, the way they are now.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2003-02-21 at 02:02, Branko Čibej wrote:
> Because we want to isolate node records from deltification. When we
> deltify a file's contents, we create a completely new string (with a new
> key) and delete the old one, but only modify the representation record
> without changing its key.
But if we're already locking the representation (using transactions or
not), why can't we just rewrite the representation--without changing its
key, of course--in multiple steps? (Unless there's some complication
with doing that in the presence of duplicate keys.)
(That might mean keeping representations locked for a longer period of
time during redeltification. But it would mean one less layer of
indirection every time we access a representation, which has to make
more of a difference.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Greg Hudson wrote:
>You're right; given that we use transactions, I'm not sure why we have
>separate representations and strings tables.
>
>
Because we want to isolate node records from deltification. When we
deltify a file's contents, we create a completely new string (with a new
key) and delete the old one, but only modify the representation record
without changing its key. Nodes contain representation keys, and many
nodes can use the same representation. Using a separate table for the
strings means that we don't have to update all the nodes when we deltify
the contents, or indeed even maintain back references from
representations to nodes.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2003-02-20 at 23:57, Karl Fogel wrote:
> We do change the file in safe steps behind a static reference point --
> as long as we use some sort of locking mechanism (such as BDB
> transactions) to ensure that the old reference point still has
> something to refer to for as long as it's being used. That's
> inevitable; regular filesystems are doing their own transactioning
> behind the scenes, too.
The difference is that regular filesystems have an atomic update
operation (rename). There may be some kernel locking under the covers,
but if so, it's a very small locking window using very efficient locks.
Berkeley DB also has an atomic update operation (put), but it's not
compatible with using duplicate keys (or DB_APPEND) to create a value in
multiple steps. So we have to use a more heavyweight mechanism
(transactions) to protect readers from writers. I'm not sure if that's
responsible for the scaling problems Brandon is seeing, but it probably
limits our throughput. (Possibly in some nasty ways, since every
transaction touches the root directory.)
> I don't think it has anything to do with separate tables or not. We
> need the lock (or transaction) around read operations because we
> change the way the data is represented.
But you can read without locking if you have atomic updates.
> We could do that just as easily if it were all in one table; and we
> could avoid doing it, while still using separate tables.
You're right; given that we use transactions, I'm not sure why we have
separate representations and strings tables.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Greg Hudson <gh...@MIT.EDU> writes:
> This is an interesting weakness of Berkeley DB relative to the
> filesystem as a storage tool for large data objects.
>
> We have separate reps and strings tables so that we can have a stable
> reference point for the representation while we build up a new string
> entry in several steps (specifically, during redeltification). If we
> didn't have separate tables, then the contents of a revision would
> always be valid, and we wouldn't need to lock around read operations.
>
> In the filesystem, you can build up a file in several steps and rename
> it into place (at least on Unix, and I think also on Windows), without
> changing the reference point.
I didn't understand this.
We do change the file in safe steps behind a static reference point --
as long as we use some sort of locking mechanism (such as BDB
transactions) to ensure that the old reference point still has
something to refer to for as long as it's being used. That's
inevitable; regular filesystems are doing their own transactioning
behind the scenes, too.
> If we
> didn't have separate tables, then the contents of a revision would
> always be valid, and we wouldn't need to lock around read operations.
I don't think it has anything to do with separate tables or not. We
need the lock (or transaction) around read operations because we
change the way the data is represented. We could do that just as
easily if it were all in one table; and we could avoid doing it, while
still using separate tables.
If we wanted to, we could store large data objects in BDB and simply
agree never to change their representations; then BDB would behave
just like a filesystem.
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2003-02-20 at 21:20, Karl Fogel wrote:
> For example, imagine a checkout without transactions:
[Failure scenario involving the reps/strings table indirection]
This is an interesting weakness of Berkeley DB relative to the
filesystem as a storage tool for large data objects.
We have separate reps and strings tables so that we can have a stable
reference point for the representation while we build up a new string
entry in several steps (specifically, during redeltification). If we
didn't have separate tables, then the contents of a revision would
always be valid, and we wouldn't need to lock around read operations.
In the filesystem, you can build up a file in several steps and rename
it into place (at least on Unix, and I think also on Windows), without
changing the reference point.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Branko Čibej <br...@xbc.nu> writes:
> I'd say create a branch and do one of the simple modifications (nodes,
> changes, representations, revisions, uuids), and test. If it looks fine,
> move on to strings and test. Then test. After that, test some more. In
> the meantime, while the stress tests are running, read the BDB
> documentation again. After that, test again. Then ask people to pull the
> sources off that branch and test.
>
> Finally, merge to trunk and pray. :-)
:-).
Unless we face an unacceptable performance problem that cannot be
solved some other way, though, I doubt we'll see this before 1.0. At
least, I know I sure don't have time to be rewriting our fs locking
guarantees at the moment, and we've already seen how much gain can be
got through much simpler changes to calling txn_checkpoint.
Not trying to blow your ideas off, just being realistic (I think)
about resources.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>
>It would be much better, from a coding point of view, if your change
>went in first. As to priority... Well, it's likely that performance and
>scalability are more important ATM than supporting SQL backends.
>
I really tried to *not* make my refactoring changes SQL specific. I'm
working for peaceful coexistence :-)
There are plenty of people who will never embrace a SQL DB. And for
valid reasons.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Glenn A. Thompson wrote:
>>
>>
>> (Yes, I'll volunteer to do this, but after the issue 1003 branch gets
>> merged to trunk. And after I get back from vacation, whichever comes
>> last.)
>>
> ~ How much time would this give me if I wanted to get my patch in
> before this change?
I can't speak for the 1003 change, but I get back from vacatoin in 10
days from now.
> Or is this change a higher priority?
It would be much better, from a coding point of view, if your change
went in first. As to priority... Well, it's likely that performance and
scalability are more important ATM than supporting SQL backends. But
it's not my call. Personally, I'd perfer to start hacking on this after
your DB backend changes are in.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by cm...@collab.net.
"Glenn A. Thompson" <gt...@cdr.net> writes:
> >(Yes, I'll volunteer to do this, but after the issue 1003 branch gets
> >merged to trunk. And after I get back from vacation, whichever comes last.)
>
> ~ How much time would this give me if I wanted to get my patch in
> before this change? Or is this change a higher priority?
I'm messing with 1003 right now. It has a couple of bugs in it, and
I've sent a mail to Bill Tutt to see how he feels about reverting a
part of his change that I don't see the need for.
As for priority, 1003 definitely has it. Part of the changes there
address existing bugs and inefficiencies in the filesystem code. As
far as I can recall, your changes don't address any run-time
brokenness -- they address theoretical / modularity issue.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>It would be easier to make that judgement if your code was on a branch.
>
>
As I understand it, the SQL stuff will be.
I'm only refering to my first round of re-factoring.
IMHO my changes will also help anyone building a "file system" repo.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: When to use Berkeley transactions.
Posted by Sander Striker <st...@apache.org>.
> From: Glenn A. Thompson [mailto:gthompson@cdr.net]
> Sent: Friday, February 21, 2003 7:35 PM
>> (Yes, I'll volunteer to do this, but after the issue 1003 branch gets
>> merged to trunk. And after I get back from vacation, whichever comes last.)
>>
> ~ How much time would this give me if I wanted to get my patch in before
> this change?
It would be easier to make that judgement if your code was on a branch.
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>(Yes, I'll volunteer to do this, but after the issue 1003 branch gets
>merged to trunk. And after I get back from vacation, whichever comes last.)
>
~ How much time would this give me if I wanted to get my patch in before
this change?
Or is this change a higher priority?
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Branko Čibej <br...@xbc.nu> writes:
> (Yes, I'll volunteer to do this, but after the issue 1003 branch gets
> merged to trunk. And after I get back from vacation, whichever comes last.)
My ankles just exploded in gratitude.
It's painful, but I don't mind.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Branko Čibej wrote:
>Karl Fogel wrote:
>
>
>
>>So the question is, what's the best course of action? Is there an
>>initial small-change-with-big-effect that you think would be best to
>>start with first?
>>
>>
>>
>>
>I think nodes, revisions and representations would be a good beginning.
>
>I'd say create a branch and do one of the simple modifications (nodes,
>changes, representations, revisions, uuids), and test. If it looks fine,
>move on to strings and test. Then test. After that, test some more. In
>the meantime, while the stress tests are running, read the BDB
>documentation again. After that, test again. Then ask people to pull the
>sources off that branch and test.
>
>Finally, merge to trunk and pray. :-)
>
>
(Yes, I'll volunteer to do this, but after the issue 1003 branch gets
merged to trunk. And after I get back from vacation, whichever comes last.)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Karl Fogel wrote:
>So the question is, what's the best course of action? Is there an
>initial small-change-with-big-effect that you think would be best to
>start with first?
>
>
I think nodes, revisions and representations would be a good beginning.
I'd say create a branch and do one of the simple modifications (nodes,
changes, representations, revisions, uuids), and test. If it looks fine,
move on to strings and test. Then test. After that, test some more. In
the meantime, while the stress tests are running, read the BDB
documentation again. After that, test again. Then ask people to pull the
sources off that branch and test.
Finally, merge to trunk and pray. :-)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Branko Čibej <br...@xbc.nu> writes:
> No it's not. A transaction is _not_ invisible to other users of the
> database, even when not committed. Nobody outside the transaction sees
> the changes until they're committed, but everybody sees the locks that
> the transaction creates -- and the transaction sees other people's locks.
Uh, right. Since I said the same thing about the locks in my original
post, I assumed it was obvious what "invisible" meant in this context
:-). I could describe it more elaborately if needed; perhaps "opaque"
would have been a better word.
> [skip lots of txn-101 that Karl just invented]
Brane, wouldn't you rather be specific than rude, dear? :-)
> >Of course, we could do it without transactions, if we implemented our
> >own locking scheme. And that locking scheme would be equivalent in
> >power and scope to... Berkeley transactions!
>
> There's another locking scheme, and it's called Berkeley locks. You can
> lock and unlock an object (a table row, if you will) without starting a
> transaction (and all the logging that involves). It's just that locking
> happens automatically when you touch an object from within a transaction.
Yup.
> >and it would come at a high cost in complexity and new bugs.
>
> Our code wouldn't become that much more complex.
That's great! (Surprising to me, but it's clear you've given it more
thought.)
> The nice thing about this, implementation-wise, is that we can make a
> /gradual/ move from reads-with-txns to reads-with-locks. Do the easy
> ones first ('revisions', 'nodes', 'representations'), then move on to
> the hard ones.
Gotcha.
> P.S.: All of the above assumes that we don't have "svn obliterate". Once
> we have that operations, we'll definitely have to lock 'nodes' and
> 'changes'.
Sure, we can cross that bridge when we come to it.
So the question is, what's the best course of action? Is there an
initial small-change-with-big-effect that you think would be best to
start with first?
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>There's no reason for it to be any more BDB specific than the current
>transactions are. I was thinking of expanding the trail interface
>slightly. The BDB implementation would handle locking for read-only
>operations. What the *SQL implementation does is then irrelevant, as
>long as it avoids race conditions on table access.
>
This sounds good.
>
>
>
>>I have no doubt that you can make it cleaner and faster. But will it
>>be as flexible?
>>
>>
>
>Can it get any less flexible than it is now? :-)
>
Sure it could.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Glenn A. Thompson wrote:
> But will it become more BDB specific?
There's no reason for it to be any more BDB specific than the current
transactions are. I was thinking of expanding the trail interface
slightly. The BDB implementation would handle locking for read-only
operations. What the *SQL implementation does is then irrelevant, as
long as it avoids race conditions on table access.
> I have no doubt that you can make it cleaner and faster. But will it
> be as flexible?
Can it get any less flexible than it is now? :-)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>Aaargh! The bookkeeping is still there (more so, I'd guess), and on top
>of that, the code becomes totally unreadabla. I can see it now... a ton
>of questions to the tune, "Why're we aborting this transaction if
>nothing failed?"
>
>
Yikes! I meant as a test. I'm crazy but not quite that crazy:-)
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Glenn A. Thompson wrote:
> I don't get this. If nothing changed why all the fsyncs. They are not
> related to the locks are they? They are only releated to the log
> flushes. Right?
And to environment cleanup, but yes, the log flushes are the major
slowdown. Any time you create and commit a transaction, BDB will write a
bit of into a log file even if you don't make any changes. I don't konw
_what_ it writes, but I guess it's probably bookkeeping.
> I wonder if read-only transactions were aborted instead of commited
> what would happen?
Aaargh! The bookkeeping is still there (more so, I'd guess), and on top
of that, the code becomes totally unreadabla. I can see it now... a ton
of questions to the tune, "Why're we aborting this transaction if
nothing failed?"
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
Hey,
>>What is a Berkeley transaction?
>>
>>A transaction is effectively a private copy of the entire database.
>>
>>
>>
>No it's not. A transaction is _not_ invisible to other users of the
>database, even when not committed. Nobody outside the transaction sees
>the changes until they're committed, but everybody sees the locks that
>the transaction creates -- and the transaction sees other people's locks.
>
>
This is implementation dependent. In a multi-versioning scheme like
used in Oracle and MySQL InnoDB tables, what Karl said is more true than
not true.
He did use the word effectively not invisible:-)
>
>
>>I'm not saying it wouldn't be more efficient to have our own locking
>>scheme. It probably would.
>>
>>
>>
>It would be a _lot_ more efficient. Every time you create a transaction,
>you end up writing to the log files. And every time you commit it, you
>have to sync the logs. That's a _lot_ of fsyncs, and they're slowing us
>down tremendously. What's worse, we're writing and sync'ing logs even
>for read-only operations that don't change a single bit in the database.
>
>
>>
>>
>However hard they work on making transactions efficient, they can't
>avoid the fsyncs. In case you're wondering if those fsyncs are really
>that bad, hear this: I hardcoded a DB_NOSYNC when opening the database
>in fs.c -- and reduced "make check" time on Windows by *half*. Of
>course, we can't do that for a production repository, but it makes you
>think, doesn't it?
>
>
I don't get this. If nothing changed why all the fsyncs. They are not
related to the locks are they? They are only releated to the log
flushes. Right?
I wonder if read-only transactions were aborted instead of commited what
would happen?
>
>
>>and it would come at a high cost in complexity and new bugs.
>>
>>
>>
>Our code wouldn't become that much more complex.
>
>
But will it become more BDB specific?
Let's face it. Over time there will be two camps form around the data
store. The use "a file system" folks and the use "a more sophisticated
DB" folks.
IMHO Both camps will be hurt if this imposes more specific access
patterns. I plan on having more than one physical schema to support the
same logical model. Will your changes impose even more special access
patterns? If so, where will the specific locking be handled? Would it
be handled inside, outside or both inside and outside of the current
trail framework? You won't promote it to the fs api right?
If you had a chance to apply and look at the all-in-one version of my
refactoring patch you may have noticed that I have two levels of
indirection in there.
One at fs api level and one at the DB access level. I convinced myself
that I can make a SQL back end that fits the current mold quite cleanly
at the DB access level. This would allow me to create a functional SQL
Back end fairly quickly. However I also want to be able to "go crazy"
and jump outside the current framework. The fs api indirection provides
more flexibility for this type of thing. The "DB access level"
indirection is very much predicated on the trail being the center of the
universe. If it is not, this layer may become obsolete for my purposes.
I have no doubt that you can make it cleaner and faster. But will it be
as flexible?
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by Branko Čibej <br...@xbc.nu>.
Karl Fogel wrote:
>Brane hasn't described his vision for replacing Berkeley transactions
>with locking yet, so it's possible that what I write here will be
>superseded by his proposal.
>
I hope so.
>However, I'm *pretty* confident that
>Subversion's current use of transactions is necessary and appropriate,
>and will try to explain why we shouldn't reduce their use.
>
>What is a Berkeley transaction?
>
>A transaction is effectively a private copy of the entire database.
>
No it's not. A transaction is _not_ invisible to other users of the
database, even when not committed. Nobody outside the transaction sees
the changes until they're committed, but everybody sees the locks that
the transaction creates -- and the transaction sees other people's locks.
[skip lots of txn-101 that Karl just invented]
>Of course, we could do it without transactions, if we implemented our
>own locking scheme. And that locking scheme would be equivalent in
>power and scope to... Berkeley transactions!
>
There's another locking scheme, and it's called Berkeley locks. You can
lock and unlock an object (a table row, if you will) without starting a
transaction (and all the logging that involves). It's just that locking
happens automatically when you touch an object from within a transaction.
>I suspect this is how transactions got invented. People realized that
>individual locking schemes could be abstracted and implemented inside
>the database itself, and then they'd never have to worry about it
>again.
>
Oh my. You've missed your vocation, you should be writing history books. :-)
>I'm not saying it wouldn't be more efficient to have our own locking
>scheme. It probably would.
>
It would be a _lot_ more efficient. Every time you create a transaction,
you end up writing to the log files. And every time you commit it, you
have to sync the logs. That's a _lot_ of fsyncs, and they're slowing us
down tremendously. What's worse, we're writing and sync'ing logs even
for read-only operations that don't change a single bit in the database.
> We know the data intimately; we can take
>shortcuts that Berkeley would never dare. But I don't think the gain
>would be very large (after all, Sleepcat works helluva hard to
>implement transactions as efficiently as they can),
>
However hard they work on making transactions efficient, they can't
avoid the fsyncs. In case you're wondering if those fsyncs are really
that bad, hear this: I hardcoded a DB_NOSYNC when opening the database
in fs.c -- and reduced "make check" time on Windows by *half*. Of
course, we can't do that for a production repository, but it makes you
think, doesn't it?
>and it would come at a high cost in complexity and new bugs.
>
Our code wouldn't become that much more complex.
>So, barring a surprisingly brilliant insight from Brane (I'm not
>saying it can't happen...), my feeling is that we should leave
>Subversion's use of BDB transactions alone, and concentrate on
>changing our checkpointing and other things.
>
>
There's no surprisingly brilliant insight here. Locking and unlocking a
DB object does not incur the logging and fsync overhead, and it's
sufficient for most read-only operations. That would speed up the reads,
but it would _also_ speed up writes on average because each
txn_checkpoint would have much less work to do.
The question now is, where can we replace txns with locks?
Quoting from your other mail:
> 'revisions':
> Well, only the revprops might be changing. I guess one wants a
> consistent picture of those. So we'd lock just the revision
> record we're reading from, for the duration of the read. During
> that time, someone changing a revprop on that revision would be
> blocked, but that wouldn't be very long, so it's okay.
>
> 'nodes', 'representations', 'strings':
> What do we lock here? Would the locking interfere with
> deltification?
>
I think we always read revision, node and representation records into
memory at once. That means we can safely move from a
start txn; read; commit txn
pattern to a
lock row; read; unlock row
pattern, without sacrificing either safety, correctness or code
complexity. We have to lock 'revisions' because of rev props, and
'representations' because of deltification, and we _might_ have to lock
'nodes' (have to think about that, it's concievable that we can skip
locking there completely).
In 'strings', we'd have to lock the record so that deltification doesn't
delete it while it's still half read. I can't remember exactly at the
moment (and haven't time to look at the code -- have to get to work),
but I seem to recall that we create a transaction for every partial read
of a string. If that's so, we can easily change to the same pattern as
for the other tables.
I can't recall if we have any operations that read from more than one
table in the same transaction. But even if we do, we can easily extend
the trail concept to locking, and all we have to be careful about is the
order of the locks.
The nice thing about this, implementation-wise, is that we can make a
/gradual/ move from reads-with-txns to reads-with-locks. Do the easy
ones first ('revisions', 'nodes', 'representations'), then move on to
the hard ones.
All of the above reasoning extends naturally to 'uuids', too.
> 'changes':
> No need to lock this for read-only operations, right?
>
>
I suspect you're right.
Brane
P.S.: All of the above assumes that we don't have "svn obliterate". Once
we have that operations, we'll definitely have to lock 'nodes' and
'changes'.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to use Berkeley transactions.
Posted by cm...@collab.net.
Karl Fogel <kf...@newton.ch.collab.net> writes:
> So, barring a surprisingly brilliant insight from Brane (I'm not
> saying it can't happen...), my feeling is that we should leave
> Subversion's use of BDB transactions alone, and concentrate on
> changing our checkpointing and other things.
I agree, but yeah, would like to hear some proposed BRilliANcE. ;-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org