You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Karl Fogel <kf...@newton.ch.collab.net> on 2003/02/21 01:43:54 UTC
When to call txn_checkpoint().
I've created a new thread for this, to keep it separate from the
question of whether to reduce our use of transactions (I'll send
another mail about that in a moment).
Any solution that depends on people remembering to run Berkeley's
db_checkpoint utility, or on running hot-backup.py, is probably asking
for trouble. However, I don't think we need to do that. We can
control our checkpointing granularity easily enough internally. Here
are two ways we might do that.
Solution 1:
===========
Right now, we only call txn_checkpoint() in two places in Subversion:
libsvn_fs/trail.c:commit_trail()
libsvn_fs/fs.c:cleanup_fs()
The former happens when a BDB transaction is closed, that is, whenever
svn_fs__retry_txn() succeeds in running the txn_body func. Remember
that the "commit" in "commit_trail" is *not* talking about Subversion
commits; it's about any operation involving a Berkeley transaction,
which currently includes virtually everything. Note also that this
call to txn_checkpoint is often a no-op:
fs->env->txn_checkpoint(fs->env, 1024, 5, 0)
Those parameters mean that txn_checkpoint() will only do something if
more than 1024 kbytes of new log data, or 5 minutes, have passed since
the last checkpoint that did something. (The last parameter, 0, is
just a flag mask that we can ignore for this discussion.)
The other call, in fs.c, is stricter:
env->txn_checkpoint(env, 0, 0, 0)
(This is the one that Brandon patched to say 8000 kb and 60 minutes
instead.)
I frankly think that call can just go away. Everything we do, we do
in transactions, so the first checkpoint call will get crossed plenty
often. This second call is redundant; I don't even know why it's
there. (Again, whether or not our use of BDB transactions will be
reduced will be addressed in a separate thread.)
The remaining question, then, is whether to tune the parameters of the
first call. I doubt we need to, but we'll see. If we do, we should
make them be parameters in some sort of repository config file, so the
repository admin can tweak them easily. (I don't think there's any
way to use db/DB_CONFIG for this, unfortunately, since they're
obviously just regular function parameters as far as BDB is concerned.
So we'd have to use a new file.)
So, would anyone object to my removing the second call? (Yes, I'm
volunteering to do the change, run stress.pl and the test suite, etc
:-). )
Solution 2:
===========
This was Greg Stein's idea during a phone call. It's possible he
didn't know at the time what txn_checkpoint's parameters do, and might
well prefer Solution 1 himself (I know I do, now that I've taken a
closer look at how txn_checkpoint works).
The idea is to move the txn_checkpoint() calls out of the regular FS
code paths entirely, and instead make it a public API function of the
Berkeley back end:
svn_error_t *svn_fs_berkeley_checkpoint (const char *path,
apr_pool_t *pool);
Each RA layer would be responsible for calling the function "at the
appropriate time", in practice, when an operation completes and/or a
connection shuts down. For example, from inside the FS code, it can
be hard to tell when an update begins and ends, because it just looks
like a series of svn_fs_foo() calls. But the caller -- the RA layer
-- can say when the operation is over, and run checkpoint based on
that.
In a sense, this is applying our APR pool management strategy to
Berkeley logfiles / memory pools :-).
I'll also volunteer to do this, if that's what we settle on. But
right now I prefer Solution 1 (possibly with tuneable params for the
remaining call). Solution 1 retains the ability to tune how often
checkpointing really happens, while placing no new burden on RA
layers.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Branko Čibej <br...@xbc.nu> writes:
> > OH, I know just trying to couch my opinion a little:-) But I can't +1
> > -1 right?
>
> Of course you can, keeping in mind that any veto (-1) needs technical
> argumentation. At least that's how I've always understood our unwritten
> policy.
Any technical veto will be listened to, whether from a committer or
not. *Formally* the procedure only applies to full committers, yes,
but that's not a formality that we've really invoked much, as things
have worked out quite fine with informal list discussion.
(I'm just pointing it out in case it should ever be an issue later. A
voting policy where the only qualification for joining the electorate
is the ability to sign up on a mailing list would be a pretty absurd
result, one must admit :-) ).
These rules aren't as unwritten as one might think, as we just adopted
the Apache guidelines:
http://httpd.apache.org/dev/guidelines.html
However, it's true we haven't been following them very closely, and
probably won't unless some conflict forces us to :-).
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Branko Čibej <br...@xbc.nu>.
Glenn A. Thompson wrote:
>>
>>
>> Who said you couldn't vote? Of course you can vote! You may not be a
>> committer, but technical discussions on this list have never been
>> limited to committers.
>>
> Discuss yes, vote no.
> OH, I know just trying to couch my opinion a little:-) But I can't +1
> -1 right?
Of course you can, keeping in mind that any veto (-1) needs technical
argumentation. At least that's how I've always understood our unwritten
policy.
> I thought there was a formal voting process for committers only.
There is, but that's only about giving (or taking away) commit access,
which is not a technical decision. And it's not very formal, either --
I've never seen a quorum count yet. :-)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>Who said you couldn't vote? Of course you can vote! You may not be a
>committer, but technical discussions on this list have never been
>limited to committers.
>
Discuss yes, vote no.
OH, I know just trying to couch my opinion a little:-)
But I can't +1 -1 right?
I thought there was a formal voting process for committers only.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Branko Čibej <br...@xbc.nu>.
Glenn A. Thompson wrote:
> Hey,
>
>> Solution 2:
>> ===========
>>
>>
>>
> I know I can't vote here
Who said you couldn't vote? Of course you can vote! You may not be a
committer, but technical discussions on this list have never been
limited to committers.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
Hey,
>Solution 2:
>===========
>
>
>
I know I can't vote here but I'm really not in favor of RA layers having
this much FS knowledge.
Unless it was done in a "very" generic way like a generic cleanup or
shutdown type routine.
The main goal of my "yet to be re-synced patch" was to eliminate all
specific DB knowledge.
I was very comfortable with where I was at and heading. It was just too
big for anyone else to get comfortable with:-) Well CMike seemed fairly
comfortable but didn't care for the mega patch version:-)
I believe that any "tricky-ness" required to keep it hidden is well
worth the effort. If it goes beyond the scope of what you "Karl" want
to do at this time, I'll make it part of my short term goals. If
another BDB only call were to creep into the FS api I would make it my
mission to get rid of it anyway.
I'm just a couple days from jumping completely out of my tunnel into the
light. At that point I'm going to be dedicating the majority of my time
to Subversion for a month or two or *three* if I can get away with it.
IOW I like removing the second call only for now.
Later,
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Brandon Ehle <az...@yahoo.com> writes:
> We might also need some way of detecting duplicate txn_checkpoint()'s
> so that different ra layers aren't attempting to checkpoint at the
> same time like if you are using mod_dav_svn and svnserve -d at the
> same time. I'm not entirely sure how svnserve works in tunnel mode,
> but I suspect that many processes could be active?
Why is it bad if two ra layers attempt to checkpoint at the same time?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Brandon Ehle <az...@yahoo.com>.
>
>
>Each RA layer would be responsible for calling the function "at the
>appropriate time", in practice, when an operation completes and/or a
>connection shuts down. For example, from inside the FS code, it can
>be hard to tell when an update begins and ends, because it just looks
>like a series of svn_fs_foo() calls. But the caller -- the RA layer
>-- can say when the operation is over, and run checkpoint based on
>that.
>
>
We might also need some way of detecting duplicate txn_checkpoint()'s so
that different ra layers aren't attempting to checkpoint at the same
time like if you are using mod_dav_svn and svnserve -d at the same
time. I'm not entirely sure how svnserve works in tunnel mode, but I
suspect that many processes could be active?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Feb 21, 2003 at 06:44:39AM +0100, Branko Cibej wrote:
> cmpilato@collab.net wrote:
> >Karl Fogel <kf...@newton.ch.collab.net> writes:
> >>So, would anyone object to my removing the second call? (Yes, I'm
> >>volunteering to do the change, run stress.pl and the test suite, etc
> >>:-). )
> >
> >That's fine. Go for it.
Yup, sounds like a great plan!
It might be helpful to go ahead and increase the values a bit. We increase
the default lock counts and stuff, too, if I recall correctly. It would be
"even nicer" to have admin-tunable, but the code defaults can help out.
> >>This was Greg Stein's idea during a phone call. It's possible he
> >>didn't know at the time what txn_checkpoint's parameters do, and might
> >>well prefer Solution 1 himself (I know I do, now that I've taken a
> >>closer look at how txn_checkpoint works).
Had no idea they were tunable. I figured it was binary: call it to
checkpoint, don't call to not checkpoint. With those params in there? Hoo
yah! Great.
> >>The idea is to move the txn_checkpoint() calls out of the regular FS
> >>code paths entirely, and instead make it a public API function of the
> >>Berkeley back end:
> >>
> >> svn_error_t *svn_fs_berkeley_checkpoint (const char *path,
> >> apr_pool_t *pool);
> >
> >-1, absolutely not. The longterm goal here is to *lose*
> >database-specific public API calls.
> >
> >
> I coudn't agree more. What a horrible idea!
It is a fine idea IFF txn_checkpoint didn't have the params. Exposing that
function gives the application a way to better control the times that you
want to spend the effort to do the checkpoint. The other alternative to the
API would have been some kind of cron job (even worse!) or sticking with
some kind of fine-grained happens-way-too-often checkpointing.
But given the params? Heck ya. Forget the new function.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Branko Čibej <br...@xbc.nu> writes:
> >> svn_error_t *svn_fs_berkeley_checkpoint (const char *path,
> >
> >-1, absolutely not. The longterm goal here is to *lose*
> >database-specific public API calls.
>
> I coudn't agree more. What a horrible idea!
:-) Keep calm, I wasn't advocating it either.
However, if (by some horrible process of winnowing) it comes down to a
choice between this versus putting the checkpointing into
hot-backup.py or some other external process, then I'd certainly
prefer that this API be available, and that our RA layers call it. We
have to keep repository maintenance stuff as internal and automatic as
possible; the less dependent on external processes and doc-reading
admins we are, the better.
Hopefully we won't have to do either, though, because Solution 1 will
solve all our problems. Now if only the U.N. would approve it.
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:
>Karl Fogel <kf...@newton.ch.collab.net> writes:
>
>
>
>>So, would anyone object to my removing the second call? (Yes, I'm
>>volunteering to do the change, run stress.pl and the test suite, etc
>>:-). )
>>
>>
>
>That's fine. Go for it.
>
>
>
>>This was Greg Stein's idea during a phone call. It's possible he
>>didn't know at the time what txn_checkpoint's parameters do, and might
>>well prefer Solution 1 himself (I know I do, now that I've taken a
>>closer look at how txn_checkpoint works).
>>
>>The idea is to move the txn_checkpoint() calls out of the regular FS
>>code paths entirely, and instead make it a public API function of the
>>Berkeley back end:
>>
>> svn_error_t *svn_fs_berkeley_checkpoint (const char *path,
>> apr_pool_t *pool);
>>
>>
>
>-1, absolutely not. The longterm goal here is to *lose*
>database-specific public API calls.
>
>
I coudn't agree more. What a horrible idea!
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
>
>
>-1, absolutely not. The longterm goal here is to *lose*
>database-specific public API calls.
>
>
>
Yeah! What he said:-)
Boy! I'm starting to get stoked again. Thanks guys!
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by cm...@collab.net.
Karl Fogel <kf...@newton.ch.collab.net> writes:
> So, would anyone object to my removing the second call? (Yes, I'm
> volunteering to do the change, run stress.pl and the test suite, etc
> :-). )
That's fine. Go for it.
> This was Greg Stein's idea during a phone call. It's possible he
> didn't know at the time what txn_checkpoint's parameters do, and might
> well prefer Solution 1 himself (I know I do, now that I've taken a
> closer look at how txn_checkpoint works).
>
> The idea is to move the txn_checkpoint() calls out of the regular FS
> code paths entirely, and instead make it a public API function of the
> Berkeley back end:
>
> svn_error_t *svn_fs_berkeley_checkpoint (const char *path,
> apr_pool_t *pool);
-1, absolutely not. The longterm goal here is to *lose*
database-specific public API calls.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: When to call txn_checkpoint().
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On Thursday, February 20, 2003, at 08:43 PM, Karl Fogel wrote:
> So, would anyone object to my removing the second call? (Yes, I'm
> volunteering to do the change, run stress.pl and the test suite, etc
> :-). )
removing the second call makes sense to me.
> Each RA layer would be responsible for calling the function "at the
> appropriate time", in practice, when an operation completes and/or a
> connection shuts down. For example, from inside the FS code, it can
> be hard to tell when an update begins and ends, because it just looks
> like a series of svn_fs_foo() calls. But the caller -- the RA layer
> -- can say when the operation is over, and run checkpoint based on
> that.
this seems to be letting berkeley db specific operations further into
our filesystem code, which doesn't seem like a good idea if we want to
be able to drop something else in at some point.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org