You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Raymond Raymond <ra...@hotmail.com> on 2005/08/17 18:56:24 UTC
Can anyone give me some suggestions?
Hi, everyone, I am a graduate student and trying to do some research with
Derby.
I am interested in the "Autonomic checkpointing timing and log file size"
issue on the
to-do list. I would like to know is there anyone else who is interested in
that? or
anyone who can give me some suggestions or direction about that issue?
Thanks lots.
Raymond
_________________________________________________________________
Powerful Parental Controls Let your child discover the best the Internet has
to offer.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Can anyone give me some suggestions?
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> As Oystein says I think the main reason to tune checkpoint interval is
MM> some sort of balance between recovery time and performance overhead of
MM> the checkpoint. The last change in this area bumped the log file size
MM> from 100k to 1 meg, and the checkpoint size from 1 log file to 10 log
MM> files. Even this change makes us do many more checkpoints than
MM> competing databases, especially in benchmark situations when other dbs
MM> don't do any checkpoints. Some time based approach may be interesting,
MM> with inputs of how long does a checkpoint take and how long since the
MM> last checkpoint and total log volume rate. There is also the log file
MM> garbage collection issue.
It would be interesting to see a study of how different settings for
the checkpoint interval impact throughput and recovery times. That
seems like well suited for a student project.
MM> I don't think checkpointing is the way to control # dirty pages in the
MM> cache. The current system tries to do this in the clock algorithm, by
MM> having a background thread "clean" dirty pages when it thinks there
MM> are too many. This buffer cache, clock algorithm is pretty simple so
MM> there may be improvements possible there - I believe some have already
MM> been discussed on the list with respect to weighting various
MM> pages.
OK, I was not aware of this mechanism. I guess the benefit of having
two mechanisms that write pages to disk is that the clock algorithm
only writes those pages that are likely to soon be thrown of the
cache. Hence, hot spot pages are normally only written at each
checkpoint.
--
Øystein
Re: Can anyone give me some suggestions?
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "RR" == Raymond Raymond <ra...@hotmail.com> writes:
RR> Fuzzy checkpointing means checkpoints are asynchronous and occur ant the
RR> same time as new database updates are being made. The general actions
RR> in a fuzzy checkpoint are:
RR> 1. Force log buffer to disk from memory.
RR> 2. To check if there are any buffer still on the checkpoint list form
RR> the previous checkpoint.
RR> If there are any, then write them out to disk.(the list is
RR> supposed to be empty under
RR> some control).
RR> 3. All dirty buffers will be listed in the current checkpoint list.
RR> 4. Gather other useful information.
RR> Once these actions have done, the remainder of the checkpoint can also
RR> proceed in
RR> parallel with future database updates. e.t.: a write process will
RR> examine the checkpoint
RR> list periodically, write some of the buffers on the list to disk and
RR> remove thme from the
RR> list.Under some controls, it is supposed that the list will be empty
RR> just befor the next
RR> checkpoint is taken.
RR> Those are just the general processes of fuzzy checkpointing. I think
RR> we can use some
RR> technic to make it works more efficient. That is what I am trying to do.^_^.
RR> Anyone has some ideas about that?
Raymond,
What you describe is one implementation of fuzzy checkpointing. The
term fuzzy checkpoint refers to a checkpoint where the checkpointed
pages are not mutually consistent (ref. Gray&Reuter, "Transaction
Processing"). The significant part about fuzzy checkpointing is that
checkpointing occur in parallel with normal activity. Hence, one will
have to apply the log records generated in parallel with the
checkpoint in order to reach a consistent database state. According
to that definition, Derby implements fuzzy checkpointing.
I am not convinced that you will achieve much by your suggested change
to Derby's checkpointing mechanism. If I understand you correctly,
your idea is that by writing out only what was dirty before the
previous checkpoint there will be less work to do. However, this
means that you will have to do recovery from the penultimate
checkpoint. Derby, on the other hand updates the log control file
when the checkpoint has completed. Hence, it will be able to recovery
from the last completed checkpoint. This way, to support a given
recovery time, Derby can have larger checkpoint intervals than your
method. All in all, I am not convinced that the total number of
writes in your case will necessarily be less than for Derby.
--
Øystein
Re: Some other questions on Checkpointing.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Raymond Raymond wrote:
>> From: Mike Matrigali <mi...@sbcglobal.net>
>> Reply-To: "Derby Development" <de...@db.apache.org>
>> To: Derby Development <de...@db.apache.org>
>> Subject: Re: Can anyone give me some suggestions?
>> Date: Tue, 23 Aug 2005 10:03:35 -0700
>>
>> Yes, the key is that in normal operation the call to these routines
>> is always done by a background process which is a different thread
>> than normal client threads. So database client threads procede
>> while the checkpoint procedes. The buffer cache is not locked for
>> the duration of the checkpoint, only individual pages for the time
>> it takes to copy the page from the buffer cache into the OS cache.
>
>
>
> Thanks lots, I got what you said. But I still have some questions about
> that.
>
> 1. If a buffer page, we call it B1 here, is updated before a checkpoint
> is taken
> and the log of the update is generated as L1. During the following
> checkpoint,
> depending on the WAL policy, log buffer will be first forced to disk, so
> L1 will
> be flushed to disk. Then the checkpoint process will try to write the
> cache buffer
> to disk. My question is, since the checkpoint thread and the database
> client threads
> are asynchronous, it is possible another update on B1 happens before B1
> is latched and
> written out, later B1 will be written out but the log of the second
> update is not
> written out yet. How does derby ensure the WAL policy in this case?
Derby maintains in memory the last log record that changed the page.
Before writing the page it always asks the log to flush up to that page,
usually this request is a no-op as the log has already flushed up to
that point. See CachedPage.java!writePage()
// force WAL - and check to see if database is corrupt or is frozen.
// last log Instant may be null if the page is being forced
// to disk on a createPage (which violates the WAL protocol actually).
// See FileContainer.newPage
LogInstant flushLogTo = getLastLogInstant();
dataFactory.flush(flushLogTo);
>
> 2. During a checkpoint, now derby will search all the cache buffer for
> dirty pages
> and write them out. Why don't we keep a dirty page list? Just IDs that
> can identify
> the corresponding dirty page are stored in the list, so it will not take
> a lot of
> space. The first time a buffer page is updated, its ID will be appended
> to the list,
> and after the dirty page has been written out, it will be released from
> the list.
> During a checkpoint, we just search from the head of the list to the end
> and write
> the corresponding dirty pages out.
The current cache has no lists, lists tend to be a point of contention
in between users of the cache. The current cache design was mostly
picked to be correct and to be simple as possible. Walking an array
once for the checkpoint does not seem like much overhead, over
maintaining dirty page list. I guess if the usual case of a checkpoint
there were no dirty pages that would be an issue, but at least the
current checkpoints are driven by amount of log being written which in
most cases will indicate lots of dirty pages.
The tradeoff here is that performance of
a checkpoint is not that important, especially making it go faster in
anyway slows down the "real" work of the system. If I did any work on
checkpoints I would actually try to figure out how to make it slower,
ie. spread the I/O out over non-active times in the server - so that it
would have little affect on user queries. It is not as simple as
priority as what is mostly affected is I/O load of the system rather
than cpu.
>
>
>
> Raymond
>
> _________________________________________________________________
> Powerful Parental Controls Let your child discover the best the Internet
> has to offer.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>
>
Some other questions on Checkpointing.
Posted by Raymond Raymond <ra...@hotmail.com>.
>From: Mike Matrigali <mi...@sbcglobal.net>
>Reply-To: "Derby Development" <de...@db.apache.org>
>To: Derby Development <de...@db.apache.org>
>Subject: Re: Can anyone give me some suggestions?
>Date: Tue, 23 Aug 2005 10:03:35 -0700
>
>Yes, the key is that in normal operation the call to these routines
>is always done by a background process which is a different thread
>than normal client threads. So database client threads procede
>while the checkpoint procedes. The buffer cache is not locked for
>the duration of the checkpoint, only individual pages for the time
>it takes to copy the page from the buffer cache into the OS cache.
Thanks lots, I got what you said. But I still have some questions about
that.
1. If a buffer page, we call it B1 here, is updated before a checkpoint is
taken
and the log of the update is generated as L1. During the following
checkpoint,
depending on the WAL policy, log buffer will be first forced to disk, so L1
will
be flushed to disk. Then the checkpoint process will try to write the cache
buffer
to disk. My question is, since the checkpoint thread and the database client
threads
are asynchronous, it is possible another update on B1 happens before B1 is
latched and
written out, later B1 will be written out but the log of the second update
is not
written out yet. How does derby ensure the WAL policy in this case?
2. During a checkpoint, now derby will search all the cache buffer for dirty
pages
and write them out. Why don't we keep a dirty page list? Just IDs that can
identify
the corresponding dirty page are stored in the list, so it will not take a
lot of
space. The first time a buffer page is updated, its ID will be appended to
the list,
and after the dirty page has been written out, it will be released from the
list.
During a checkpoint, we just search from the head of the list to the end and
write
the corresponding dirty pages out.
Raymond
_________________________________________________________________
Powerful Parental Controls Let your child discover the best the Internet has
to offer.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Can anyone give me some suggestions?
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Yes, the key is that in normal operation the call to these routines
is always done by a background process which is a different thread
than normal client threads. So database client threads procede
while the checkpoint procedes. The buffer cache is not locked for
the duration of the checkpoint, only individual pages for the time
it takes to copy the page from the buffer cache into the OS cache.
Raymond Raymond wrote:
>> From: Mike Matrigali <mi...@sbcglobal.net>
>> Reply-To: "Derby Development" <de...@db.apache.org>
>> Ok, just to be clear Derby checkpoints are already asynchronous.
>> The one synchrounous point is that while a page is being written
>> no update that exact page can be happening.
>> Updates to all other pages in the
>> database may be going on concurrently with this operation. So unless
>> you are worried about concurrency on the single page, I believe
>> Derby checkpoints already do what you describe.
>>
>> In the case of Derby the list is the numerical order list of buffers
>> in the cache, and is always traversed in order. The checkpoint is
>> done by a background thread asynchronous to other user threads.
>>
>
>
> hi, Dear Mike Matrigali ,thanks for your comment. I am a novice of derby.
> I still have some questions about this issue. I looked
> into the present derby source code. I read the source
> of org.apache.derby.impl.store.raw.log.LogToFile.checkpoint()
> and org.apache.derby.impl.store.raw.log.LogToFile.checkpointWithTran(),
> I found that it clean the buffer cache during the checkpoint by calling
> df.checkpoint() (df is an instance of DataFactory)
> I also looked into the source code of
> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.checkpoint()
> and org.apache.derby.impl.services.cache.Clock.cleanAll()
> and org.apache.derby.impl.services.cache.Clock.cleanCache()
> which are called nested to write out the dirty pages.
> So, I think, presently in derby,when an checkpoint is being taken, it will
> wait for all the dirty pages being written out and then write out the
> checkpoint
> log and any other things needed to be done.
>
>
>
>
> Raymond
>
> _________________________________________________________________
> Take charge with a pop-up guard built on patented Microsoft® SmartScreen
> Technology.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>
Re: Re: Can anyone give me some suggestions?
Posted by Raymond Raymond <ra...@hotmail.com>.
>From: Mike Matrigali <mi...@sbcglobal.net>
>Reply-To: "Derby Development" <de...@db.apache.org>
>Ok, just to be clear Derby checkpoints are already asynchronous.
>The one synchrounous point is that while a page is being written
>no update that exact page can be happening.
>Updates to all other pages in the
>database may be going on concurrently with this operation. So unless
>you are worried about concurrency on the single page, I believe
>Derby checkpoints already do what you describe.
>
>In the case of Derby the list is the numerical order list of buffers
>in the cache, and is always traversed in order. The checkpoint is
>done by a background thread asynchronous to other user threads.
>
hi, Dear Mike Matrigali ,thanks for your comment. I am a novice of derby.
I still have some questions about this issue. I looked
into the present derby source code. I read the source
of org.apache.derby.impl.store.raw.log.LogToFile.checkpoint()
and org.apache.derby.impl.store.raw.log.LogToFile.checkpointWithTran(),
I found that it clean the buffer cache during the checkpoint by calling
df.checkpoint() (df is an instance of DataFactory)
I also looked into the source code of
org.apache.derby.impl.store.raw.data.BaseDataFileFactory.checkpoint()
and org.apache.derby.impl.services.cache.Clock.cleanAll()
and org.apache.derby.impl.services.cache.Clock.cleanCache()
which are called nested to write out the dirty pages.
So, I think, presently in derby,when an checkpoint is being taken, it will
wait for all the dirty pages being written out and then write out the
checkpoint
log and any other things needed to be done.
Raymond
_________________________________________________________________
Take charge with a pop-up guard built on patented MicrosoftŽ SmartScreen
Technology.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Can anyone give me some suggestions?
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Ok, just to be clear Derby checkpoints are already asynchronous.
The one synchrounous point is that while a page is being written
no update that exact page can be happening.
Updates to all other pages in the
database may be going on concurrently with this operation. So unless
you are worried about concurrency on the single page, I believe
Derby checkpoints already do what you describe.
In the case of Derby the list is the numerical order list of buffers
in the cache, and is always traversed in order. The checkpoint is
done by a background thread asynchronous to other user threads.
Raymond Raymond wrote:
>
>
>> From: Mike Matrigali <mi...@sbcglobal.net>
>> Reply-To: "Derby Development" <de...@db.apache.org>
>>
>> Could you define "fuzzy" checkpoint or site what update interference
>> you are trying to avoid.
>>
>
> Fuzzy checkpointing means checkpoints are asynchronous and occur ant the
> same time as new database updates are being made. The general actions
> in a fuzzy checkpoint are:
> 1. Force log buffer to disk from memory.
> 2. To check if there are any buffer still on the checkpoint list form
> the previous checkpoint.
> If there are any, then write them out to disk.(the list is supposed
> to be empty under
> some control).
> 3. All dirty buffers will be listed in the current checkpoint list.
> 4. Gather other useful information.
> Once these actions have done, the remainder of the checkpoint can also
> proceed in
> parallel with future database updates. e.t.: a write process will
> examine the checkpoint
> list periodically, write some of the buffers on the list to disk and
> remove thme from the
> list.Under some controls, it is supposed that the list will be empty
> just befor the next
> checkpoint is taken.
>
> Those are just the general processes of fuzzy checkpointing. I think we
> can use some
> technic to make it works more efficient. That is what I am trying to
> do.^_^.
> Anyone has some ideas about that?
>
>
>
>
> Raymond
>
> _________________________________________________________________
> Scan and help eliminate destructive viruses from your inbound and
> outbound e-mail and attachments.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>
>
Re: Re: Can anyone give me some suggestions?
Posted by Raymond Raymond <ra...@hotmail.com>.
>From: Mike Matrigali <mi...@sbcglobal.net>
>Reply-To: "Derby Development" <de...@db.apache.org>
>
>Could you define "fuzzy" checkpoint or site what update interference
>you are trying to avoid.
>
Fuzzy checkpointing means checkpoints are asynchronous and occur ant the
same time as new database updates are being made. The general actions
in a fuzzy checkpoint are:
1. Force log buffer to disk from memory.
2. To check if there are any buffer still on the checkpoint list form the
previous checkpoint.
If there are any, then write them out to disk.(the list is supposed to
be empty under
some control).
3. All dirty buffers will be listed in the current checkpoint list.
4. Gather other useful information.
Once these actions have done, the remainder of the checkpoint can also
proceed in
parallel with future database updates. e.t.: a write process will examine
the checkpoint
list periodically, write some of the buffers on the list to disk and remove
thme from the
list.Under some controls, it is supposed that the list will be empty just
befor the next
checkpoint is taken.
Those are just the general processes of fuzzy checkpointing. I think we can
use some
technic to make it works more efficient. That is what I am trying to do.^_^.
Anyone has some ideas about that?
Raymond
_________________________________________________________________
Scan and help eliminate destructive viruses from your inbound and outbound
e-mail and attachments.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Can anyone give me some suggestions?
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Could you define "fuzzy" checkpoint or site what update interference
you are trying to avoid.
Derby does the following (suresh let me know if I am off here):
o loops once around the page cache and queues async writes of
of every dirty page. Note that on all the JVM's/OS's I have
looked at this write just copies the data to OS buffer so is
fast - thus the latch on the page is not held very long.
o At end syncs all writes to each table affected using single jvm
call per file. These syncs may take awhile, but since they
happen at the end, many if not all of the writes have already
hit disk so usually there is nothing to wait for.
o logs the checkpoint record
As Oystein says I think the main reason to tune checkpoint interval is
some sort of balance between recovery time and performance overhead of
the checkpoint. The last change in this area bumped the log file size
from 100k to 1 meg, and the checkpoint size from 1 log file to 10 log
files. Even this change makes us do many more checkpoints than
competing databases, especially in benchmark situations when other dbs
don't do any checkpoints. Some time based approach may be interesting,
with inputs of how long does a checkpoint take and how long since the
last checkpoint and total log volume rate. There is also the log file
garbage collection issue.
I don't think checkpointing is the way to control # dirty pages in the
cache. The current system tries to do this in the clock algorithm, by
having a background thread "clean" dirty pages when it thinks there
are too many. This buffer cache, clock algorithm is pretty simple so
there may be improvements possible there - I believe some have already
been discussed on the list with respect to weighting various pages. It
is also interesting in that it should be possible to create a separate
implementation of it as a separate module and one could pick an choose
and easily compare performance differences. Again the interesting
problem is to maintain as many dirty pages as possible while while
maintaining enough clean pages such that a user transaction never has
to wait for a write to get a buffer.
Note that log switching and checkpoints are 2 orthogonal issues.
Historically the code has done checkpoints at the same time as
log switches, but there is not a requirement. The reason was mostly
that log garbage collection is done at the log file level in the
current log implementation. Be careful making log files too big, as
there are some JVM's/OS's where the sync performance is linearly
dependent on the size of the file. I don't believe this is true
of the default syncing used on the log file, but is true in the
pre-1.4.1 log file sync implementation.
Raymond Raymond wrote:
> Oystein.Grovlen,Thanks your for your suggestions. It is exactly what I
> am thinking
> about.I am considering two aspects of the checkpointing issue.
> 1. How to make the engine tune the interval of checkpointing by itsself.
> I think
> It depends on the database buffer size, log buffer size and how many
> dirty pages in the
> database buffer. And you give me a good suggestion about the machine
> performance
> factor. I will take that into account.
> 2. Although the derby implemeted ARIES algorithsm in its recovery
> function, it did not
> adopt fuzzy checkpointing. The current checkpointing approach is not
> very efficient, just
> as what you said, it will interfere with updates requires from other
> transactions. I am
> trying to find a better way to do that.
>
> Anyone else has any good ideas about that?^_^.
>
>
>
> Raymond
>
>
>> From: Oystein.Grovlen@Sun.COM (Øystein Grøvlen)
>>
>> Currently, you can configure the checkpoint interval and log file size
>> of Derby by setting the properties:
>>
>> derby.storage.logSwitchInterval (default 1 MB)
>> derby.storage.checkpointInterval (default 10 MB)
>>
>> (None of these seems to be documented in the manuals, and the JavaDoc
>> for LogFactory.recover() gives wrong (out-dated?) defaults).
>>
>> This means that by default all log files will be 1 MB, and a checkpoint
>> is made for every tenth log file.
>>
>> In order to know when it is useful to change the defaults, one has to
>> consider the purpose of a checkpoint:
>>
>> 1) Reduced recovery times. Only log create after the penultimate
>> checkpoint needs to be redone at recovery. This also means that
>> older log files may be garbage-collected (as long as they do not
>> contain log records for transactions that are still not
>> terminated.)
>>
>> To get short recovery times, one should keep the checkpoint
>> interval low. The trade-off is that frequent checkpoints will
>> increase I/O since you will have less updates to the same page
>> between two checkpoints. Hence, you will get more I/O per
>> database operation.
>>
>> 2) Flush dirty pages to disk. A checkpoint is a much more efficient
>> way to clean dirty pages in the db cache than to do it on demand
>> on a single page when one need to replace it with another.
>> Hence, one should make sure to do checkpoints often enough to
>> avoid that the whole cache is dirty.
>>
>> Based on 2), one could initiate a new checkpoint when many pages in
>> the cache are dirty (e.g., 50% of the pages) and postpone a checkpoint
>> if few pages are dirty. The difficult part would be to determine how
>> long checkpoint intervals is acceptable with respect to impact on
>> recovery times.
>>
>> I guess one could argue that for recovery times, it is the clock time
>> that matters. Hence, one could automatically increase the value of
>> derby.storage.checkpointInterval on more performant computers since it
>> will be able to process more log per time unit.
>>
>> When would want to change the log switch interval? I think few would
>> care, but since the log files per default are preallocated, space will
>> be wasted if operations that perform a log switch (e.g., backup) is
>> performed when the current log file is nearly empty. On the other
>> hand, a small log file size will result many concurrent log files if
>> the checkpoint interval is very large.
>>
>> Hope this helps a little,
>>
>> --
>> Øystein
>>
>
> _________________________________________________________________
> Scan and help eliminate destructive viruses from your inbound and
> outbound e-mail and attachments.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>
Re: Re: Can anyone give me some suggestions?
Posted by Raymond Raymond <ra...@hotmail.com>.
Oystein.Grovlen,Thanks your for your suggestions. It is exactly what I am
thinking
about.I am considering two aspects of the checkpointing issue.
1. How to make the engine tune the interval of checkpointing by itsself. I
think
It depends on the database buffer size, log buffer size and how many dirty
pages in the
database buffer. And you give me a good suggestion about the machine
performance
factor. I will take that into account.
2. Although the derby implemeted ARIES algorithsm in its recovery function,
it did not
adopt fuzzy checkpointing. The current checkpointing approach is not very
efficient, just
as what you said, it will interfere with updates requires from other
transactions. I am
trying to find a better way to do that.
Anyone else has any good ideas about that?^_^.
Raymond
>From: Oystein.Grovlen@Sun.COM (Řystein Grřvlen)
>
>Currently, you can configure the checkpoint interval and log file size
>of Derby by setting the properties:
>
>derby.storage.logSwitchInterval (default 1 MB)
>derby.storage.checkpointInterval (default 10 MB)
>
>(None of these seems to be documented in the manuals, and the JavaDoc
>for LogFactory.recover() gives wrong (out-dated?) defaults).
>
>This means that by default all log files will be 1 MB, and a checkpoint
>is made for every tenth log file.
>
>In order to know when it is useful to change the defaults, one has to
>consider the purpose of a checkpoint:
>
> 1) Reduced recovery times. Only log create after the penultimate
> checkpoint needs to be redone at recovery. This also means that
> older log files may be garbage-collected (as long as they do not
> contain log records for transactions that are still not
> terminated.)
>
> To get short recovery times, one should keep the checkpoint
> interval low. The trade-off is that frequent checkpoints will
> increase I/O since you will have less updates to the same page
> between two checkpoints. Hence, you will get more I/O per
> database operation.
>
> 2) Flush dirty pages to disk. A checkpoint is a much more efficient
> way to clean dirty pages in the db cache than to do it on demand
> on a single page when one need to replace it with another.
> Hence, one should make sure to do checkpoints often enough to
> avoid that the whole cache is dirty.
>
>Based on 2), one could initiate a new checkpoint when many pages in
>the cache are dirty (e.g., 50% of the pages) and postpone a checkpoint
>if few pages are dirty. The difficult part would be to determine how
>long checkpoint intervals is acceptable with respect to impact on
>recovery times.
>
>I guess one could argue that for recovery times, it is the clock time
>that matters. Hence, one could automatically increase the value of
>derby.storage.checkpointInterval on more performant computers since it
>will be able to process more log per time unit.
>
>When would want to change the log switch interval? I think few would
>care, but since the log files per default are preallocated, space will
>be wasted if operations that perform a log switch (e.g., backup) is
>performed when the current log file is nearly empty. On the other
>hand, a small log file size will result many concurrent log files if
>the checkpoint interval is very large.
>
>Hope this helps a little,
>
>--
>Řystein
>
_________________________________________________________________
Scan and help eliminate destructive viruses from your inbound and outbound
e-mail and attachments.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Can anyone give me some suggestions?
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "RR" == Raymond Raymond <ra...@hotmail.com> writes:
RR> Hi, everyone, I am a graduate student and trying to do some research
RR> with Derby.
RR> I am interested in the "Autonomic checkpointing timing and log file
RR> size" issue on the
RR> to-do list. I would like to know is there anyone else who is
RR> interested in that? or
RR> anyone who can give me some suggestions or direction about that issue?
Currently, you can configure the checkpoint interval and log file size
of Derby by setting the properties:
derby.storage.logSwitchInterval (default 1 MB)
derby.storage.checkpointInterval (default 10 MB)
(None of these seems to be documented in the manuals, and the JavaDoc
for LogFactory.recover() gives wrong (out-dated?) defaults).
This means that by default all log files will be 1 MB, and a checkpoint
is made for every tenth log file.
In order to know when it is useful to change the defaults, one has to
consider the purpose of a checkpoint:
1) Reduced recovery times. Only log create after the penultimate
checkpoint needs to be redone at recovery. This also means that
older log files may be garbage-collected (as long as they do not
contain log records for transactions that are still not
terminated.)
To get short recovery times, one should keep the checkpoint
interval low. The trade-off is that frequent checkpoints will
increase I/O since you will have less updates to the same page
between two checkpoints. Hence, you will get more I/O per
database operation.
2) Flush dirty pages to disk. A checkpoint is a much more efficient
way to clean dirty pages in the db cache than to do it on demand
on a single page when one need to replace it with another.
Hence, one should make sure to do checkpoints often enough to
avoid that the whole cache is dirty.
Based on 2), one could initiate a new checkpoint when many pages in
the cache are dirty (e.g., 50% of the pages) and postpone a checkpoint
if few pages are dirty. The difficult part would be to determine how
long checkpoint intervals is acceptable with respect to impact on
recovery times.
I guess one could argue that for recovery times, it is the clock time
that matters. Hence, one could automatically increase the value of
derby.storage.checkpointInterval on more performant computers since it
will be able to process more log per time unit.
When would want to change the log switch interval? I think few would
care, but since the log files per default are preallocated, space will
be wasted if operations that perform a log switch (e.g., backup) is
performed when the current log file is nearly empty. On the other
hand, a small log file size will result many concurrent log files if
the checkpoint interval is very large.
Hope this helps a little,
--
Øystein