You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Raymond Raymond <ra...@hotmail.com> on 2005/10/11 04:55:59 UTC
Some idea about checkpoint issue, welcome to give your idea
I have been thinking of the automatic checkpointing issue
recently.I also find someone added another issue about "Use
of idle time for background checkpoint" into the to-do list.
I think we can consider these two issue together. I have
some idea about it.
Instead of doing checkpoint periodically and trying to tune the
checkpoint interval to achieve best performance, is it possible to
keep the background checkpoint process running to do checkpoint,
and the DBMS can tune the rate of checkpoint depending on the
current system situation,e.g. if the system is busy, derby will
slow down the checkpoint rate and if the system is not busy(idle),
derby will speed up the checkpoint rate.We will update the control
file periodically to let the DBMS know up to where we did checkpoint.
Maybe we can call it 'increamental checkpointing'. In my opinion,
this approach can use the disk IO resources with reason if we can
decide the checkpoint rate reasonablly.
I would like to discuss this issue with everyone. I am not
sure if this approach is doable or not. If it is doable, I will
have some further questions about how to decide the appropriate
checkpoint rate.
Thanks.
Yours, Raymond
_________________________________________________________________
Designer Mail isn't just fun to send, it's fun to receive. Use special
stationery, fonts and colors.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Some idea about checkpoint issue, welcome to give your idea
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Raymond Raymond wrote:
> I have been thinking of the automatic checkpointing issue
> recently.I also find someone added another issue about "Use
> of idle time for background checkpoint" into the to-do list.
> I think we can consider these two issue together. I have
> some idea about it.
>
> Instead of doing checkpoint periodically and trying to tune the
> checkpoint interval to achieve best performance, is it possible to
> keep the background checkpoint process running to do checkpoint,
> and the DBMS can tune the rate of checkpoint depending on the
> current system situation,e.g. if the system is busy, derby will
> slow down the checkpoint rate and if the system is not busy(idle),
> derby will speed up the checkpoint rate.We will update the control
> file periodically to let the DBMS know up to where we did checkpoint.
> Maybe we can call it 'increamental checkpointing'. In my opinion,
> this approach can use the disk IO resources with reason if we can
> decide the checkpoint rate reasonablly.
>
> I would like to discuss this issue with everyone. I am not
> sure if this approach is doable or not. If it is doable, I will
> have some further questions about how to decide the appropriate
> checkpoint rate.
Sounds like an interesting project, and I think it's doable, it's just a
matter of coding! :-)
One thing I think you want to consider up front is what problem are you
trying to solve, or what are you trying to improve or simply why is this
a good idea. Having a clear goal can make implementation decisions clearer.
Dan.
Re: how does derby avoid eating up all the system resources if it's
used embeddedly
Posted by Mike Matrigali <mi...@sbcglobal.net>.
My question was in reference to whether derby should schedule additional
background house keeping chores.
With respect to directly executing queries that have been requested by
the application Derby will attempt to use all resources available under
the following restrictions (currently):
o a single connection will only ever execute within a single thread
at one time. So a single user can never take over more than a
single cpu.
o There is currently a single background thread to do work like
reclaim committed deleted space, again currently this can at most
use a single CPU.
Raymond Raymond wrote:
> Hi, Dear Mike, since you said "Are there any opinions out there on how
> to determine
> if Derby "is busy"? " and it is not good for derby to eat up all the
> system resources,
> I am curiously want to know presently how does derby avoid eating up all
> the system
> resources if it can't dertermine it's busy or not?
>
> Thanks.
>
> Raymond
>
>
>> From: Mike Matrigali <mi...@sbcglobal.net>
>> Reply-To: "Derby Development" <de...@db.apache.org>
>> To: Derby Development <de...@db.apache.org>
>> Subject: Re: Some idea about checkpoint issue, welcome to give your idea
>> Date: Tue, 11 Oct 2005 10:57:08 -0700
>>
>> Are there any opinions out there on how to determine if
>> Derby "is busy"? Is there something better than just having
>> a low priority thread and maybe some query of cpu vs. elapsed
>> time?
>>
>> The first problem is that I don't think there are great tools
>> for this in java. The second problem is that often Derby is
>> meant to be embedded as part of another application, so we have
>> to be careful not to implement a standard server based approach
>> where it is appropriate for the "server" to use up all resources
>> available (ie. idle time may not really be best used by derby
>> admin processes).
>>
>> I have not come up with a good answer to this problem, there are
>> a number of things derby could do if it knew it had idle time
>> available for it's use. Best I have come up with is some mode
>> in the system that needs to be set by the application which
>> starts up Derby - either derby try's to limit it's use of idle
>> cycles or it enabled to try and schedule work during idle time.
>>
>> Raymond Raymond wrote:
>>
>> > I have been thinking of the automatic checkpointing issue
>> > recently.I also find someone added another issue about "Use
>> > of idle time for background checkpoint" into the to-do list.
>> > I think we can consider these two issue together. I have
>> > some idea about it.
>> >
>> > Instead of doing checkpoint periodically and trying to tune the
>> > checkpoint interval to achieve best performance, is it possible to
>> > keep the background checkpoint process running to do checkpoint,
>> > and the DBMS can tune the rate of checkpoint depending on the
>> > current system situation,e.g. if the system is busy, derby will
>> > slow down the checkpoint rate and if the system is not busy(idle),
>> > derby will speed up the checkpoint rate.We will update the control
>> > file periodically to let the DBMS know up to where we did checkpoint.
>> > Maybe we can call it 'increamental checkpointing'. In my opinion,
>> > this approach can use the disk IO resources with reason if we can
>> > decide the checkpoint rate reasonablly.
>> >
>> > I would like to discuss this issue with everyone. I am not
>> > sure if this approach is doable or not. If it is doable, I will
>> > have some further questions about how to decide the appropriate
>> > checkpoint rate.
>> >
>> > Thanks.
>> >
>> > Yours, Raymond
>> >
>> > _________________________________________________________________
>> > Designer Mail isn't just fun to send, it's fun to receive. Use special
>> > stationery, fonts and colors.
>> >
>> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
>>
>> > Start enjoying all the benefits of MSN?Premium right now and get the
>> > first two months FREE*.
>> >
>> >
>
>
> _________________________________________________________________
> Take charge with a pop-up guard built on patented Microsoft® SmartScreen
> Technology
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>
>
how does derby avoid eating up all the system resources if it's used embeddedly
Posted by Raymond Raymond <ra...@hotmail.com>.
Hi, Dear Mike, since you said "Are there any opinions out there on how to
determine
if Derby "is busy"? " and it is not good for derby to eat up all the system
resources,
I am curiously want to know presently how does derby avoid eating up all the
system
resources if it can't dertermine it's busy or not?
Thanks.
Raymond
>From: Mike Matrigali <mi...@sbcglobal.net>
>Reply-To: "Derby Development" <de...@db.apache.org>
>To: Derby Development <de...@db.apache.org>
>Subject: Re: Some idea about checkpoint issue, welcome to give your idea
>Date: Tue, 11 Oct 2005 10:57:08 -0700
>
>Are there any opinions out there on how to determine if
>Derby "is busy"? Is there something better than just having
>a low priority thread and maybe some query of cpu vs. elapsed
>time?
>
>The first problem is that I don't think there are great tools
>for this in java. The second problem is that often Derby is
>meant to be embedded as part of another application, so we have
>to be careful not to implement a standard server based approach
>where it is appropriate for the "server" to use up all resources
>available (ie. idle time may not really be best used by derby
>admin processes).
>
>I have not come up with a good answer to this problem, there are
>a number of things derby could do if it knew it had idle time
>available for it's use. Best I have come up with is some mode
>in the system that needs to be set by the application which
>starts up Derby - either derby try's to limit it's use of idle
>cycles or it enabled to try and schedule work during idle time.
>
>Raymond Raymond wrote:
>
> > I have been thinking of the automatic checkpointing issue
> > recently.I also find someone added another issue about "Use
> > of idle time for background checkpoint" into the to-do list.
> > I think we can consider these two issue together. I have
> > some idea about it.
> >
> > Instead of doing checkpoint periodically and trying to tune the
> > checkpoint interval to achieve best performance, is it possible to
> > keep the background checkpoint process running to do checkpoint,
> > and the DBMS can tune the rate of checkpoint depending on the
> > current system situation,e.g. if the system is busy, derby will
> > slow down the checkpoint rate and if the system is not busy(idle),
> > derby will speed up the checkpoint rate.We will update the control
> > file periodically to let the DBMS know up to where we did checkpoint.
> > Maybe we can call it 'increamental checkpointing'. In my opinion,
> > this approach can use the disk IO resources with reason if we can
> > decide the checkpoint rate reasonablly.
> >
> > I would like to discuss this issue with everyone. I am not
> > sure if this approach is doable or not. If it is doable, I will
> > have some further questions about how to decide the appropriate
> > checkpoint rate.
> >
> > Thanks.
> >
> > Yours, Raymond
> >
> > _________________________________________________________________
> > Designer Mail isn't just fun to send, it's fun to receive. Use special
> > stationery, fonts and colors.
> >
>http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> > Start enjoying all the benefits of MSN?Premium right now and get the
> > first two months FREE*.
> >
> >
_________________________________________________________________
Take charge with a pop-up guard built on patented MicrosoftŽ SmartScreen
Technology
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Some idea about checkpoint issue, welcome to give your idea
Posted by Raymond Raymond <ra...@hotmail.com>.
As for the checkpointing issue, the main problem is that
we want to do checkpoint as much as possible, so it will
recude the recovery time, while we don't want the checkpoint
process slows down the "real" work of the system if it
occupies too much disk IO resource( the disk IO is the
bottleneck).
Is it possible for us to determine if derby is "busy" based
on how many writes derby does in one time unit? Since
everything will be logged before being written to disk,
maybe we can figure out if derby is "busy" depending on
the growth of log. I am no sure whether it is possible to
do so.^_^.
Raymond
Mike Matrigali wrote:
>
>Are there any opinions out there on how to determine if
>Derby "is busy"? Is there something better than just having
>a low priority thread and maybe some query of cpu vs. elapsed
>time?
>
>The first problem is that I don't think there are great tools
>for this in java. The second problem is that often Derby is
>meant to be embedded as part of another application, so we have
>to be careful not to implement a standard server based approach
>where it is appropriate for the "server" to use up all resources
>available (ie. idle time may not really be best used by derby
>admin processes).
>
>I have not come up with a good answer to this problem, there are
>a number of things derby could do if it knew it had idle time
>available for it's use. Best I have come up with is some mode
>in the system that needs to be set by the application which
>starts up Derby - either derby try's to limit it's use of idle
>cycles or it enabled to try and schedule work during idle time.
>
>Raymond Raymond wrote:
>
> > I have been thinking of the automatic checkpointing issue
> > recently.I also find someone added another issue about "Use
> > of idle time for background checkpoint" into the to-do list.
> > I think we can consider these two issue together. I have
> > some idea about it.
> >
> > Instead of doing checkpoint periodically and trying to tune the
> > checkpoint interval to achieve best performance, is it possible to
> > keep the background checkpoint process running to do checkpoint,
> > and the DBMS can tune the rate of checkpoint depending on the
> > current system situation,e.g. if the system is busy, derby will
> > slow down the checkpoint rate and if the system is not busy(idle),
> > derby will speed up the checkpoint rate.We will update the control
> > file periodically to let the DBMS know up to where we did checkpoint.
> > Maybe we can call it 'increamental checkpointing'. In my opinion,
> > this approach can use the disk IO resources with reason if we can
> > decide the checkpoint rate reasonablly.
> >
> > I would like to discuss this issue with everyone. I am not
> > sure if this approach is doable or not. If it is doable, I will
> > have some further questions about how to decide the appropriate
> > checkpoint rate.
> >
> > Thanks.
> >
> > Yours, Raymond
> >
> > _________________________________________________________________
> > Designer Mail isn't just fun to send, it's fun to receive. Use special
> > stationery, fonts and colors.
> >
>http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> > Start enjoying all the benefits of MSNŽ Premium right now and get the
> > first two months FREE*.
> >
> >
_________________________________________________________________
Take advantage of powerful junk e-mail filters built on patented MicrosoftŽ
SmartScreen Technology.
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
Start enjoying all the benefits of MSNŽ Premium right now and get the
first two months FREE*.
Re: Some idea about checkpoint issue, welcome to give your idea
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Are there any opinions out there on how to determine if
Derby "is busy"? Is there something better than just having
a low priority thread and maybe some query of cpu vs. elapsed
time?
The first problem is that I don't think there are great tools
for this in java. The second problem is that often Derby is
meant to be embedded as part of another application, so we have
to be careful not to implement a standard server based approach
where it is appropriate for the "server" to use up all resources
available (ie. idle time may not really be best used by derby
admin processes).
I have not come up with a good answer to this problem, there are
a number of things derby could do if it knew it had idle time
available for it's use. Best I have come up with is some mode
in the system that needs to be set by the application which
starts up Derby - either derby try's to limit it's use of idle
cycles or it enabled to try and schedule work during idle time.
Raymond Raymond wrote:
> I have been thinking of the automatic checkpointing issue
> recently.I also find someone added another issue about "Use
> of idle time for background checkpoint" into the to-do list.
> I think we can consider these two issue together. I have
> some idea about it.
>
> Instead of doing checkpoint periodically and trying to tune the
> checkpoint interval to achieve best performance, is it possible to
> keep the background checkpoint process running to do checkpoint,
> and the DBMS can tune the rate of checkpoint depending on the
> current system situation,e.g. if the system is busy, derby will
> slow down the checkpoint rate and if the system is not busy(idle),
> derby will speed up the checkpoint rate.We will update the control
> file periodically to let the DBMS know up to where we did checkpoint.
> Maybe we can call it 'increamental checkpointing'. In my opinion,
> this approach can use the disk IO resources with reason if we can
> decide the checkpoint rate reasonablly.
>
> I would like to discuss this issue with everyone. I am not
> sure if this approach is doable or not. If it is doable, I will
> have some further questions about how to decide the appropriate
> checkpoint rate.
>
> Thanks.
>
> Yours, Raymond
>
> _________________________________________________________________
> Designer Mail isn't just fun to send, it's fun to receive. Use special
> stationery, fonts and colors.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
> Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
>
>