You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Gerhard Froehlich <g-...@gmx.de> on 2002/01/28 21:20:39 UTC
StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Hi Team,
I hope I fixed the problem now!
I changed the StoreJanitorImpl, I try to explain in
prosa:
<not to serious>
Cocoon reaches 1 million request per second and
memory is getting dangerously low. Time for the
StoreJanitor to kick in and do something. First
he forces the GC. Damn, memory is still low,
we have to remove object from the Store to get
some bytes free. Panic, which Store shall we take?
Ahh index is at -1 so we take the first at index 0
in the StoreArray. Gosh this one is pretty full how
much objects shall we kick out? Stupid question,
take 10% from all as configured. Chacka, did ya see
them fly?
Next time we take the next store on index 1.
Yeah great work dudes, take a rest.
</not to serious>
Test it!
Gerhard
------------------------------------------------
If patience is a virtue, and ignorance is bliss,
you can have a pretty good life if you're stupid
and willing to wait.
------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Gerhard Froehlich <g-...@gmx.de>.
Hi,
applied to the scratchpad. Own config is missing.
I have to go to bed now ;)...
TIA
~Gerhard
"Sorry, but my karma just ran over your dogma."
>-----Original Message-----
>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>Sent: Tuesday, February 05, 2002 1:36 AM
>To: cocoon-dev@xml.apache.org
>Subject: Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs
>System]
>
>
>Hi Gerhard,
>
>Great to hear that I've not been wasting my time :-)
>
>Please find attached the updated code.
>
>g-froehlich@gmx.de wrote:
>
>> Peter,
>>
>>
>>>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>>>
>>>Hi Gerhard,
>>>
>>>I've tested your latest Store Janitor improvements (using Optimizit and
>>>your debug messages) and although its a very great improvement, I've
>>>found some other problems :-(
>>>
>>
>> Damn...
>>
>>
>>>As my Optimizit trial was running out I decided to do some urgent work
>>>on the Checker and now have a solution which appears to work well :-)
>>>
>>>Here is a summary of the problems with ideas for correcting them (some
>>>are nitty gritty, and some you already know):
>>>
>>>*) When demand for memory is high, it is possible to run out of
>>>java.memory before the store has been emptied. Might it be possible to
>>>make sure the stores are empty before the memory can run out?
>>>
>>
>> Then we don't need the StoreJanitor, or?
>>
>>
>>>*) When memory is low, there is a risk of running out of it before the
>>>next check. Why not set the interval short enough to ensure there is no
>>>risk of running out of memory before the next check?
>>>
>>
>> Be careful with Threads. When they run in a to short interval they'll
>> kill your performance!
>>
>>
>>>*) Calling the gc once every check interval seriously undermines system
>>>performance because it can take several seconds to complete, which is
>>>comparable to the check interval. Why not call it only when some items
>>>are released from a store? Why not trust it and use its characteristics
>>>to advantage?
>>>
>>
>> That's a contradiction to previous point, I guess. On the one hand you want
>> to increase the the iteration of the Thread on the other hand you want to
>> decrease the GC calls.
>
>
>Ah, but only call the GC if stores are reduced - I think you got it
>lower down the page.
>
>
>>
>>
>>>*) When several instances of Cocoon are running (independent .WAR apps
>>>in the same servlet container), the calls to gc become very dominant.
>>>Would this affect the strategy for store-janitor settings?
>>>
>>>*) If the percent to reduce storage is set to 10%, it fails to remove
>>>any when the number of items are below 10. The number of items to be
>>>removed needs rounding upwards. Why not remove a fixed number of items
>>>instead of a percentage? (My idea and now I think it was wrong!!)
>>>
>>
>> Au, my embarrsing math capabilities strikes agein ;).
>>
>>
>>>*) It is difficult to get a good picture of what is happening by
>>>observing the debug output, so it is difficult to know if it is working
>>>and difficult use the debug output to optimise choice of janitor
>>>settings. Might it be possible to make this easy?
>>>
>>
>> Yes and no. That's always a problem in Cocoon. But you can define your
>> own log target (own file) in the logkit.xconf for the StoreJanitor. Then
>> you get rid of the noise.
>
>
>I suspected something like this might be possible - sounds great!
>
>
>>
>>
>>>The proposed, implemented and tested solution to the above works as follows:
>>>
>>>1) Monitor the rate at which memory is consumed during each check
>>>interval. Remember the biggest rate that has been found so far, to use
>>>it later. My system logs this at 2.5 megabytes per second.
>>>
>>
>> Understand.
>>
>>
>>>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>>>
>>>2.1) 'heapIsBig' means that the total memory has exceeded the parameter
>>>'maxheap'.
>>>
>>>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>>>
>>>2.3) Attempt to remove from the stores, the number of items specified by
>>>the parameter 'reduceby'.
>>>
>>>2.4) To free storage, start at the next store and remove items, moving
>>>to the next again if necessary, until the specified number of items have
>>>been removed or all the stores are empty.
>>>
>>>2.5) Then call the garbage collector, but only if some memory items were
>>>freed.
>>>
>>
>> Ok, that's nice.
>>
>>
>>>3) Sleep for an interval half that in which the memory could run out.
>>>
>>>3.1) If the remaining heap could be more than half filled during the
>>>interval specified by the 'maxsleep' parameter (at the max rate of
>>>memory consumption) then sleep for the time it would take to only half
>>>fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval.
>>>
>>
>> Damn, that's sounds great ;). Your biggest unknown variable is the
>> "could be". But I think taking the last biggest consumption is not that
>> bad. But how about taking the average of all calls. That would be more exact
>> then the last biggest value.
>
>
>You got straight to the weakest point didn't you? My fear is that a
>single rogue ridiculously high value might get stuck in there - I think
>that's possibly what you are getting at.
>
>Can't average all the rates, because you need only the periods where
>memory is trying to increase as fast as it can. Which ones are these?
>
>Could possibly, average any new highest value with the current highest.
>Or average two of the current one with the new one (i.e. give a greater
>weight to the current one).
>
>So, for the moment I've changed it to average the new highest rate with
>the old - instead of just replacing it. I'm still thinking about this one.
>
>
>>
>>>The effect of the store-janitor parameters in cocoon.xconf are now
>>>slightly changed, so I've changed the names of the parameters to reflect
>>>this. For the moment the names are changed only inside
>>>StoreJanitorImpl.java. If the solution were to be adopted then the names
>>>in cocoon.xconf would also change. The changes, with notes, are as follows:
>>>
>>
>> Naming was indeed crap.
>>
>>
>>>'maxheap' (was 'heapsize') The maximum total heap size below which
>>>stores are guaranteed to remain intact.
>>>
>>>'minfree' (was 'freememory') The minimum free heap required for stores
>>>to remain intact. I have found this should be set to less than 10% of
>>>avaliable memory, so that the jvm will respond to low memory by
>>>allocating more heap (if its available), before stores are likely to be
>>>reduced. Default value might be 2000000. Setting it to zero or very low
>>>will effectively disable store reduction.
>>>
>>>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between
>>>checks. Should be short enough to ensure that 'rate of change' data is
>>>collected before free memory runs low. Suggest 5 secs as default.
>>>
>>>'reduceby' (was 'percent_to_free') is the number of items to be removed
>>>>>from the stores, on each attempt to reduce the stores. Each successfull
>>>attempt to reduce stores results in a call to the garbage collector
>>>taking several seconds. This limits the rate a which the stores can be
>>>emptied. Removing several items at a time compensates for this. Using
>>>the debug output, choose a number of items which results in an increase
>>>of free memory between successive store reductions. Best tested with a
>>>sudden high demand for memory when the stores are full. In practice this
>>>setting is not critical. Suggested default is 10 items. Ideally
>>>'reduceby' would be in bytes!
>>>
>>>Setting 'minfree' and 'reduceby' both too low can intensify gc activity
>>>if sudden high demand requires all stores to be completely freed and can
>>>result in memory running out before all the stores are free. Verify
>>>using debug output.
>>>
>>>'priority' remains unchanged.
>>>
>>>The above solution is implemented and tested and it works very well
>>>indeed for me. The parameters do not seem to be at all critical - thank
>>>goodness! I see no reason why the default settings would need to be
>>>changed if more memory were available. I have attached the
>>>StoreJanitorImpl.java so others can test it. I find I can get a very
>>>good picture of what is happening by using my editor's repeat find
>>>command on the debug output in the core.log.000001 file. Optimizeit is
>>>no longer necessary to prove that it works effectively.
>>>
>>
>> Just define as mentioned above you own log target for the store.janitor.
>> Take a look into logkit.xconf in the WEB-INF dir!
>>
>>
>>>Beware! - there might still be things I don't know about that I have not
>>>take into account.
>>>Ideally - low memory should be detected by some sort of interrupt or
>>>exception rather than by polling.
>>>
>>
>> That's correct. But here java is a big white spot.
>>
>>
>>>If you are subsequently interested in committing these changes into
>>>Cocoon and you want me to do some more work to conform to design
>>>practice etc., then please let me know.
>>>
>>
>> Yep of course. But I switch to my picky mode now. Please reformat your
>> code to our coding standards.
>> - 4 indents, no tabs.
>> - if(foo)
>> this.isBad = true;
>> That's not good coding style. Use brackets for each condition or block.
>> - Make it more readable
>
>
>O.K. done that. It now makes a thrilling read! I'm not sure about a
>sequel though.
>
>
>>
>> Then I will commit your stuff into the Scratchpad area.
>> (That's the normal way this days)
>>
>
>
>Peter.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Peter Hargreaves <pd...@netscape.net>.
Hi Gerhard,
Great to hear that I've not been wasting my time :-)
Please find attached the updated code.
g-froehlich@gmx.de wrote:
> Peter,
>
>
>>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>>
>>Hi Gerhard,
>>
>>I've tested your latest Store Janitor improvements (using Optimizit and
>>your debug messages) and although its a very great improvement, I've
>>found some other problems :-(
>>
>
> Damn...
>
>
>>As my Optimizit trial was running out I decided to do some urgent work
>>on the Checker and now have a solution which appears to work well :-)
>>
>>Here is a summary of the problems with ideas for correcting them (some
>>are nitty gritty, and some you already know):
>>
>>*) When demand for memory is high, it is possible to run out of
>>java.memory before the store has been emptied. Might it be possible to
>>make sure the stores are empty before the memory can run out?
>>
>
> Then we don't need the StoreJanitor, or?
>
>
>>*) When memory is low, there is a risk of running out of it before the
>>next check. Why not set the interval short enough to ensure there is no
>>risk of running out of memory before the next check?
>>
>
> Be careful with Threads. When they run in a to short interval they'll
> kill your performance!
>
>
>>*) Calling the gc once every check interval seriously undermines system
>>performance because it can take several seconds to complete, which is
>>comparable to the check interval. Why not call it only when some items
>>are released from a store? Why not trust it and use its characteristics
>>to advantage?
>>
>
> That's a contradiction to previous point, I guess. On the one hand you want
> to increase the the iteration of the Thread on the other hand you want to
> decrease the GC calls.
Ah, but only call the GC if stores are reduced - I think you got it
lower down the page.
>
>
>>*) When several instances of Cocoon are running (independent .WAR apps
>>in the same servlet container), the calls to gc become very dominant.
>>Would this affect the strategy for store-janitor settings?
>>
>>*) If the percent to reduce storage is set to 10%, it fails to remove
>>any when the number of items are below 10. The number of items to be
>>removed needs rounding upwards. Why not remove a fixed number of items
>>instead of a percentage? (My idea and now I think it was wrong!!)
>>
>
> Au, my embarrsing math capabilities strikes agein ;).
>
>
>>*) It is difficult to get a good picture of what is happening by
>>observing the debug output, so it is difficult to know if it is working
>>and difficult use the debug output to optimise choice of janitor
>>settings. Might it be possible to make this easy?
>>
>
> Yes and no. That's always a problem in Cocoon. But you can define your
> own log target (own file) in the logkit.xconf for the StoreJanitor. Then
> you get rid of the noise.
I suspected something like this might be possible - sounds great!
>
>
>>The proposed, implemented and tested solution to the above works as follows:
>>
>>1) Monitor the rate at which memory is consumed during each check
>>interval. Remember the biggest rate that has been found so far, to use
>>it later. My system logs this at 2.5 megabytes per second.
>>
>
> Understand.
>
>
>>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>>
>>2.1) 'heapIsBig' means that the total memory has exceeded the parameter
>>'maxheap'.
>>
>>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>>
>>2.3) Attempt to remove from the stores, the number of items specified by
>>the parameter 'reduceby'.
>>
>>2.4) To free storage, start at the next store and remove items, moving
>>to the next again if necessary, until the specified number of items have
>>been removed or all the stores are empty.
>>
>>2.5) Then call the garbage collector, but only if some memory items were
>>freed.
>>
>
> Ok, that's nice.
>
>
>>3) Sleep for an interval half that in which the memory could run out.
>>
>>3.1) If the remaining heap could be more than half filled during the
>>interval specified by the 'maxsleep' parameter (at the max rate of
>>memory consumption) then sleep for the time it would take to only half
>>fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval.
>>
>
> Damn, that's sounds great ;). Your biggest unknown variable is the
> "could be". But I think taking the last biggest consumption is not that
> bad. But how about taking the average of all calls. That would be more exact
> then the last biggest value.
You got straight to the weakest point didn't you? My fear is that a
single rogue ridiculously high value might get stuck in there - I think
that's possibly what you are getting at.
Can't average all the rates, because you need only the periods where
memory is trying to increase as fast as it can. Which ones are these?
Could possibly, average any new highest value with the current highest.
Or average two of the current one with the new one (i.e. give a greater
weight to the current one).
So, for the moment I've changed it to average the new highest rate with
the old - instead of just replacing it. I'm still thinking about this one.
>
>>The effect of the store-janitor parameters in cocoon.xconf are now
>>slightly changed, so I've changed the names of the parameters to reflect
>>this. For the moment the names are changed only inside
>>StoreJanitorImpl.java. If the solution were to be adopted then the names
>>in cocoon.xconf would also change. The changes, with notes, are as follows:
>>
>
> Naming was indeed crap.
>
>
>>'maxheap' (was 'heapsize') The maximum total heap size below which
>>stores are guaranteed to remain intact.
>>
>>'minfree' (was 'freememory') The minimum free heap required for stores
>>to remain intact. I have found this should be set to less than 10% of
>>avaliable memory, so that the jvm will respond to low memory by
>>allocating more heap (if its available), before stores are likely to be
>>reduced. Default value might be 2000000. Setting it to zero or very low
>>will effectively disable store reduction.
>>
>>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between
>>checks. Should be short enough to ensure that 'rate of change' data is
>>collected before free memory runs low. Suggest 5 secs as default.
>>
>>'reduceby' (was 'percent_to_free') is the number of items to be removed
>>>from the stores, on each attempt to reduce the stores. Each successfull
>>attempt to reduce stores results in a call to the garbage collector
>>taking several seconds. This limits the rate a which the stores can be
>>emptied. Removing several items at a time compensates for this. Using
>>the debug output, choose a number of items which results in an increase
>>of free memory between successive store reductions. Best tested with a
>>sudden high demand for memory when the stores are full. In practice this
>>setting is not critical. Suggested default is 10 items. Ideally
>>'reduceby' would be in bytes!
>>
>>Setting 'minfree' and 'reduceby' both too low can intensify gc activity
>>if sudden high demand requires all stores to be completely freed and can
>>result in memory running out before all the stores are free. Verify
>>using debug output.
>>
>>'priority' remains unchanged.
>>
>>The above solution is implemented and tested and it works very well
>>indeed for me. The parameters do not seem to be at all critical - thank
>>goodness! I see no reason why the default settings would need to be
>>changed if more memory were available. I have attached the
>>StoreJanitorImpl.java so others can test it. I find I can get a very
>>good picture of what is happening by using my editor's repeat find
>>command on the debug output in the core.log.000001 file. Optimizeit is
>>no longer necessary to prove that it works effectively.
>>
>
> Just define as mentioned above you own log target for the store.janitor.
> Take a look into logkit.xconf in the WEB-INF dir!
>
>
>>Beware! - there might still be things I don't know about that I have not
>>take into account.
>>Ideally - low memory should be detected by some sort of interrupt or
>>exception rather than by polling.
>>
>
> That's correct. But here java is a big white spot.
>
>
>>If you are subsequently interested in committing these changes into
>>Cocoon and you want me to do some more work to conform to design
>>practice etc., then please let me know.
>>
>
> Yep of course. But I switch to my picky mode now. Please reformat your
> code to our coding standards.
> - 4 indents, no tabs.
> - if(foo)
> this.isBad = true;
> That's not good coding style. Use brackets for each condition or block.
> - Make it more readable
O.K. done that. It now makes a thrilling read! I'm not sure about a
sequel though.
>
> Then I will commit your stuff into the Scratchpad area.
> (That's the normal way this days)
>
Peter.
RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Gerhard Froehlich <g-...@gmx.de>.
Peter,
>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>
>Hi Gerhard,
>
>I've tested your latest Store Janitor improvements (using Optimizit and
>your debug messages) and although its a very great improvement, I've
>found some other problems :-(
Damn...
>As my Optimizit trial was running out I decided to do some urgent work
>on the Checker and now have a solution which appears to work well :-)
>
>Here is a summary of the problems with ideas for correcting them (some
>are nitty gritty, and some you already know):
>
>*) When demand for memory is high, it is possible to run out of
>java.memory before the store has been emptied. Might it be possible to
>make sure the stores are empty before the memory can run out?
Then we don't need the StoreJanitor, or?
>*) When memory is low, there is a risk of running out of it before the
>next check. Why not set the interval short enough to ensure there is no
>risk of running out of memory before the next check?
Be careful with Threads. When they run in a to short interval they'll
kill your performance!
>*) Calling the gc once every check interval seriously undermines system
>performance because it can take several seconds to complete, which is
>comparable to the check interval. Why not call it only when some items
>are released from a store? Why not trust it and use its characteristics
>to advantage?
That's a contradiction to previous point, I guess. On the one hand you want
to increase the the iteration of the Thread on the other hand you want to
decrease the GC calls.
>*) When several instances of Cocoon are running (independent .WAR apps
>in the same servlet container), the calls to gc become very dominant.
>Would this affect the strategy for store-janitor settings?
>
>*) If the percent to reduce storage is set to 10%, it fails to remove
>any when the number of items are below 10. The number of items to be
>removed needs rounding upwards. Why not remove a fixed number of items
>instead of a percentage? (My idea and now I think it was wrong!!)
Au, my embarrsing math capabilities strikes agein ;).
>*) It is difficult to get a good picture of what is happening by
>observing the debug output, so it is difficult to know if it is working
>and difficult use the debug output to optimise choice of janitor
>settings. Might it be possible to make this easy?
Yes and no. That's always a problem in Cocoon. But you can define your
own log target (own file) in the logkit.xconf for the StoreJanitor. Then
you get rid of the noise.
>The proposed, implemented and tested solution to the above works as follows:
>
>1) Monitor the rate at which memory is consumed during each check
>interval. Remember the biggest rate that has been found so far, to use
>it later. My system logs this at 2.5 megabytes per second.
Understand.
>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>
>2.1) 'heapIsBig' means that the total memory has exceeded the parameter
>'maxheap'.
>
>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>
>2.3) Attempt to remove from the stores, the number of items specified by
>the parameter 'reduceby'.
>
>2.4) To free storage, start at the next store and remove items, moving
>to the next again if necessary, until the specified number of items have
>been removed or all the stores are empty.
>
>2.5) Then call the garbage collector, but only if some memory items were
>freed.
Ok, that's nice.
>3) Sleep for an interval half that in which the memory could run out.
>
>3.1) If the remaining heap could be more than half filled during the
>interval specified by the 'maxsleep' parameter (at the max rate of
>memory consumption) then sleep for the time it would take to only half
>fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval.
Damn, that's sounds great ;). Your biggest unknown variable is the
"could be". But I think taking the last biggest consumption is not that
bad. But how about taking the average of all calls. That would be more exact
then the last biggest value.
>The effect of the store-janitor parameters in cocoon.xconf are now
>slightly changed, so I've changed the names of the parameters to reflect
>this. For the moment the names are changed only inside
>StoreJanitorImpl.java. If the solution were to be adopted then the names
>in cocoon.xconf would also change. The changes, with notes, are as follows:
Naming was indeed crap.
>'maxheap' (was 'heapsize') The maximum total heap size below which
>stores are guaranteed to remain intact.
>
>'minfree' (was 'freememory') The minimum free heap required for stores
>to remain intact. I have found this should be set to less than 10% of
>avaliable memory, so that the jvm will respond to low memory by
>allocating more heap (if its available), before stores are likely to be
>reduced. Default value might be 2000000. Setting it to zero or very low
>will effectively disable store reduction.
>
>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between
>checks. Should be short enough to ensure that 'rate of change' data is
>collected before free memory runs low. Suggest 5 secs as default.
>
>'reduceby' (was 'percent_to_free') is the number of items to be removed
>>from the stores, on each attempt to reduce the stores. Each successfull
>attempt to reduce stores results in a call to the garbage collector
>taking several seconds. This limits the rate a which the stores can be
>emptied. Removing several items at a time compensates for this. Using
>the debug output, choose a number of items which results in an increase
>of free memory between successive store reductions. Best tested with a
>sudden high demand for memory when the stores are full. In practice this
>setting is not critical. Suggested default is 10 items. Ideally
>'reduceby' would be in bytes!
>
>Setting 'minfree' and 'reduceby' both too low can intensify gc activity
>if sudden high demand requires all stores to be completely freed and can
>result in memory running out before all the stores are free. Verify
>using debug output.
>
>'priority' remains unchanged.
>
>The above solution is implemented and tested and it works very well
>indeed for me. The parameters do not seem to be at all critical - thank
>goodness! I see no reason why the default settings would need to be
>changed if more memory were available. I have attached the
>StoreJanitorImpl.java so others can test it. I find I can get a very
>good picture of what is happening by using my editor's repeat find
>command on the debug output in the core.log.000001 file. Optimizeit is
>no longer necessary to prove that it works effectively.
Just define as mentioned above you own log target for the store.janitor.
Take a look into logkit.xconf in the WEB-INF dir!
>Beware! - there might still be things I don't know about that I have not
>take into account.
>Ideally - low memory should be detected by some sort of interrupt or
>exception rather than by polling.
That's correct. But here java is a big white spot.
>If you are subsequently interested in committing these changes into
>Cocoon and you want me to do some more work to conform to design
>practice etc., then please let me know.
Yep of course. But I switch to my picky mode now. Please reformat your
code to our coding standards.
- 4 indents, no tabs.
- if(foo)
this.isBad = true;
That's not good coding style. Use brackets for each condition or block.
- Make it more readable
Then I will commit your stuff into the Scratchpad area.
(That's the normal way this days)
What do you think!
>Very humble appologies, Gerhard (and others), for intruding into your
>code. If you like my changes, I hope you feel that I have released your
>brilliance (not undermined it). I feel that by far the biggest part of
>any credit should very definately still be yours (I only fixed the
>checker) :-).
Wah, no apologizes please! I always wished, that somebody start to refactor
this idea, because I stuck in the moment.
PLEASE PEOPLE DON'T APLOGIZES FOR SENDING PATCHES. THAT'S OSS HERE AND NOT
A DAMN CATHEDRAL. We need more people sending patches. Make yourself heard here!!
~Gerhard
+----------------------------------------------+
My parents have been visiting me for a few days.
I just dropped them off at the airport.
They leave tomorrow.
+----------------------------------------------+
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Vadim Gritsenko <va...@verizon.net>.
> From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>
> Hi Vadim,
>
> vadim.gritsenko@verizon.net wrote:
>
> >>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
> >>
> >>*) If the percent to reduce storage is set to 10%, it fails to
remove
> >>any when the number of items are below 10. The number of items to be
> >>removed needs rounding upwards. Why not remove a fixed number of
items
> >>instead of a percentage? (My idea and now I think it was wrong!!)
> >>
> >
> > +1 on rounding up,
> > -1 on removing percentage. Applications highly differ on number of
> > objects and average size of the object. Percentage gives the
advantage
> > to have generic configuration. If this is not over-configuration,
number
> > of objects to remove may be configured with or without percent sign:
> >
> > <parameter name="reduceby" value="10%"/>
> > <parameter name="reduceby" value="100"/>
>
>
> My thinking is this. Ideally you want to remove a defined number of
> bytes from the stores. If you can guess the average size of an item
then
> by removing a defined number of items you are removing an approximate
> number of bytes.
That's when you tune for one particular application - then yes. If you
try to put something that works well when you have 200 huge objects, and
when you have 20000 tiny one - I think defined number won't work. What
can you suggest here? I hoped that percentage may work for these
situations.
> When I tried removing a percentage, I found that successive attempts
> freed up less and less storage
This makes sence, I did not thought of it this way...
> then my system ran out of memory before the store was empty.
...but on the other hand: it means that you have leaks in some other
place! May be you will want to find them first before blaming memory
store ;)
It may be not leaks, but uncontrolled memory consumption by some
business logic of your site. Then why not just increase free-memory
parameter to factor in these requirements?
> The trouble with convincing people is that you have to unconvince them
> if you change you mind ;-)
:)
> >>Ideally - low memory should be detected by some sort of interrupt or
> >>exception rather than by polling.
> >>
> >
> > AFAIK, when you get an exception it's already late to take any
action.
>
> Ah, but what about the undo command. Or the retrospective exception
> handler. You know the one I mean - "It was left at that previous
junction".
>
> Seriously though. Are there any plans for java to include a mechanism
> that tells folks when memory is getting low. Surely lots of people
have
> this problem.
Don't know. Search bug database - may be there is already couple of
requests in there.
Regards,
Vadim
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Peter Hargreaves <pd...@netscape.net>.
Hi Vadim,
vadim.gritsenko@verizon.net wrote:
> Beware: multiple snips inside...
>
>
>>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>>
>>*) Calling the gc once every check interval seriously undermines
>>
> system
>
>>performance because it can take several seconds to complete, which is
>>comparable to the check interval. Why not call it only when some items
>>are released from a store? Why not trust it and use its
>>
> characteristics
>
>>to advantage?
>>
>>*) When several instances of Cocoon are running (independent .WAR apps
>>in the same servlet container), the calls to gc become very dominant.
>>Would this affect the strategy for store-janitor settings?
>>
>
> +1 on reducing gc() calls whenever possible.
>
>
>
>>*) If the percent to reduce storage is set to 10%, it fails to remove
>>any when the number of items are below 10. The number of items to be
>>removed needs rounding upwards. Why not remove a fixed number of items
>>instead of a percentage? (My idea and now I think it was wrong!!)
>>
>
> +1 on rounding up,
> -1 on removing percentage. Applications highly differ on number of
> objects and average size of the object. Percentage gives the advantage
> to have generic configuration. If this is not over-configuration, number
> of objects to remove may be configured with or without percent sign:
>
> <parameter name="reduceby" value="10%"/>
> <parameter name="reduceby" value="100"/>
My thinking is this. Ideally you want to remove a defined number of
bytes from the stores. If you can guess the average size of an item then
by removing a defined number of items you are removing an approximate
number of bytes.
When I tried removing a percentage, I found that successive attempts
freed up less and less storage then my system ran out of memory before
the store was empty.
The trouble with convincing people is that you have to unconvince them
if you change you mind ;-)
>
>
>>2.5) Then call the garbage collector, but only if some memory items
>>
> were
>
>>freed.
>>
>
> Do we need to call gc() at all? I suppose that if memory is really low
> gc() will be called at next new().
>
>
>
>>The above solution is implemented and tested and it works very well
>>indeed for me. The parameters do not seem to be at all critical -
>>
> thank
>
>>goodness! I see no reason why the default settings would need to be
>>changed if more memory were available. I have attached the
>>StoreJanitorImpl.java so others can test it. I find I can get a very
>>good picture of what is happening by using my editor's repeat find
>>command on the debug output in the core.log.000001 file. Optimizeit is
>>no longer necessary to prove that it works effectively.
>>
>
> That's sounds promising.
>
>
>
>>Ideally - low memory should be detected by some sort of interrupt or
>>exception rather than by polling.
>>
>
> AFAIK, when you get an exception it's already late to take any action.
Ah, but what about the undo command. Or the retrospective exception
handler. You know the one I mean - "It was left at that previous junction".
Seriously though. Are there any plans for java to include a mechanism
that tells folks when memory is getting low. Surely lots of people have
this problem.
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Vadim Gritsenko <va...@verizon.net>.
Beware: multiple snips inside...
> From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>
> *) Calling the gc once every check interval seriously undermines
system
> performance because it can take several seconds to complete, which is
> comparable to the check interval. Why not call it only when some items
> are released from a store? Why not trust it and use its
characteristics
> to advantage?
>
> *) When several instances of Cocoon are running (independent .WAR apps
> in the same servlet container), the calls to gc become very dominant.
> Would this affect the strategy for store-janitor settings?
+1 on reducing gc() calls whenever possible.
> *) If the percent to reduce storage is set to 10%, it fails to remove
> any when the number of items are below 10. The number of items to be
> removed needs rounding upwards. Why not remove a fixed number of items
> instead of a percentage? (My idea and now I think it was wrong!!)
+1 on rounding up,
-1 on removing percentage. Applications highly differ on number of
objects and average size of the object. Percentage gives the advantage
to have generic configuration. If this is not over-configuration, number
of objects to remove may be configured with or without percent sign:
<parameter name="reduceby" value="10%"/>
<parameter name="reduceby" value="100"/>
> 2.5) Then call the garbage collector, but only if some memory items
were
> freed.
Do we need to call gc() at all? I suppose that if memory is really low
gc() will be called at next new().
> The above solution is implemented and tested and it works very well
> indeed for me. The parameters do not seem to be at all critical -
thank
> goodness! I see no reason why the default settings would need to be
> changed if more memory were available. I have attached the
> StoreJanitorImpl.java so others can test it. I find I can get a very
> good picture of what is happening by using my editor's repeat find
> command on the debug output in the core.log.000001 file. Optimizeit is
> no longer necessary to prove that it works effectively.
That's sounds promising.
> Ideally - low memory should be detected by some sort of interrupt or
> exception rather than by polling.
AFAIK, when you get an exception it's already late to take any action.
Vadim
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Posted by Peter Hargreaves <pd...@netscape.net>.
Hi Gerhard,
I've tested your latest Store Janitor improvements (using Optimizit and
your debug messages) and although its a very great improvement, I've
found some other problems :-(
As my Optimizit trial was running out I decided to do some urgent work
on the Checker and now have a solution which appears to work well :-)
Here is a summary of the problems with ideas for correcting them (some
are nitty gritty, and some you already know):
*) When demand for memory is high, it is possible to run out of
java.memory before the store has been emptied. Might it be possible to
make sure the stores are empty before the memory can run out?
*) When memory is low, there is a risk of running out of it before the
next check. Why not set the interval short enough to ensure there is no
risk of running out of memory before the next check?
*) Calling the gc once every check interval seriously undermines system
performance because it can take several seconds to complete, which is
comparable to the check interval. Why not call it only when some items
are released from a store? Why not trust it and use its characteristics
to advantage?
*) When several instances of Cocoon are running (independent .WAR apps
in the same servlet container), the calls to gc become very dominant.
Would this affect the strategy for store-janitor settings?
*) If the percent to reduce storage is set to 10%, it fails to remove
any when the number of items are below 10. The number of items to be
removed needs rounding upwards. Why not remove a fixed number of items
instead of a percentage? (My idea and now I think it was wrong!!)
*) It is difficult to get a good picture of what is happening by
observing the debug output, so it is difficult to know if it is working
and difficult use the debug output to optimise choice of janitor
settings. Might it be possible to make this easy?
The proposed, implemented and tested solution to the above works as follows:
1) Monitor the rate at which memory is consumed during each check
interval. Remember the biggest rate that has been found so far, to use
it later. My system logs this at 2.5 megabytes per second.
2) If heapIsBig() && freeIsLow(), then attempt to free storage.
2.1) 'heapIsBig' means that the total memory has exceeded the parameter
'maxheap'.
2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
2.3) Attempt to remove from the stores, the number of items specified by
the parameter 'reduceby'.
2.4) To free storage, start at the next store and remove items, moving
to the next again if necessary, until the specified number of items have
been removed or all the stores are empty.
2.5) Then call the garbage collector, but only if some memory items were
freed.
3) Sleep for an interval half that in which the memory could run out.
3.1) If the remaining heap could be more than half filled during the
interval specified by the 'maxsleep' parameter (at the max rate of
memory consumption) then sleep for the time it would take to only half
fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval.
The effect of the store-janitor parameters in cocoon.xconf are now
slightly changed, so I've changed the names of the parameters to reflect
this. For the moment the names are changed only inside
StoreJanitorImpl.java. If the solution were to be adopted then the names
in cocoon.xconf would also change. The changes, with notes, are as follows:
'maxheap' (was 'heapsize') The maximum total heap size below which
stores are guaranteed to remain intact.
'minfree' (was 'freememory') The minimum free heap required for stores
to remain intact. I have found this should be set to less than 10% of
avaliable memory, so that the jvm will respond to low memory by
allocating more heap (if its available), before stores are likely to be
reduced. Default value might be 2000000. Setting it to zero or very low
will effectively disable store reduction.
'maxsleep' (was 'cleanupthreadinterval') The maximum interval between
checks. Should be short enough to ensure that 'rate of change' data is
collected before free memory runs low. Suggest 5 secs as default.
'reduceby' (was 'percent_to_free') is the number of items to be removed
from the stores, on each attempt to reduce the stores. Each successfull
attempt to reduce stores results in a call to the garbage collector
taking several seconds. This limits the rate a which the stores can be
emptied. Removing several items at a time compensates for this. Using
the debug output, choose a number of items which results in an increase
of free memory between successive store reductions. Best tested with a
sudden high demand for memory when the stores are full. In practice this
setting is not critical. Suggested default is 10 items. Ideally
'reduceby' would be in bytes!
Setting 'minfree' and 'reduceby' both too low can intensify gc activity
if sudden high demand requires all stores to be completely freed and can
result in memory running out before all the stores are free. Verify
using debug output.
'priority' remains unchanged.
The above solution is implemented and tested and it works very well
indeed for me. The parameters do not seem to be at all critical - thank
goodness! I see no reason why the default settings would need to be
changed if more memory were available. I have attached the
StoreJanitorImpl.java so others can test it. I find I can get a very
good picture of what is happening by using my editor's repeat find
command on the debug output in the core.log.000001 file. Optimizeit is
no longer necessary to prove that it works effectively.
Beware! - there might still be things I don't know about that I have not
take into account.
Ideally - low memory should be detected by some sort of interrupt or
exception rather than by polling.
If you are subsequently interested in committing these changes into
Cocoon and you want me to do some more work to conform to design
practice etc., then please let me know.
Very humble appologies, Gerhard (and others), for intruding into your
code. If you like my changes, I hope you feel that I have released your
brilliance (not undermined it). I feel that by far the biggest part of
any credit should very definately still be yours (I only fixed the
checker) :-).
Feedback very welcome please.
Pete H.
g-froehlich@gmx.de wrote:
> Hi Team,
> I hope I fixed the problem now!
>
> I changed the StoreJanitorImpl, I try to explain in
> prosa:
>
> <not to serious>
> Cocoon reaches 1 million request per second and
> memory is getting dangerously low. Time for the
> StoreJanitor to kick in and do something. First
> he forces the GC. Damn, memory is still low,
> we have to remove object from the Store to get
> some bytes free. Panic, which Store shall we take?
> Ahh index is at -1 so we take the first at index 0
> in the StoreArray. Gosh this one is pretty full how
> much objects shall we kick out? Stupid question,
> take 10% from all as configured. Chacka, did ya see
> them fly?
> Next time we take the next store on index 1.
> Yeah great work dudes, take a rest.
> </not to serious>
>
> Test it!
>
> Gerhard
>
> ------------------------------------------------
> If patience is a virtue, and ignorance is bliss,
> you can have a pretty good life if you're stupid
> and willing to wait.
> ------------------------------------------------
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
> For additional commands, email: cocoon-dev-help@xml.apache.org
>
>