You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Gerhard Froehlich <g-...@gmx.de> on 2001/08/28 16:31:06 UTC
[c2] EventCache questions
Hi,
When starting cocoon2 2 instances of the EventCache(MRUMemoryStore) are
created from the CachingEventPipeline and CachingStreamPipeline. For my
understanding, now 2 different instances are existing. One for the
C.Stream.P
and another for the C.Event.P.
-> 2 checker Threads, which are checking mem in the jvm and free() mem
-> 2 writer Threads, which serializing objects to the fs.
Is this correct? Is this necessary?
I think one instance of the EventCache would be sufficiently, because the
EventCache
it doesn't matter which pipline, class, etc stores objects.
Cheers
Gerhard
--
Gerhard Fröhlich
g-froehlich@gmx.de
"black holes are,
when GOD is dividing by zero"
GMX - Die Kommunikationsplattform im Internet.
http://www.gmx.net
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: [C2 VOTE] StoreJanitor
Posted by Vadim Gritsenko <vg...@hns.com>.
I think you should go ahead with your changes.
I will not be around for a while, so don't count on my help for a week or two,
hope somebody else from committers would help you.
Regards,
Vadim
> -----Original Message-----
> From: Gerhard Froehlich [mailto:g-froehlich@gmx.de]
> Sent: Wednesday, August 29, 2001 6:28 PM
> To: cocoon-dev@xml.apache.org
> Subject: [C2 VOTE] StoreJanitor
>
>
> Hi Team,
> I hope I do everything right now with this voting. First I want
> to say, that following idea is a summing of different postings
> from several peoples few days ago. I don't want to steal someones
> idea and I hope no one is feeling repositioned (esp. Christian,Peter)!
>
> Few days ago I mentioned, that the EventCache (MRUMemoryStore) class
> is initialize twice. This has the effect, that two cleanupthreads cleaning
> up memory. That can be a performance issue. So the initial idea from Peter Royal
> was to write a "StoreJanitor" which is responsible for the memory clean up.
> Every Store class (MRUMemoryStore, MemoryStore,...) must register to the
> StoreJanitor. The StoreJanitor, see prototype from Christian Schmitt, will
> cleanup memory by calling the free() method of the different registerd stores
> with some alghorithm. The precondition is, that every store must implement the
> free() method. So we have to broaden the Store interface by a public free() method.
> With the StoreJanitor we can have _hundreds_ :) different stores, but we have
> one intelligent background process which keeps an eye on memory.
>
> Cheers
> Gerhard
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
> For additional commands, email: cocoon-dev-help@xml.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
[C2 VOTE] StoreJanitor
Posted by Gerhard Froehlich <g-...@gmx.de>.
Hi Team,
I hope I do everything right now with this voting. First I want
to say, that following idea is a summing of different postings
from several peoples few days ago. I don't want to steal someones
idea and I hope no one is feeling repositioned (esp. Christian,Peter)!
Few days ago I mentioned, that the EventCache (MRUMemoryStore) class
is initialize twice. This has the effect, that two cleanupthreads cleaning
up memory. That can be a performance issue. So the initial idea from Peter Royal
was to write a "StoreJanitor" which is responsible for the memory clean up.
Every Store class (MRUMemoryStore, MemoryStore,...) must register to the
StoreJanitor. The StoreJanitor, see prototype from Christian Schmitt, will
cleanup memory by calling the free() method of the different registerd stores
with some alghorithm. The precondition is, that every store must implement the
free() method. So we have to broaden the Store interface by a public free() method.
With the StoreJanitor we can have _hundreds_ :) different stores, but we have
one intelligent background process which keeps an eye on memory.
Cheers
Gerhard
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: [c2] EventCache questions
Posted by Christian Schmitt <cs...@ffzj0ia9.bank.dresdner.net>.
On Wed, Aug 29, 2001 at 07:29:58PM +0200, Gerhard Froehlich wrote:
> Cool idea, but some things are missing I think:
> - Now the the StoreJanitor is the master of memory. So he must do the
> finalization and forcing the gc(). I added a freeMemory() method in
> your code and logic.
> - you have to loop till memory is normal. I changed the while loop
> a little bit and added some comments. What happens, when the loop comes
> to the end of your list of stores and memory is still low?
>
> Sorry I just overlooked it in few minutes (less time today). Tomorrow I
> will take a closer look.
>
> I added my changes to this postings. But this is just a prototype!
>
> Cheers
> Gerhard
Hi,
thanks for your comments. As I said, this thing is FAR from anywhere
near complete. I just wnated to get some feedback, whether this is
an idea I should pursue or not waste time on it.
Cheers,
Christian
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
> For additional commands, email: cocoon-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
AW: [c2] EventCache questions
Posted by Gerhard Froehlich <g-...@gmx.de>.
>Hi there,
>I've been following this thread with quite some interest.
>
>I like the idea of one monitoring thread that checks memory
>and then calls on the different stores to free some up if
>necessary.
>
>As an attachment I included a (very rough, first version, pre-alpha...
>you get it;-) StoreJanitor.
>
>To get this to run in your environment do the following:
>1) add a new role named "janitor" to org/apache/cocoon/cocoon.roles
>2) add an <janitor> element to cocoon.xconf,
> similar to the MRUMemoryStore ones
>3) Expand the Store interface by a "public void free()" method.
>4) Add the Store to the Janitor surveillance, i.e.:
> StoreJanitor janitor = (StoreJanitor)manager.lookup(StoreJanitor.ROLE);
> janitor.addToSurveillance(this);
>
>As mentioned above, this is far from complete. I'm sending this in
>so people that are more knowledgable than I am can take a look at
>it.
>
>I'd appreciate any (rocks!, sucks) feedback.
Cool idea, but some things are missing I think:
- Now the the StoreJanitor is the master of memory. So he must do the
finalization and forcing the gc(). I added a freeMemory() method in
your code and logic.
- you have to loop till memory is normal. I changed the while loop
a little bit and added some comments. What happens, when the loop comes
to the end of your list of stores and memory is still low?
Sorry I just overlooked it in few minutes (less time today). Tomorrow I
will take a closer look.
I added my changes to this postings. But this is just a prototype!
Cheers
Gerhard
Re: [c2] EventCache questions
Posted by Christian Schmitt <cs...@ffzj0ia9.bank.dresdner.net>.
Hi there,
I've been following this thread with quite some interest.
I like the idea of one monitoring thread that checks memory
and then calls on the different stores to free some up if
necessary.
As an attachment I included a (very rough, first version, pre-alpha...
you get it;-) StoreJanitor.
To get this to run in your environment do the following:
1) add a new role named "janitor" to org/apache/cocoon/cocoon.roles
2) add an <janitor> element to cocoon.xconf,
similar to the MRUMemoryStore ones
3) Expand the Store interface by a "public void free()" method.
4) Add the Store to the Janitor surveillance, i.e.:
StoreJanitor janitor = (StoreJanitor)manager.lookup(StoreJanitor.ROLE);
janitor.addToSurveillance(this);
As mentioned above, this is far from complete. I'm sending this in
so people that are more knowledgable than I am can take a look at
it.
I'd appreciate any (rocks!, sucks) feedback.
Thanks,
Christian
On Wed, Aug 29, 2001 at 12:18:46AM +0200, Gerhard Froehlich wrote:
> >> >Wow, where have you been before? :)))
> Always there watching and learning much through this project and its
> participants :). But most of the time in the dungeons of commercial
> software development and its deadlines.
> >> >Yes, you are right - that's not necessary and this might be the
> >> >performance issue.
> >> >Reducing checker threads amount down to one would decrease amount of
> >> >runFinalization()/gc()
> >> >calls in two times - giving better performance.
> >> >
> >> >Could we use just one static checker thread?
> We have to test this. When static, we have namely one cleanup thread but
> still two stores to free(). For the moment I don't know if this work.
> But I have the strong feeling that it won't. Too bad that Carsten is
> away for a couple of weeks. He would certainly have some good ideas for
> the pipline configuration. But I will test this with a static thread.
> >> >There is an issue with configuration though... Which
> >configuration to use,
> >> >from what store...
> Maybe we can add a sample configuration to the cocoon.xconf according
> to a reference config for the Xms Xmx parameters. Berin made some
> good suggestions some threads below. I can add it to the cocoon.xconf.
> >> How about a separate component to do the checking?
> >(MRUMemoryStoreJanitor?)
> >> Each MRUMemoryStore would register its requested heapsize &
> >freememory with
> >> the janitor, and it would be responsible for the worker thread. It would
> >
> >Good suggestion and I kind of like this name ;)
> >Gerhard, what do you think?
> Good suggestion. But I would name it StoreJanitor. Because the
> MRUMemoryStore
> is *only* a component. Maybe there will be a LRUMemoryStore or combination
> of
> both in nearer future :). Every store component implements the store
> interface, so
> the StoreJanitor would have access to the free() method of every store
> component.
> Then we would have a seperate cleanup component for every store
> implementation.
> I would implement the stores as singelton classes with one instance for
> better
> control about the memory. Because, how do you decide from which store
> instance
> to free memory. Maybe there is one with many small objects and another with
> less
> but huge objects. You can't determine, like in C, the byte size of each
> object in java easy.
> >> try to keep the JVM totalMemory below the sum of all the registered
> >> heapsizes, and freeMemory below the sum of all registered freememories.
> >Here I think that idea (originally) was that heapsize specifies
> >heapsize of the JVM,
> >but not the size of the store (because it is (almost) impossible
> >to determine), so my
> >understanding is that these parameters (heapsize and freememory)
> >should also
> >go into janitor configuration.
> Yeah, thats right. But I had the opinion that there is only one store
> instance.
> (I tested it too carelessly :( )
> This gc()->jvm.FreeMemory()->gc()->jvm.FreeMemory() is the proposed
> implementation (hack) from javasoft to determine memory consumption
> of objects in java.
> >> If runFinalization/gc isn't enough to create the free memory, a
> >round-robin
> >> approach (or other algorithm) could be used to select a
> >MRUMemoryStore to
> >> free an object from.
> Yes, that would be a possible approach to decide from which store to free().
> But
> I still don't understand why 2 or more stores?
> >> Just my .02 :)
> Mine too :)
>
> Cheers
> Gerhard
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
> For additional commands, email: cocoon-dev-help@xml.apache.org
>
AW: [c2] EventCache questions
Posted by Gerhard Froehlich <g-...@gmx.de>.
>> >Wow, where have you been before? :)))
Always there watching and learning much through this project and its
participants :). But most of the time in the dungeons of commercial
software development and its deadlines.
>> >Yes, you are right - that's not necessary and this might be the
>> >performance issue.
>> >Reducing checker threads amount down to one would decrease amount of
>> >runFinalization()/gc()
>> >calls in two times - giving better performance.
>> >
>> >Could we use just one static checker thread?
We have to test this. When static, we have namely one cleanup thread but
still two stores to free(). For the moment I don't know if this work.
But I have the strong feeling that it won't. Too bad that Carsten is
away for a couple of weeks. He would certainly have some good ideas for
the pipline configuration. But I will test this with a static thread.
>> >There is an issue with configuration though... Which
>configuration to use,
>> >from what store...
Maybe we can add a sample configuration to the cocoon.xconf according
to a reference config for the Xms Xmx parameters. Berin made some
good suggestions some threads below. I can add it to the cocoon.xconf.
>> How about a separate component to do the checking?
>(MRUMemoryStoreJanitor?)
>> Each MRUMemoryStore would register its requested heapsize &
>freememory with
>> the janitor, and it would be responsible for the worker thread. It would
>
>Good suggestion and I kind of like this name ;)
>Gerhard, what do you think?
Good suggestion. But I would name it StoreJanitor. Because the
MRUMemoryStore
is *only* a component. Maybe there will be a LRUMemoryStore or combination
of
both in nearer future :). Every store component implements the store
interface, so
the StoreJanitor would have access to the free() method of every store
component.
Then we would have a seperate cleanup component for every store
implementation.
I would implement the stores as singelton classes with one instance for
better
control about the memory. Because, how do you decide from which store
instance
to free memory. Maybe there is one with many small objects and another with
less
but huge objects. You can't determine, like in C, the byte size of each
object in java easy.
>> try to keep the JVM totalMemory below the sum of all the registered
>> heapsizes, and freeMemory below the sum of all registered freememories.
>Here I think that idea (originally) was that heapsize specifies
>heapsize of the JVM,
>but not the size of the store (because it is (almost) impossible
>to determine), so my
>understanding is that these parameters (heapsize and freememory)
>should also
>go into janitor configuration.
Yeah, thats right. But I had the opinion that there is only one store
instance.
(I tested it too carelessly :( )
This gc()->jvm.FreeMemory()->gc()->jvm.FreeMemory() is the proposed
implementation (hack) from javasoft to determine memory consumption
of objects in java.
>> If runFinalization/gc isn't enough to create the free memory, a
>round-robin
>> approach (or other algorithm) could be used to select a
>MRUMemoryStore to
>> free an object from.
Yes, that would be a possible approach to decide from which store to free().
But
I still don't understand why 2 or more stores?
>> Just my .02 :)
Mine too :)
Cheers
Gerhard
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: [c2] EventCache questions
Posted by Vadim Gritsenko <vg...@hns.com>.
> -----Original Message-----
> From: Peter Royal [mailto:proyal@managingpartners.com]
> Sent: Tuesday, August 28, 2001 11:09 AM
> To: cocoon-dev@xml.apache.org
> Subject: RE: [c2] EventCache questions
>
>
> At 10:48 AM 8/28/2001 -0400, you wrote:
> >Wow, where have you been before? :)))
> >Yes, you are right - that's not necessary and this might be the
> >performance issue.
> >Reducing checker threads amount down to one would decrease amount of
> >runFinalization()/gc()
> >calls in two times - giving better performance.
> >
> >Could we use just one static checker thread?
> >There is an issue with configuration though... Which configuration to use,
> >from what store...
>
> How about a separate component to do the checking? (MRUMemoryStoreJanitor?)
> Each MRUMemoryStore would register its requested heapsize & freememory with
> the janitor, and it would be responsible for the worker thread. It would
Good suggestion and I kind of like this name ;)
Gerhard, what do you think?
> try to keep the JVM totalMemory below the sum of all the registered
> heapsizes, and freeMemory below the sum of all registered freememories.
Here I think that idea (originally) was that heapsize specifies heapsize of the JVM,
but not the size of the store (because it is (almost) impossible to determine), so my
understanding is that these parameters (heapsize and freememory) should also
go into janitor configuration.
>
> If runFinalization/gc isn't enough to create the free memory, a round-robin
> approach (or other algorithm) could be used to select a MRUMemoryStore to
> free an object from.
Agree.
>
> Some of the current MRUMemoryStore parameters would now be Janitor
> parameters (namely cleanupthreadinterval and threadpriority).
heapsize and freememory also (see above)
>
> Just my .02 :)
Thanks,
Vadim
> -pete
>
> --
> peter royal -> proyal@managingpartners.com
> managing partners, inc. -> http://www.managingpartners.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: [c2] EventCache questions
Posted by Peter Royal <pr...@managingpartners.com>.
At 10:48 AM 8/28/2001 -0400, you wrote:
>Wow, where have you been before? :)))
>Yes, you are right - that's not necessary and this might be the
>performance issue.
>Reducing checker threads amount down to one would decrease amount of
>runFinalization()/gc()
>calls in two times - giving better performance.
>
>Could we use just one static checker thread?
>There is an issue with configuration though... Which configuration to use,
>from what store...
How about a separate component to do the checking? (MRUMemoryStoreJanitor?)
Each MRUMemoryStore would register its requested heapsize & freememory with
the janitor, and it would be responsible for the worker thread. It would
try to keep the JVM totalMemory below the sum of all the registered
heapsizes, and freeMemory below the sum of all registered freememories.
If runFinalization/gc isn't enough to create the free memory, a round-robin
approach (or other algorithm) could be used to select a MRUMemoryStore to
free an object from.
Some of the current MRUMemoryStore parameters would now be Janitor
parameters (namely cleanupthreadinterval and threadpriority).
Just my .02 :)
-pete
--
peter royal -> proyal@managingpartners.com
managing partners, inc. -> http://www.managingpartners.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: [c2] EventCache questions
Posted by Vadim Gritsenko <vg...@hns.com>.
> -----Original Message-----
> From: Gerhard Froehlich [mailto:g-froehlich@gmx.de]
> Sent: Tuesday, August 28, 2001 10:31 AM
> To: cocoon-dev@xml.apache.org
> Subject: [c2] EventCache questions
>
>
> Hi,
> When starting cocoon2 2 instances of the EventCache(MRUMemoryStore) are
> created from the CachingEventPipeline and CachingStreamPipeline. For my
> understanding, now 2 different instances are existing. One for the
> C.Stream.P
> and another for the C.Event.P.
> -> 2 checker Threads, which are checking mem in the jvm and free() mem
> -> 2 writer Threads, which serializing objects to the fs.
>
> Is this correct? Is this necessary?
Wow, where have you been before? :)))
Yes, you are right - that's not necessary and this might be the performance issue.
Reducing checker threads amount down to one would decrease amount of runFinalization()/gc()
calls in two times - giving better performance.
Could we use just one static checker thread?
There is an issue with configuration though... Which configuration to use, from what store...
>
> I think one instance of the EventCache would be sufficiently, because the
> EventCache
> it doesn't matter which pipline, class, etc stores objects.
I don't know is it OK or not and how to do this...
Vadim
>
> Cheers
> Gerhard
>
>
>
> --
> Gerhard Fröhlich
> g-froehlich@gmx.de
>
> "black holes are,
> when GOD is dividing by zero"
>
> GMX - Die Kommunikationsplattform im Internet.
> http://www.gmx.net
>
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org