You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jcs-users@jakarta.apache.org by Matthew Cooke <mp...@lineone.net> on 2004/04/25 13:22:47 UTC
Additional problems.
The original problem was that the removal code was commented out in the src.
Then I needed to change the code to use keys for removal (I believe this
is already fixed in your latest version)
Finally the remotecache.ccf file had the client call back port commented
out in the example.
We have now run the Cache for 2 days with fairly high put/get volumes in
a production enviroment with 2 machines doing puts and 12 machines doing
gets.
The whole solution falls over after a couple days with Out Of Memorys
and the remotecache needs restarting. the 12 machines all started
running slowly and retrieval from JCS was not returning valid contents.
Apache also then started backlogging causing every machine to go tits up.
The data we are caching comes to about 10Meg maybe 20 at most and the
remotecache was running with 350Meg memory!
I noticed in your example you had given the remotecache 1.8gigs ram. Is
this because you were storing more data, because JCS is very inefficient
or due to a leak?
Now I have 12 machines connected to the remote cache does it require 12x
the memory?
Wearily,
Matt
PS Up at 8.30am on a sunday restarting servers.
Travis Savo wrote:
>Your cache.ccf looks fine.
>
>If I understand you correctly, the problem is the original put works fine,
>but subsequent puts for that same region/key don't update the object
>remotely, correct?
>
>I'll have to test this scenario locally and get back to you.
>
>Writing unit tests for remote cache has been rapidly approaching the top of
>my todo stack, and this may bump it up another notch, depending on what the
>rest of my work load is like.
>
>Things to try right off the bat: Turn on logging. See the puts getting done
>to remote cache, and the removes getting broadcast from remote cache and
>received on the other clients? You should....
>
>-Travis Savo
>
>-----Original Message-----
>From: Matthew Cooke [mailto:mpcooke3@lineone.net]
>Sent: Monday, April 19, 2004 1:34 PM
>To: Turbine JCS Users List
>Subject: JCS remote cache problems
>
>
>Travis (or anyone else!),
>
>I'm having some problems with the remote cache implementation, it seems
>that items aren't getting updated on the servers after a put is
>performed on one machine.
>I'm not sure what the problem is, I've got a feeling only the first put
>is working and I didn't notice because the cached content doesn't change
>that fast in our system.
>
>This could be a bug in my usage of JCS or in the config file it may well
>not be a bug in the improved JCS.
>What might cause items to not be deleted in the connected machines
>caches when a new put occurs?
>
>Any ideas or suggestions much appreciated, I will be trying to reproduce
>this problem back in testing again tomorrow!
>
>Kind Regards,
>Matt.
>
>PS Below I attach first the ccf file used by most machines and below
>that the remote.ccf file used on the RemoteServer
>
>
>
># Normal config for MFN Production
>servers
>
>
>##############################################################
>################## DEFAULT CACHE REGION #####################
># sets the default aux value for any non configured caches
>jcs.default=DC,RC
>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
>jcs.default.cacheattributes.MaxObjects=1000
>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru
>.LRUMemoryCache
>jcs.default.cacheattributes.UseMemoryShrinker=true
>jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
>jcs.default.elementattributes.IsEternal=false
>jcs.default.elementattributes.MaxLifeSeconds=7
>jcs.default.elementattributes.IdleTime=1800
>jcs.default.elementattributes.IsSpool=true
>jcs.default.elementattributes.IsRemote=true
>jcs.default.elementattributes.IsLateral=true
>
>
>
># SYSTEM CACHE
># should be defined for the storage of group attribute list
>jcs.system.groupIdCache=DC,RC
>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeCache
>Attributes
>jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.engin
>e.memory.lru.LRUMemoryCache
>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.ElementAttri
>butes
>jcs.system.groupIdCache.elementattributes.IsEternal=true
>jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
>jcs.system.groupIdCache.elementattributes.IdleTime=1800
>jcs.system.groupIdCache.elementattributes.IsSpool=true
>jcs.system.groupIdCache.elementattributes.IsRemote=true
>jcs.system.groupIdCache.elementattributes.IsLateral=true
>
>
>
>
>
>
>##############################################################
>################## CACHE REGIONS AVAILABLE ###################
># Regions preconfirgured for caching
>
>
>
>jcs.region.primaryCache=DC,RC
>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.CompositeCache
>Attributes
>jcs.region.primaryCache.cacheattributes.MaxObjects=4000
>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.engin
>e.memory.lru.LRUMemoryCache
>jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
>jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
>jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.ElementAttri
>butes
>jcs.region.primaryCache.elementattributes.IsEternal=false
>jcs.region.primaryCache.elementattributes.MaxLifeSeconds=259200
>jcs.region.primaryCache.elementattributes.IsSpool=true
>jcs.region.primaryCache.elementattributes.IsRemote=true
>jcs.region.primaryCache.elementattributes.IsLateral=true
>
>
>
>##############################################################
>################## AUXILIARY CACHES AVAILABLE ################
>
>
>
># Disk Cache used as an overflow cache when the main memory cache is full.
>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFacto
>ry
>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDis
>kCacheAttributes
>jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/jcscache
>
>
>
># This is an RMI based remote cache.
>jcs.auxiliary.RC=org.apache.jcs.auxiliary.remote.RemoteCacheFactory
>jcs.auxiliary.RC.attributes=org.apache.jcs.auxiliary.remote.RemoteCacheAttri
>butes
>jcs.auxiliary.RC.attributes.RemoteHost=builder
>jcs.auxiliary.RC.attributes.RemotePort=1099
>jcs.auxiliary.RC.attributes.LocalPort=1103
>jcs.auxiliary.RC.attributes.RemoveUponRemotePut=false
>#jcs.auxiliary.RC.attributes.RemoteServiceName=RemoteCache
>
>
>
>-----------
>
>
>
>
>
># Config for MFN Central JCS server.
>
>
>
>##############################################################
>################## DEFAULT CACHE REGION #####################
># Registry used to register and provide the
># IRemoteCacheService service.
>registry.host=localhost
>registry.port=1099
>
>
>
>
>
>
>##############################################################
>################## DEFAULT CACHE REGION #####################
># sets the default aux value for any non configured caches
>jcs.default=DC
>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
>jcs.default.cacheattributes.MaxObjects=1000
>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru
>.LRUMemoryCache
>jcs.default.cacheattributes.UseMemoryShrinker=true
>jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
>jcs.default.elementattributes.IsEternal=false
>jcs.default.elementattributes.MaxLifeSeconds=7
>jcs.default.elementattributes.IdleTime=1800
>jcs.default.elementattributes.IsSpool=true
>jcs.default.elementattributes.IsRemote=true
>jcs.default.elementattributes.IsLateral=true
>
>
>
># SYSTEM CACHE
># should be defined for the storage of group attribute list
>jcs.system.groupIdCache=DC
>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeCache
>Attributes
>jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.engin
>e.memory.lru.LRUMemoryCache
>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.ElementAttri
>butes
>jcs.system.groupIdCache.elementattributes.IsEternal=true
>jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
>jcs.system.groupIdCache.elementattributes.IdleTime=1800
>jcs.system.groupIdCache.elementattributes.IsSpool=true
>jcs.system.groupIdCache.elementattributes.IsRemote=true
>jcs.system.groupIdCache.elementattributes.IsLateral=true
>
>
>
>
>
>
>##############################################################
>################## CACHE REGIONS AVAILABLE ###################
># Regions preconfirgured for caching
>
>
>
>jcs.region.primaryCache=DC
>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.CompositeCache
>Attributes
>jcs.region.primaryCache.cacheattributes.MaxObjects=10
>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.engin
>e.memory.lru.LRUMemoryCache
>jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
>jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
>jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.ElementAttri
>butes
>jcs.region.primaryCache.elementattributes.IsEternal=false
>jcs.region.primaryCache.elementattributes.MaxLifeSeconds=60000
>jcs.region.primaryCache.elementattributes.IsSpool=true
>jcs.region.primaryCache.elementattributes.IsRemote=true
>jcs.region.primaryCache.elementattributes.IsLateral=true
>
>
>
># Primary Disk Cache-- faster than the rest because of memory key storage
>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFacto
>ry
>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDis
>kCacheAttributes
>jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/MasterJcsCache
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
RE: Additional problems.
Posted by Aaron Smuts <aa...@wisc.edu>.
> >
> > I feel like I missed the first part of this conversation.
> >
> > So you have the updated code?
>
> Well I uncommented out the code that was clearly mean't to
> be in there. The part where the remote server recieves a
> "put" and then iterates over all the clients and sends them
> remove commands. I can do a diff check if there is any other
> patches since I recieved the code.
>
>
I meant, do you have the code from cvs with the patches from last week.
I'm not sure what commented section you are talking about.
> > Can you paste your configuration files into an email.
>
> Apart from uncommenting the callback ports they haven't
> changed much, on monday I will repost the remote and normal
> ccf files.
>
Are you running the memory shrinker? If so turn it off and see if
things improve.
Are the local caches and the remote cache using a lot of memory, just
the locals, or just the remote?
> >>The data we are caching comes to about 10Meg maybe 20 at most and
the
> >>remotecache was running with 350Meg memory!
> >
> >
> > What kind of stuff are you caching?
>
> Binary data byte[]. Mainly this binary data is actually HTML
> pages probably varying in size between 4k and 10k but some
> of the binary data is image data (gifs). both are put in
> simply as byte[]'s
>
>
> It is using the disk cache, however we only cache around
> 3000-4000 items. I increased the memory cache size to 4000
> as I thought the whole cache should fit easily into memory.
> Since I did this the disk cache remains empty. FYI when the
> memory cache was size 10 the size of the disk cache was
> about 10meg.
>
> All the machines have the same items which only come to
> around 10meg in size. 2 machines are generating updated
> content and frequently putting new data into JCS to replace
> old (with the same keys of course). So around 60% of the
> data in the cache is overwritten with new data every
> 15minutes or so. This is because we process realtime-ish
> data and need to update the html in the cache to reflect
> this data.
>
> Even if items were never expiring based on time then it
> shouldn't be a big problem as new items are rare.
>
So you are using the shrinker. Turn it off. I think your old ccf file
was in your message. It had the maxlife set to 7 seconds or something
very low and the shrinker on. If you have constant updates, just let
the updates replace the old content and anything that expires die when
it is queried for now.
jcs.default.cacheattributes.UseMemoryShrinker=false
Let me know if killing the shrinker helps. Something must be keeping
references to the discarded items.
Aaron
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
RE: Additional problems.
Posted by Aaron Smuts <aa...@wisc.edu>.
I'm not supporting any distribution that didn't come from cvs.
Cheers,
Aaron
> -----Original Message-----
> From: Matthew Cooke [mailto:mpcooke3@lineone.net]
> Sent: Monday, April 26, 2004 5:05 PM
> To: Turbine JCS Users List
> Subject: Re: Additional problems.
>
> I'm using the code I got from Travis - probably quite an
> early verion he put on his website. No other custom stuff. I
> realise that Travis's code at the time was not checked into
> CVS - is it in now? I was too rushed to wait for a CVS check
> in as I'd just wasted a lot of time trying to use the
> definately buggy LTCP distribution mechanism.
>
> I haven't had time today to look into it much more. I have
> increased the max size for the cache to 800MB.
> We have half a day scheduled in this week to profile the
> live JCS cache with jprofiler so I should have more info
> after that if it's just a problem we can fix via config
> changes or if something more weird is going on.
>
> I'll get back to you then, hopefully with something more
> concrete.
>
> Thanks,
> Matt.
>
> Aaron Smuts wrote:
> > Are you using any element event handlers that might be keeping
> > references to deleted objects?
> >
> > I've looked over the shrinker again and can't imagine how it would
cause
> > anything like this, but turn it off anyway.
> >
> > Are you getting any errors in the logs?
> >
> > Aaron
> >
> >
> >>-----Original Message-----
> >>From: Matthew Cooke [mailto:mpcooke3@lineone.net]
> >>Sent: Sunday, April 25, 2004 2:12 PM
> >>To: Turbine JCS Users List; Support
> >>Subject: Re: Additional problems.
> >>
> >
> >
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: turbine-jcs-user-
> help@jakarta.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
Re: Additional problems.
Posted by Matthew Cooke <mp...@lineone.net>.
I'm using the code I got from Travis - probably quite an
early verion he put on his website. No other custom stuff. I
realise that Travis's code at the time was not checked into
CVS - is it in now? I was too rushed to wait for a CVS check
in as I'd just wasted a lot of time trying to use the
definately buggy LTCP distribution mechanism.
I haven't had time today to look into it much more. I have
increased the max size for the cache to 800MB.
We have half a day scheduled in this week to profile the
live JCS cache with jprofiler so I should have more info
after that if it's just a problem we can fix via config
changes or if something more weird is going on.
I'll get back to you then, hopefully with something more
concrete.
Thanks,
Matt.
Aaron Smuts wrote:
> Are you using any element event handlers that might be keeping
> references to deleted objects?
>
> I've looked over the shrinker again and can't imagine how it would cause
> anything like this, but turn it off anyway.
>
> Are you getting any errors in the logs?
>
> Aaron
>
>
>>-----Original Message-----
>>From: Matthew Cooke [mailto:mpcooke3@lineone.net]
>>Sent: Sunday, April 25, 2004 2:12 PM
>>To: Turbine JCS Users List; Support
>>Subject: Re: Additional problems.
>>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
RE: Additional problems.
Posted by Aaron Smuts <aa...@wisc.edu>.
Are you using any element event handlers that might be keeping
references to deleted objects?
I've looked over the shrinker again and can't imagine how it would cause
anything like this, but turn it off anyway.
Are you getting any errors in the logs?
Aaron
> -----Original Message-----
> From: Matthew Cooke [mailto:mpcooke3@lineone.net]
> Sent: Sunday, April 25, 2004 2:12 PM
> To: Turbine JCS Users List; Support
> Subject: Re: Additional problems.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
Re: Additional problems.
Posted by Matthew Cooke <mp...@lineone.net>.
>>
>>The original problem was that the removal code was commented out in
>
> the
>
>>src.
>>Then I needed to change the code to use keys for removal (I believe
>
> this
>
>>is already fixed in your latest version)
>
>
> I feel like I missed the first part of this conversation.
>
> So you have the updated code?
Well I uncommented out the code that was clearly mean't to
be in there. The part where the remote server recieves a
"put" and then iterates over all the clients and sends them
remove commands. I can do a diff check if there is any other
patches since I recieved the code.
> Can you paste your configuration files into an email.
Apart from uncommenting the callback ports they haven't
changed much, on monday I will repost the remote and normal
ccf files.
>>The data we are caching comes to about 10Meg maybe 20 at most and the
>>remotecache was running with 350Meg memory!
>
>
> What kind of stuff are you caching?
Binary data byte[]. Mainly this binary data is actually HTML
pages probably varying in size between 4k and 10k but some
of the binary data is image data (gifs). both are put in
simply as byte[]'s
>>I noticed in your example you had given the remotecache 1.8gigs ram.
>
> Is
>
>>this because you were storing more data, because JCS is very
>
> inefficient
>
>>or due to a leak?
>
>
> I gave the remote cache that much memory because I had it and it was
> connected to many machines.
>
> Clearly, the situation you describe shouldn't be happening. Is the
> remote cache using the disk cache? Is the disk file growing?
It is using the disk cache, however we only cache around
3000-4000 items. I increased the memory cache size to 4000
as I thought the whole cache should fit easily into memory.
Since I did this the disk cache remains empty. FYI when the
memory cache was size 10 the size of the disk cache was
about 10meg.
>
>>Now I have 12 machines connected to the remote cache does it require
>
> 12x
>
>>the memory?
>
>
> If all 12 machines have unique data and you have the memory set to some
> huge number then it should be even bigger, since the cache itself
> requires some memory to run.
All the machines have the same items which only come to
around 10meg in size. 2 machines are generating updated
content and frequently putting new data into JCS to replace
old (with the same keys of course). So around 60% of the
data in the cache is overwritten with new data every
15minutes or so. This is because we process realtime-ish
data and need to update the html in the cache to reflect
this data.
Even if items were never expiring based on time then it
shouldn't be a big problem as new items are rare.
>
> Send the .ccf files.
Will do.
Thanks,
Matt
>
> Aaron
>
>
>>Wearily,
>>Matt
>>PS Up at 8.30am on a sunday restarting servers.
>>
>>
>>Travis Savo wrote:
>>
>>
>>>Your cache.ccf looks fine.
>>>
>>>If I understand you correctly, the problem is the original put works
>>
>>fine,
>>
>>>but subsequent puts for that same region/key don't update the object
>>>remotely, correct?
>>>
>>>I'll have to test this scenario locally and get back to you.
>>>
>>>Writing unit tests for remote cache has been rapidly approaching the
>
> top
>
>>of
>>
>>>my todo stack, and this may bump it up another notch, depending on
>
> what
>
>>the
>>
>>>rest of my work load is like.
>>>
>>>Things to try right off the bat: Turn on logging. See the puts
>
> getting
>
>>done
>>
>>>to remote cache, and the removes getting broadcast from remote cache
>
> and
>
>>>received on the other clients? You should....
>>>
>>>-Travis Savo
>>>
>>>-----Original Message-----
>>>From: Matthew Cooke [mailto:mpcooke3@lineone.net]
>>>Sent: Monday, April 19, 2004 1:34 PM
>>>To: Turbine JCS Users List
>>>Subject: JCS remote cache problems
>>>
>>>
>>>Travis (or anyone else!),
>>>
>>>I'm having some problems with the remote cache implementation, it
>
> seems
>
>>>that items aren't getting updated on the servers after a put is
>>>performed on one machine.
>>>I'm not sure what the problem is, I've got a feeling only the first
>
> put
>
>>>is working and I didn't notice because the cached content doesn't
>
> change
>
>>>that fast in our system.
>>>
>>>This could be a bug in my usage of JCS or in the config file it may
>
> well
>
>>>not be a bug in the improved JCS.
>>>What might cause items to not be deleted in the connected machines
>>>caches when a new put occurs?
>>>
>>>Any ideas or suggestions much appreciated, I will be trying to
>
> reproduce
>
>>>this problem back in testing again tomorrow!
>>>
>>>Kind Regards,
>>>Matt.
>>>
>>>PS Below I attach first the ccf file used by most machines and below
>>>that the remote.ccf file used on the RemoteServer
>>>
>>>
>>>
>>># Normal config for MFN Production
>>>servers
>>>
>>>
>>>##############################################################
>>>################## DEFAULT CACHE REGION #####################
>>># sets the default aux value for any non configured caches
>>>jcs.default=DC,RC
>>
>>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribu
>
> te
>
>>s
>>
>>>jcs.default.cacheattributes.MaxObjects=1000
>>
>>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memor
>
> y.
>
>>lru
>>
>>>.LRUMemoryCache
>>>jcs.default.cacheattributes.UseMemoryShrinker=true
>>>jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
>>>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>>>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>>>jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
>>>jcs.default.elementattributes.IsEternal=false
>>>jcs.default.elementattributes.MaxLifeSeconds=7
>>>jcs.default.elementattributes.IdleTime=1800
>>>jcs.default.elementattributes.IsSpool=true
>>>jcs.default.elementattributes.IsRemote=true
>>>jcs.default.elementattributes.IsLateral=true
>>>
>>>
>>>
>>># SYSTEM CACHE
>>># should be defined for the storage of group attribute list
>>>jcs.system.groupIdCache=DC,RC
>>
>>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.Composite
>
> Ca
>
>>che
>>
>>>Attributes
>>>jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>>
>>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.
>
> en
>
>>gin
>>
>>>e.memory.lru.LRUMemoryCache
>>
>>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.Element
>
> At
>
>>tri
>>
>>>butes
>>>jcs.system.groupIdCache.elementattributes.IsEternal=true
>>>jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
>>>jcs.system.groupIdCache.elementattributes.IdleTime=1800
>>>jcs.system.groupIdCache.elementattributes.IsSpool=true
>>>jcs.system.groupIdCache.elementattributes.IsRemote=true
>>>jcs.system.groupIdCache.elementattributes.IsLateral=true
>>>
>>>
>>>
>>>
>>>
>>>
>>>##############################################################
>>>################## CACHE REGIONS AVAILABLE ###################
>>># Regions preconfirgured for caching
>>>
>>>
>>>
>>>jcs.region.primaryCache=DC,RC
>>
>>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.Composite
>
> Ca
>
>>che
>>
>>>Attributes
>>>jcs.region.primaryCache.cacheattributes.MaxObjects=4000
>>
>>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.
>
> en
>
>>gin
>>
>>>e.memory.lru.LRUMemoryCache
>>>jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
>>>jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
>>>jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>>
>>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.Element
>
> At
>
>>tri
>>
>>>butes
>>>jcs.region.primaryCache.elementattributes.IsEternal=false
>>>jcs.region.primaryCache.elementattributes.MaxLifeSeconds=259200
>>>jcs.region.primaryCache.elementattributes.IsSpool=true
>>>jcs.region.primaryCache.elementattributes.IsRemote=true
>>>jcs.region.primaryCache.elementattributes.IsLateral=true
>>>
>>>
>>>
>>>##############################################################
>>>################## AUXILIARY CACHES AVAILABLE ################
>>>
>>>
>>>
>>># Disk Cache used as an overflow cache when the main memory cache is
>>
>>full.
>>
>>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache
>
> Fa
>
>>cto
>>
>>>ry
>>
>>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Index
>
> ed
>
>>Dis
>>
>>>kCacheAttributes
>>>jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/jcscache
>>>
>>>
>>>
>>># This is an RMI based remote cache.
>>>jcs.auxiliary.RC=org.apache.jcs.auxiliary.remote.RemoteCacheFactory
>>
>>jcs.auxiliary.RC.attributes=org.apache.jcs.auxiliary.remote.RemoteCache
>
> At
>
>>tri
>>
>>>butes
>>>jcs.auxiliary.RC.attributes.RemoteHost=builder
>>>jcs.auxiliary.RC.attributes.RemotePort=1099
>>>jcs.auxiliary.RC.attributes.LocalPort=1103
>>>jcs.auxiliary.RC.attributes.RemoveUponRemotePut=false
>>>#jcs.auxiliary.RC.attributes.RemoteServiceName=RemoteCache
>>>
>>>
>>>
>>>-----------
>>>
>>>
>>>
>>>
>>>
>>># Config for MFN Central JCS server.
>>>
>>>
>>>
>>>##############################################################
>>>################## DEFAULT CACHE REGION #####################
>>># Registry used to register and provide the
>>># IRemoteCacheService service.
>>>registry.host=localhost
>>>registry.port=1099
>>>
>>>
>>>
>>>
>>>
>>>
>>>##############################################################
>>>################## DEFAULT CACHE REGION #####################
>>># sets the default aux value for any non configured caches
>>>jcs.default=DC
>>
>>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribu
>
> te
>
>>s
>>
>>>jcs.default.cacheattributes.MaxObjects=1000
>>
>>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memor
>
> y.
>
>>lru
>>
>>>.LRUMemoryCache
>>>jcs.default.cacheattributes.UseMemoryShrinker=true
>>>jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
>>>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>>>jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
>>>jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
>>>jcs.default.elementattributes.IsEternal=false
>>>jcs.default.elementattributes.MaxLifeSeconds=7
>>>jcs.default.elementattributes.IdleTime=1800
>>>jcs.default.elementattributes.IsSpool=true
>>>jcs.default.elementattributes.IsRemote=true
>>>jcs.default.elementattributes.IsLateral=true
>>>
>>>
>>>
>>># SYSTEM CACHE
>>># should be defined for the storage of group attribute list
>>>jcs.system.groupIdCache=DC
>>
>>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.Composite
>
> Ca
>
>>che
>>
>>>Attributes
>>>jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>>
>>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.
>
> en
>
>>gin
>>
>>>e.memory.lru.LRUMemoryCache
>>
>>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.Element
>
> At
>
>>tri
>>
>>>butes
>>>jcs.system.groupIdCache.elementattributes.IsEternal=true
>>>jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
>>>jcs.system.groupIdCache.elementattributes.IdleTime=1800
>>>jcs.system.groupIdCache.elementattributes.IsSpool=true
>>>jcs.system.groupIdCache.elementattributes.IsRemote=true
>>>jcs.system.groupIdCache.elementattributes.IsLateral=true
>>>
>>>
>>>
>>>
>>>
>>>
>>>##############################################################
>>>################## CACHE REGIONS AVAILABLE ###################
>>># Regions preconfirgured for caching
>>>
>>>
>>>
>>>jcs.region.primaryCache=DC
>>
>>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.Composite
>
> Ca
>
>>che
>>
>>>Attributes
>>>jcs.region.primaryCache.cacheattributes.MaxObjects=10
>>
>>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.
>
> en
>
>>gin
>>
>>>e.memory.lru.LRUMemoryCache
>>>jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
>>>jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
>>>jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>>
>>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.Element
>
> At
>
>>tri
>>
>>>butes
>>>jcs.region.primaryCache.elementattributes.IsEternal=false
>>>jcs.region.primaryCache.elementattributes.MaxLifeSeconds=60000
>>>jcs.region.primaryCache.elementattributes.IsSpool=true
>>>jcs.region.primaryCache.elementattributes.IsRemote=true
>>>jcs.region.primaryCache.elementattributes.IsLateral=true
>>>
>>>
>>>
>>># Primary Disk Cache-- faster than the rest because of memory key
>
> storage
>
>>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache
>
> Fa
>
>>cto
>>
>>>ry
>>
>>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Index
>
> ed
>
>>Dis
>>
>>>kCacheAttributes
>>>jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/MasterJcsCache
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail:
>
> turbine-jcs-user-unsubscribe@jakarta.apache.org
>
>>>For additional commands, e-mail:
>
> turbine-jcs-user-help@jakarta.apache.org
>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail:
>
> turbine-jcs-user-unsubscribe@jakarta.apache.org
>
>>>For additional commands, e-mail:
>
> turbine-jcs-user-help@jakarta.apache.org
>
>>>
>>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail:
>
> turbine-jcs-user-unsubscribe@jakarta.apache.org
>
>>For additional commands, e-mail:
>
> turbine-jcs-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
RE: Additional problems.
Posted by Aaron Smuts <aa...@wisc.edu>.
> -----Original Message-----
> From: Matthew Cooke [mailto:mpcooke3@lineone.net]
> Sent: Sunday, April 25, 2004 6:23 AM
> To: Turbine JCS Users List
> Subject: Additional problems.
>
> The original problem was that the removal code was commented out in
the
> src.
> Then I needed to change the code to use keys for removal (I believe
this
> is already fixed in your latest version)
I feel like I missed the first part of this conversation.
So you have the updated code?
> Finally the remotecache.ccf file had the client call back port
commented
> out in the example.
>
> We have now run the Cache for 2 days with fairly high put/get volumes
in
> a production enviroment with 2 machines doing puts and 12 machines
doing
> gets.
>
> The whole solution falls over after a couple days with Out Of Memorys
> and the remotecache needs restarting. the 12 machines all started
> running slowly and retrieval from JCS was not returning valid
contents.
> Apache also then started backlogging causing every machine to go tits
up.
>
Can you paste your configuration files into an email.
> The data we are caching comes to about 10Meg maybe 20 at most and the
> remotecache was running with 350Meg memory!
What kind of stuff are you caching?
>
> I noticed in your example you had given the remotecache 1.8gigs ram.
Is
> this because you were storing more data, because JCS is very
inefficient
> or due to a leak?
I gave the remote cache that much memory because I had it and it was
connected to many machines.
Clearly, the situation you describe shouldn't be happening. Is the
remote cache using the disk cache? Is the disk file growing?
> Now I have 12 machines connected to the remote cache does it require
12x
> the memory?
If all 12 machines have unique data and you have the memory set to some
huge number then it should be even bigger, since the cache itself
requires some memory to run.
Send the .ccf files.
Aaron
>
> Wearily,
> Matt
> PS Up at 8.30am on a sunday restarting servers.
>
>
> Travis Savo wrote:
>
> >Your cache.ccf looks fine.
> >
> >If I understand you correctly, the problem is the original put works
> fine,
> >but subsequent puts for that same region/key don't update the object
> >remotely, correct?
> >
> >I'll have to test this scenario locally and get back to you.
> >
> >Writing unit tests for remote cache has been rapidly approaching the
top
> of
> >my todo stack, and this may bump it up another notch, depending on
what
> the
> >rest of my work load is like.
> >
> >Things to try right off the bat: Turn on logging. See the puts
getting
> done
> >to remote cache, and the removes getting broadcast from remote cache
and
> >received on the other clients? You should....
> >
> >-Travis Savo
> >
> >-----Original Message-----
> >From: Matthew Cooke [mailto:mpcooke3@lineone.net]
> >Sent: Monday, April 19, 2004 1:34 PM
> >To: Turbine JCS Users List
> >Subject: JCS remote cache problems
> >
> >
> >Travis (or anyone else!),
> >
> >I'm having some problems with the remote cache implementation, it
seems
> >that items aren't getting updated on the servers after a put is
> >performed on one machine.
> >I'm not sure what the problem is, I've got a feeling only the first
put
> >is working and I didn't notice because the cached content doesn't
change
> >that fast in our system.
> >
> >This could be a bug in my usage of JCS or in the config file it may
well
> >not be a bug in the improved JCS.
> >What might cause items to not be deleted in the connected machines
> >caches when a new put occurs?
> >
> >Any ideas or suggestions much appreciated, I will be trying to
reproduce
> >this problem back in testing again tomorrow!
> >
> >Kind Regards,
> >Matt.
> >
> >PS Below I attach first the ccf file used by most machines and below
> >that the remote.ccf file used on the RemoteServer
> >
> >
> >
> ># Normal config for MFN Production
> >servers
> >
> >
> >##############################################################
> >################## DEFAULT CACHE REGION #####################
> ># sets the default aux value for any non configured caches
> >jcs.default=DC,RC
>
>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribu
te
> s
> >jcs.default.cacheattributes.MaxObjects=1000
>
>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memor
y.
> lru
> >.LRUMemoryCache
> >jcs.default.cacheattributes.UseMemoryShrinker=true
> >jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
> >jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
> >jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
> >jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
> >jcs.default.elementattributes.IsEternal=false
> >jcs.default.elementattributes.MaxLifeSeconds=7
> >jcs.default.elementattributes.IdleTime=1800
> >jcs.default.elementattributes.IsSpool=true
> >jcs.default.elementattributes.IsRemote=true
> >jcs.default.elementattributes.IsLateral=true
> >
> >
> >
> ># SYSTEM CACHE
> ># should be defined for the storage of group attribute list
> >jcs.system.groupIdCache=DC,RC
>
>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.Composite
Ca
> che
> >Attributes
> >jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>
>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.
en
> gin
> >e.memory.lru.LRUMemoryCache
>
>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.Element
At
> tri
> >butes
> >jcs.system.groupIdCache.elementattributes.IsEternal=true
> >jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
> >jcs.system.groupIdCache.elementattributes.IdleTime=1800
> >jcs.system.groupIdCache.elementattributes.IsSpool=true
> >jcs.system.groupIdCache.elementattributes.IsRemote=true
> >jcs.system.groupIdCache.elementattributes.IsLateral=true
> >
> >
> >
> >
> >
> >
> >##############################################################
> >################## CACHE REGIONS AVAILABLE ###################
> ># Regions preconfirgured for caching
> >
> >
> >
> >jcs.region.primaryCache=DC,RC
>
>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.Composite
Ca
> che
> >Attributes
> >jcs.region.primaryCache.cacheattributes.MaxObjects=4000
>
>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.
en
> gin
> >e.memory.lru.LRUMemoryCache
> >jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
> >jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
> >jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>
>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.Element
At
> tri
> >butes
> >jcs.region.primaryCache.elementattributes.IsEternal=false
> >jcs.region.primaryCache.elementattributes.MaxLifeSeconds=259200
> >jcs.region.primaryCache.elementattributes.IsSpool=true
> >jcs.region.primaryCache.elementattributes.IsRemote=true
> >jcs.region.primaryCache.elementattributes.IsLateral=true
> >
> >
> >
> >##############################################################
> >################## AUXILIARY CACHES AVAILABLE ################
> >
> >
> >
> ># Disk Cache used as an overflow cache when the main memory cache is
> full.
>
>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache
Fa
> cto
> >ry
>
>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Index
ed
> Dis
> >kCacheAttributes
> >jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/jcscache
> >
> >
> >
> ># This is an RMI based remote cache.
> >jcs.auxiliary.RC=org.apache.jcs.auxiliary.remote.RemoteCacheFactory
>
>jcs.auxiliary.RC.attributes=org.apache.jcs.auxiliary.remote.RemoteCache
At
> tri
> >butes
> >jcs.auxiliary.RC.attributes.RemoteHost=builder
> >jcs.auxiliary.RC.attributes.RemotePort=1099
> >jcs.auxiliary.RC.attributes.LocalPort=1103
> >jcs.auxiliary.RC.attributes.RemoveUponRemotePut=false
> >#jcs.auxiliary.RC.attributes.RemoteServiceName=RemoteCache
> >
> >
> >
> >-----------
> >
> >
> >
> >
> >
> ># Config for MFN Central JCS server.
> >
> >
> >
> >##############################################################
> >################## DEFAULT CACHE REGION #####################
> ># Registry used to register and provide the
> ># IRemoteCacheService service.
> >registry.host=localhost
> >registry.port=1099
> >
> >
> >
> >
> >
> >
> >##############################################################
> >################## DEFAULT CACHE REGION #####################
> ># sets the default aux value for any non configured caches
> >jcs.default=DC
>
>jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribu
te
> s
> >jcs.default.cacheattributes.MaxObjects=1000
>
>jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memor
y.
> lru
> >.LRUMemoryCache
> >jcs.default.cacheattributes.UseMemoryShrinker=true
> >jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=3600
> >jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
> >jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
> >jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
> >jcs.default.elementattributes.IsEternal=false
> >jcs.default.elementattributes.MaxLifeSeconds=7
> >jcs.default.elementattributes.IdleTime=1800
> >jcs.default.elementattributes.IsSpool=true
> >jcs.default.elementattributes.IsRemote=true
> >jcs.default.elementattributes.IsLateral=true
> >
> >
> >
> ># SYSTEM CACHE
> ># should be defined for the storage of group attribute list
> >jcs.system.groupIdCache=DC
>
>jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.Composite
Ca
> che
> >Attributes
> >jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
>
>jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.
en
> gin
> >e.memory.lru.LRUMemoryCache
>
>jcs.system.groupIdCache.elementattributes=org.apache.jcs.engine.Element
At
> tri
> >butes
> >jcs.system.groupIdCache.elementattributes.IsEternal=true
> >jcs.system.groupIdCache.elementattributes.MaxLifeSeconds=3600
> >jcs.system.groupIdCache.elementattributes.IdleTime=1800
> >jcs.system.groupIdCache.elementattributes.IsSpool=true
> >jcs.system.groupIdCache.elementattributes.IsRemote=true
> >jcs.system.groupIdCache.elementattributes.IsLateral=true
> >
> >
> >
> >
> >
> >
> >##############################################################
> >################## CACHE REGIONS AVAILABLE ###################
> ># Regions preconfirgured for caching
> >
> >
> >
> >jcs.region.primaryCache=DC
>
>jcs.region.primaryCache.cacheattributes=org.apache.jcs.engine.Composite
Ca
> che
> >Attributes
> >jcs.region.primaryCache.cacheattributes.MaxObjects=10
>
>jcs.region.primaryCache.cacheattributes.MemoryCacheName=org.apache.jcs.
en
> gin
> >e.memory.lru.LRUMemoryCache
> >jcs.region.primaryCache.cacheattributes.UseMemoryShrinker=true
> >jcs.region.primaryCache.cacheattributes.MaxMemoryIdleTimeSeconds=10
> >jcs.region.primaryCache.cacheattributes.ShrinkerIntervalSeconds=60
>
>jcs.region.primaryCache.elementattributes=org.apache.jcs.engine.Element
At
> tri
> >butes
> >jcs.region.primaryCache.elementattributes.IsEternal=false
> >jcs.region.primaryCache.elementattributes.MaxLifeSeconds=60000
> >jcs.region.primaryCache.elementattributes.IsSpool=true
> >jcs.region.primaryCache.elementattributes.IsRemote=true
> >jcs.region.primaryCache.elementattributes.IsLateral=true
> >
> >
> >
> ># Primary Disk Cache-- faster than the rest because of memory key
storage
>
>jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache
Fa
> cto
> >ry
>
>jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Index
ed
> Dis
> >kCacheAttributes
> >jcs.auxiliary.DC.attributes.DiskPath=/usr/local/web/MasterJcsCache
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org