1.2.4-dev fix -- RE: Excessive number of JCS objects in memory

[Aaron Smuts <aa...@wisc.edu>]

I'm pretty sure that there is no memory leak with the disk cache in
version 1.2.4-dev.  I recommend upgrading to this version.  The release
is in the tempbuild folder and the repository is tagged.

http://cvs.apache.org/viewcvs.cgi/jakarta-turbine-jcs/tempbuild/

I've been running in pure disk mode (max memory size = 0) for some time
and with a limit placed on the number of disk cache keys, I can keep the
memory steady.  It increased 5k after 1 million operations (a combo of
puts, gets, and removes) but I think this is due to additional stats
data.  

I added a gc call in the manual tester program that works nicely.  Here
is the stats and gc info after one test:


Called system.gc()
enter command:
stats
Region Name = testCache1
HitCountRam = 0
HitCountAux = 366613
---------------------------LRU Memory Cache
List Size = 0
Map Size = 0
Put Count = 315742
Hit Count = 0
Miss Count = 590662
---------------------------Indexed Disk Cache
Is Alive = true
Key Map Size = 10000
Data File Length = 13515586
Optimize Opertaion Count = 249597
Times Optimized = 0
Recycle Count = 146319
Purgatory Hits = 0
Purgatory Size = 0
Working = true
Alive = false
Empty = true
Size = 0
enter command:
gc
[Full GC 2019K->2006K(91520K), 0.1179804 secs]
[Full GC 2006K->2006K(91520K), 0.1187335 secs]
Called system.gc()
enter command:
gc
[Full GC 2006K->2006K(91520K), 0.1178323 secs]
[Full GC 2006K->2006K(91520K), 0.1176138 secs]
Called system.gc()



My disk cache configuration is here:

# Primary Disk Cache-- faster than the rest because of memory key
storage
jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheF
actory
jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Indexe
dDiskCacheAttributes
jcs.auxiliary.DC.attributes.DiskPath=target/test-sandbox/raf
jcs.auxiliary.DC.attributes.MaxPurgatorySize=10000
jcs.auxiliary.DC.attributes.MaxKeySize=10000
jcs.auxiliary.DC.attributes.MaxRecycleBinSize=5000
jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300000

I never optimized, in my test.  I suggest setting OptimizeAtRemoveCount
to -1 for now.

My region config is here:

##############################################################
################## CACHE REGIONS AVAILABLE ###################
# Regions preconfirgured for caching
jcs.region.testCache1=DC
jcs.region.testCache1.cacheattributes=org.apache.jcs.engine.CompositeCac
heAttributes
jcs.region.testCache1.cacheattributes.MaxObjects=0
jcs.region.testCache1.cacheattributes.MemoryCacheName=org.apache.jcs.eng
ine.memory.lru.LRUMemoryCache
jcs.region.testCache1.cacheattributes.UseMemoryShrinker=true
jcs.region.testCache1.cacheattributes.ShrinkerIntervalSeconds=30
jcs.region.testCache1.cacheattributes.MaxMemoryIdleTimeSeconds=300
jcs.region.testCache1.cacheattributes.MaxSpoolPerRun=100
jcs.region.testCache1.elementattributes=org.apache.jcs.engine.ElementAtt
ributes
jcs.region.testCache1.elementattributes.IsEternal=false
jcs.region.testCache1.elementattributes.MaxLifeSeconds=60000
jcs.region.testCache1.elementattributes.IsLateral=true
jcs.region.testCache1.elementattributes.IsRemote=true

> -----Original Message-----
> From: Wyatt, Allen [mailto:Allen.Wyatt@travelocity.com]
> Sent: Monday, January 31, 2005 4:39 PM
> To: Turbine JCS Users List
> Subject: RE: Excessive number of JCS objects in memory
> 
> Regarding using newer JCS - which one?  What is the latest "stable"
> version - is it "jcs-1.2.3-dev.jar" (the "-dev" in the name makes me
> think not).
> 
> Right now we're using JCS 1.1 (downloaded Oct 19 2004) and commons
> collections 2.1.
> 
> Putting heavy load on the disk cache (each of the 100s of user
sessions
> to each webserver on our website has the potential to put something in
> the disk cache as well as other parts of the application putting stuff
> into it).
> 
> Thanks for all the help.
> 
> -----Original Message-----
> From: Smuts, Aaron [mailto:aaronsm@amazon.com]
> Sent: Monday, January 31, 2005 5:52 PM
> To: Turbine JCS Users List
> Subject: RE: Excessive number of JCS objects in memory
> 
> The abstract disk cache still uses that old write lock for each item
put
> into the disk cache.  The indexed disk cache used to use it.  The Core
> disk cache class now uses the Doug Lea read write lock classes.
> 
> I will convert the abstact disk cache to use the Doug Lea read write
> lock also.
> 
> What version of JCS are you using?  You should probably upgrade JCS
and
> the commons collections libraries.  Your problem earlier looked like a
> commons issue.
> 
> What kind of load are you putting on the disk cache?
> 
> Aaron
> 
> -----Original Message-----
> From: Wyatt, Allen [mailto:Allen.Wyatt@travelocity.com]
> Sent: Monday, January 31, 2005 3:05 PM
> To: turbine-jcs-user@jakarta.apache.org
> Subject: Excessive number of JCS objects in memory
> 
> We're having a problem with a slow memory leak in production.  We took
a
> heapdump and saw that there were 337,999
> org.apache.jcs.utils.locking.ReadWriteLock objects in memory and
337,967
> org.apache.jcs.utils.locking.RwLockHolder objects in memory.  These
> objects are taking about 21.5 megabytes of memory altogether.  Any
> reason for having this many of these objects?  What might cause there
to
> be an excessive number of these objects?
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org

RE: 1.2.4-dev fix -- RE: Excessive number of JCS objects in memory

[Aaron Smuts <aa...@wisc.edu>]

I'm running with commons collections 3.0 with jcs 1.2.4-dev.  The
collections LRUMap is used for disk key storage and purgatory.  We have
our own LRU for the memory cache, which performs better.

I'll add the stats data to the admin jsp soon.  

I'm also trying to beat the hell out of the tcp lateral cache.  It's
damn fast and seems to recover nicely when a node goes down. 

It is very difficult to unit test the remote and lateral auxiliaries
since the cache only works as a singleton.  Any suggestions are welcome.

Aaron

> -----Original Message-----
> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> Sent: Monday, January 31, 2005 9:04 PM
> To: 'Turbine JCS Users List'
> Subject: 1.2.4-dev fix -- RE: Excessive number of JCS objects in
memory
> 
> I'm pretty sure that there is no memory leak with the disk cache in
> version 1.2.4-dev.  I recommend upgrading to this version.  The
release
> is in the tempbuild folder and the repository is tagged.
> 
> http://cvs.apache.org/viewcvs.cgi/jakarta-turbine-jcs/tempbuild/
> 
> I've been running in pure disk mode (max memory size = 0) for some
time
> and with a limit placed on the number of disk cache keys, I can keep
the
> memory steady.  It increased 5k after 1 million operations (a combo of
> puts, gets, and removes) but I think this is due to additional stats
> data.
> 
> I added a gc call in the manual tester program that works nicely.
Here
> is the stats and gc info after one test:
> 
> 
> Called system.gc()
> enter command:
> stats
> Region Name = testCache1
> HitCountRam = 0
> HitCountAux = 366613
> ---------------------------LRU Memory Cache
> List Size = 0
> Map Size = 0
> Put Count = 315742
> Hit Count = 0
> Miss Count = 590662
> ---------------------------Indexed Disk Cache
> Is Alive = true
> Key Map Size = 10000
> Data File Length = 13515586
> Optimize Opertaion Count = 249597
> Times Optimized = 0
> Recycle Count = 146319
> Purgatory Hits = 0
> Purgatory Size = 0
> Working = true
> Alive = false
> Empty = true
> Size = 0
> enter command:
> gc
> [Full GC 2019K->2006K(91520K), 0.1179804 secs]
> [Full GC 2006K->2006K(91520K), 0.1187335 secs]
> Called system.gc()
> enter command:
> gc
> [Full GC 2006K->2006K(91520K), 0.1178323 secs]
> [Full GC 2006K->2006K(91520K), 0.1176138 secs]
> Called system.gc()
> 
> 
> 
> My disk cache configuration is here:
> 
> # Primary Disk Cache-- faster than the rest because of memory key
> storage
>
jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheF
> actory
>
jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Indexe
> dDiskCacheAttributes
> jcs.auxiliary.DC.attributes.DiskPath=target/test-sandbox/raf
> jcs.auxiliary.DC.attributes.MaxPurgatorySize=10000
> jcs.auxiliary.DC.attributes.MaxKeySize=10000
> jcs.auxiliary.DC.attributes.MaxRecycleBinSize=5000
> jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300000
> 
> I never optimized, in my test.  I suggest setting
OptimizeAtRemoveCount
> to -1 for now.
> 
> My region config is here:
> 
> ##############################################################
> ################## CACHE REGIONS AVAILABLE ###################
> # Regions preconfirgured for caching
> jcs.region.testCache1=DC
>
jcs.region.testCache1.cacheattributes=org.apache.jcs.engine.CompositeCac
> heAttributes
> jcs.region.testCache1.cacheattributes.MaxObjects=0
>
jcs.region.testCache1.cacheattributes.MemoryCacheName=org.apache.jcs.eng
> ine.memory.lru.LRUMemoryCache
> jcs.region.testCache1.cacheattributes.UseMemoryShrinker=true
> jcs.region.testCache1.cacheattributes.ShrinkerIntervalSeconds=30
> jcs.region.testCache1.cacheattributes.MaxMemoryIdleTimeSeconds=300
> jcs.region.testCache1.cacheattributes.MaxSpoolPerRun=100
>
jcs.region.testCache1.elementattributes=org.apache.jcs.engine.ElementAtt
> ributes
> jcs.region.testCache1.elementattributes.IsEternal=false
> jcs.region.testCache1.elementattributes.MaxLifeSeconds=60000
> jcs.region.testCache1.elementattributes.IsLateral=true
> jcs.region.testCache1.elementattributes.IsRemote=true
> 
> > -----Original Message-----
> > From: Wyatt, Allen [mailto:Allen.Wyatt@travelocity.com]
> > Sent: Monday, January 31, 2005 4:39 PM
> > To: Turbine JCS Users List
> > Subject: RE: Excessive number of JCS objects in memory
> >
> > Regarding using newer JCS - which one?  What is the latest "stable"
> > version - is it "jcs-1.2.3-dev.jar" (the "-dev" in the name makes me
> > think not).
> >
> > Right now we're using JCS 1.1 (downloaded Oct 19 2004) and commons
> > collections 2.1.
> >
> > Putting heavy load on the disk cache (each of the 100s of user
> sessions
> > to each webserver on our website has the potential to put something
in
> > the disk cache as well as other parts of the application putting
stuff
> > into it).
> >
> > Thanks for all the help.
> >
> > -----Original Message-----
> > From: Smuts, Aaron [mailto:aaronsm@amazon.com]
> > Sent: Monday, January 31, 2005 5:52 PM
> > To: Turbine JCS Users List
> > Subject: RE: Excessive number of JCS objects in memory
> >
> > The abstract disk cache still uses that old write lock for each item
> put
> > into the disk cache.  The indexed disk cache used to use it.  The
Core
> > disk cache class now uses the Doug Lea read write lock classes.
> >
> > I will convert the abstact disk cache to use the Doug Lea read write
> > lock also.
> >
> > What version of JCS are you using?  You should probably upgrade JCS
> and
> > the commons collections libraries.  Your problem earlier looked like
a
> > commons issue.
> >
> > What kind of load are you putting on the disk cache?
> >
> > Aaron
> >
> > -----Original Message-----
> > From: Wyatt, Allen [mailto:Allen.Wyatt@travelocity.com]
> > Sent: Monday, January 31, 2005 3:05 PM
> > To: turbine-jcs-user@jakarta.apache.org
> > Subject: Excessive number of JCS objects in memory
> >
> > We're having a problem with a slow memory leak in production.  We
took
> a
> > heapdump and saw that there were 337,999
> > org.apache.jcs.utils.locking.ReadWriteLock objects in memory and
> 337,967
> > org.apache.jcs.utils.locking.RwLockHolder objects in memory.  These
> > objects are taking about 21.5 megabytes of memory altogether.  Any
> > reason for having this many of these objects?  What might cause
there
> to
> > be an excessive number of these objects?
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org