You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jcs-users@jakarta.apache.org by Dean Holdren <de...@gmail.com> on 2004/11/11 17:27:25 UTC
java.lang.OutOfMemoryError for more than 13 regions
I built JCS from source (using latest code) and I get a
java.lang.OutOfMemoryError when first accessing a region (I believe
inside CompositeCacheConfigurator). This only occurs (for my
configuration) if I specify more than 13 regions.
Using a build from 8/20/2003, I can have over *100* regions (that's as
high as I tested) without this error.
My test looks like the following:
**********************************************************
JCS.setConfigFilename("/myCacheRegions.ccf");
JCS.getInstance("region1"); //<---fails
**********************************************************
myCacheRegions.ccf:
jcs.default=indexedDiskCache
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=100
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
# SYSTEM GROUP ID CACHE
jcs.system.groupIdCache=indexedDiskCache
jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
##### CACHE REGIONS FOR TEST
jcs.region.indexedRegion1=indexedDiskCache
jcs.region.indexedRegion1.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.region.indexedRegion1.cacheattributes.MaxObjects=100
jcs.region.indexedRegion1.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
#####...... (More regions, each configured like above example)
##### AUXILIARY CACHES
# Indexed Disk Cache
jcs.auxiliary.indexedDiskCache=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory
jcs.auxiliary.indexedDiskCache.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
jcs.auxiliary.indexedDiskCache.attributes.DiskPath=c:/cache
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
RE: java.lang.OutOfMemoryError for more than 13 regions
Posted by Aaron Smuts <aa...@wisc.edu>.
This is configurable (see the cache.ccf in src/conf), but that is way
too large as a default. Here are some new disk cache config parameters:
#new disk cache parameter.
jcs.auxiliary.DC.attributes.maxKeySize=10000
jcs.auxiliary.DC.attributes.optimizeAtRemoveCount=300
I'm setting the default to 5000 for now. I'll have the fix in tonight.
Thanks,
Aaron
> -----Original Message-----
> From: Dean Holdren [mailto:deanholdren@gmail.com]
> Sent: Monday, November 15, 2004 8:21 AM
> To: turbine-jcs-user@jakarta.apache.org; turbine-jcs-
> dev@jakarta.apache.org
> Subject: Re: java.lang.OutOfMemoryError for more than 13 regions
>
> I found the source of this memory bloat:
> IndexedDiskCacheAttributes has a property "maxKeySize" hardcoded at
> 500000.
> When each IndexedDiskCache is initialized, a HashMap and a
> SortedPreferentialArray are both initialized with a size of
> "maxKeySize" (500k). This uses aproximately 4MB of the heap PER
> IndexedDiskCache. With 14 regions using an IndexedDiskCache, this
> value is 56MB. So if I had a Heap of 64MB (default, which is what I
> used for testing), and 8MB was used for other objects (I'll assume),
> initializing these 14 regions put me over the limit of the Heap.
>
> Question is, why does the IndexedDiskCache need 4MB of memory for each
> instance? This seems grossly bloated. And why is there no way to
> configure the "maxKeySize" for an IndexedDiskCache at least? If the
> HashMap and SortedPreferentialArray need to be initialized to that
> amount for efficiency, then the size should be configurable so it
> doesn't take up so much memory unnecessarily.
>
> I apologize for posting in both lists, but JCS users should be aware
> of this problem.
>
>
> On Thu, 11 Nov 2004 11:27:25 -0500, Dean Holdren
<de...@gmail.com>
> wrote:
> > I built JCS from source (using latest code) and I get a
> > java.lang.OutOfMemoryError when first accessing a region (I believe
> > inside CompositeCacheConfigurator). This only occurs (for my
> > configuration) if I specify more than 13 regions.
> >
> > Using a build from 8/20/2003, I can have over *100* regions (that's
as
> > high as I tested) without this error.
> >
> > My test looks like the following:
> > **********************************************************
> > JCS.setConfigFilename("/myCacheRegions.ccf");
> > JCS.getInstance("region1"); //<---fails
> > **********************************************************
> >
> > myCacheRegions.ccf:
> >
> > jcs.default=indexedDiskCache
> >
>
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribut
es
> > jcs.default.cacheattributes.MaxObjects=100
> >
>
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory
.l
> ru.LRUMemoryCache
> >
> > # SYSTEM GROUP ID CACHE
> > jcs.system.groupIdCache=indexedDiskCache
> >
>
jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeC
ac
> heAttributes
> > jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
> >
>
jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.e
ng
> ine.memory.lru.LRUMemoryCache
> >
> > ##### CACHE REGIONS FOR TEST
> >
> > jcs.region.indexedRegion1=indexedDiskCache
> >
>
jcs.region.indexedRegion1.cacheattributes=org.apache.jcs.engine.Composit
eC
> acheAttributes
> > jcs.region.indexedRegion1.cacheattributes.MaxObjects=100
> >
>
jcs.region.indexedRegion1.cacheattributes.MemoryCacheName=org.apache.jcs
.e
> ngine.memory.lru.LRUMemoryCache
> >
> > #####...... (More regions, each configured like above example)
> >
> > ##### AUXILIARY CACHES
> >
> > # Indexed Disk Cache
> >
>
jcs.auxiliary.indexedDiskCache=org.apache.jcs.auxiliary.disk.indexed.Ind
ex
> edDiskCacheFactory
> >
>
jcs.auxiliary.indexedDiskCache.attributes=org.apache.jcs.auxiliary.disk.
in
> dexed.IndexedDiskCacheAttributes
> > jcs.auxiliary.indexedDiskCache.attributes.DiskPath=c:/cache
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
new features in jcs-1.2-dev
Posted by Aaron Smuts <as...@yahoo.com>.
New in jcs-1.2-dev:
http://cvs.apache.org/viewcvs.cgi/*checkout*/jakarta-turbine-jcs/tempbuild/jcs-1.2-dev.jar?rev=1.1
1. The disk cache now has more configuration options
exposed. You can set the purgatory size, recycle bin
size, etc. Also, you can set a bound on the
purgatory. This solves the possibility or an ever
growing purgatory when used with a bounded queue.
The disk cache options are described on the web site.
http://jakarta.apache.org/turbine/jcs/IndexedDiskAuxCache.html
Here are the parameters:
jcs.auxiliary.DC.attributes.MaxPurgatorySize=10000
jcs.auxiliary.DC.attributes.MaxKeySize=10000
jcs.auxiliary.DC.attributes.MaxRecycleBinSize=5000
jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300000
2. Optional, configurable thread pools can be set for
all auxiliaries. By default the single queue is used.
(Will document later.)
3. Memory shrinkers for all regions now share one
thread and are executed by a Doug Lea Clock Daemon.
This helps keep the number of threads used by the
cache to a minimum.
4. As of version 1.1.2-dev, the potential lock on
remove and put is solved.
5. As of version 1.1.2-dev, better statistics are
gathered. I will update the JCSAmin.jsp to dump the
new data and to stop counting the byte size of a
region by default.
6. As of version 1.3-dev, you can specify the remote
gets to timeout by using a configurable thread pool.
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
Re: java.lang.OutOfMemoryError for more than 13 regions
Posted by Dean Holdren <de...@gmail.com>.
I found the source of this memory bloat:
IndexedDiskCacheAttributes has a property "maxKeySize" hardcoded at 500000.
When each IndexedDiskCache is initialized, a HashMap and a
SortedPreferentialArray are both initialized with a size of
"maxKeySize" (500k). This uses aproximately 4MB of the heap PER
IndexedDiskCache. With 14 regions using an IndexedDiskCache, this
value is 56MB. So if I had a Heap of 64MB (default, which is what I
used for testing), and 8MB was used for other objects (I'll assume),
initializing these 14 regions put me over the limit of the Heap.
Question is, why does the IndexedDiskCache need 4MB of memory for each
instance? This seems grossly bloated. And why is there no way to
configure the "maxKeySize" for an IndexedDiskCache at least? If the
HashMap and SortedPreferentialArray need to be initialized to that
amount for efficiency, then the size should be configurable so it
doesn't take up so much memory unnecessarily.
I apologize for posting in both lists, but JCS users should be aware
of this problem.
On Thu, 11 Nov 2004 11:27:25 -0500, Dean Holdren <de...@gmail.com> wrote:
> I built JCS from source (using latest code) and I get a
> java.lang.OutOfMemoryError when first accessing a region (I believe
> inside CompositeCacheConfigurator). This only occurs (for my
> configuration) if I specify more than 13 regions.
>
> Using a build from 8/20/2003, I can have over *100* regions (that's as
> high as I tested) without this error.
>
> My test looks like the following:
> **********************************************************
> JCS.setConfigFilename("/myCacheRegions.ccf");
> JCS.getInstance("region1"); //<---fails
> **********************************************************
>
> myCacheRegions.ccf:
>
> jcs.default=indexedDiskCache
> jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.default.cacheattributes.MaxObjects=100
> jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> # SYSTEM GROUP ID CACHE
> jcs.system.groupIdCache=indexedDiskCache
> jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
> jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> ##### CACHE REGIONS FOR TEST
>
> jcs.region.indexedRegion1=indexedDiskCache
> jcs.region.indexedRegion1.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.region.indexedRegion1.cacheattributes.MaxObjects=100
> jcs.region.indexedRegion1.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> #####...... (More regions, each configured like above example)
>
> ##### AUXILIARY CACHES
>
> # Indexed Disk Cache
> jcs.auxiliary.indexedDiskCache=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory
> jcs.auxiliary.indexedDiskCache.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
> jcs.auxiliary.indexedDiskCache.attributes.DiskPath=c:/cache
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
Re: java.lang.OutOfMemoryError for more than 13 regions
Posted by Dean Holdren <de...@gmail.com>.
I found the source of this memory bloat:
IndexedDiskCacheAttributes has a property "maxKeySize" hardcoded at 500000.
When each IndexedDiskCache is initialized, a HashMap and a
SortedPreferentialArray are both initialized with a size of
"maxKeySize" (500k). This uses aproximately 4MB of the heap PER
IndexedDiskCache. With 14 regions using an IndexedDiskCache, this
value is 56MB. So if I had a Heap of 64MB (default, which is what I
used for testing), and 8MB was used for other objects (I'll assume),
initializing these 14 regions put me over the limit of the Heap.
Question is, why does the IndexedDiskCache need 4MB of memory for each
instance? This seems grossly bloated. And why is there no way to
configure the "maxKeySize" for an IndexedDiskCache at least? If the
HashMap and SortedPreferentialArray need to be initialized to that
amount for efficiency, then the size should be configurable so it
doesn't take up so much memory unnecessarily.
I apologize for posting in both lists, but JCS users should be aware
of this problem.
On Thu, 11 Nov 2004 11:27:25 -0500, Dean Holdren <de...@gmail.com> wrote:
> I built JCS from source (using latest code) and I get a
> java.lang.OutOfMemoryError when first accessing a region (I believe
> inside CompositeCacheConfigurator). This only occurs (for my
> configuration) if I specify more than 13 regions.
>
> Using a build from 8/20/2003, I can have over *100* regions (that's as
> high as I tested) without this error.
>
> My test looks like the following:
> **********************************************************
> JCS.setConfigFilename("/myCacheRegions.ccf");
> JCS.getInstance("region1"); //<---fails
> **********************************************************
>
> myCacheRegions.ccf:
>
> jcs.default=indexedDiskCache
> jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.default.cacheattributes.MaxObjects=100
> jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> # SYSTEM GROUP ID CACHE
> jcs.system.groupIdCache=indexedDiskCache
> jcs.system.groupIdCache.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.system.groupIdCache.cacheattributes.MaxObjects=10000
> jcs.system.groupIdCache.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> ##### CACHE REGIONS FOR TEST
>
> jcs.region.indexedRegion1=indexedDiskCache
> jcs.region.indexedRegion1.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
> jcs.region.indexedRegion1.cacheattributes.MaxObjects=100
> jcs.region.indexedRegion1.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
>
> #####...... (More regions, each configured like above example)
>
> ##### AUXILIARY CACHES
>
> # Indexed Disk Cache
> jcs.auxiliary.indexedDiskCache=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory
> jcs.auxiliary.indexedDiskCache.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
> jcs.auxiliary.indexedDiskCache.attributes.DiskPath=c:/cache
>
---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-dev-help@jakarta.apache.org