new shrinker setting, new disk cache feature-- reducing memory spikes and dead disk keys RE: Out of memory problem

[Aaron Smuts <aa...@wisc.edu>]

You can cut down on the spikes now.  Just add another line to the
cache.ccf wherever you are using the shrinker.

By default it is set to -1, or no limit.

If you set this at a few hundred, you should fair better.

jcs.region.YOURREGIONNAME.cacheattributes.MaxSpoolPerRun=250


The indexed diskcache now has a maximum number of keys that it will
hold.  By default it is 500,000.  You can set it lower with this
parameter in your diskcache defintion:

jcs.auxiliary.DC.attributes.maxKeySize=10000

I'm using the commons LRUMap for now.  When it initializes to allow for
a huge maximum number, say a million, it eats memory.  Set it at 10k or
so, in most cases you shouldn't need more than that for a single region.

This is only part of a longterm disk cache plan of improvement.

Cheers,

Aaron


> -----Original Message-----
> From: mpcooke3@lineone.net [mailto:mpcooke3@lineone.net]
> Sent: Sunday, May 09, 2004 5:04 PM
> To: Turbine JCS Users List
> Subject: RE: Out of memory problem
> 
> This sounds plausible.
> 
> Running the profiler on our production machine for 30 minutes showed
heavy
> memory spikes but no leakage when the memory shrinker was enabled
(After
> a successfull GC it always seemed to go back to a normal level). I
also
> checked the memory shrinker code manually,  I didn't dig too deep into
the
> spooling but suspected this might be causing problems simply because
the
> rest of the memory shrinker code was so simple.
> 
> I would have liked to run the profiler for a longer period but
Jprofiler
> I was using is not stable enough to leave unattended on a production
> system
> (it crashes the jvm sometimes). So I was unable to confirm if it was a
> slow
> leak or sudden spike. I have seen other libraries OutOfMemory due to
heap
> usage spikes where we previously suspected a leak, so it's not
uncommon
> (in an XML processing library).
> 
> Good luck anyway,
> Matt.
> 
> >-- Original Message --
> >Reply-To: "Turbine JCS Users List"
<tu...@jakarta.apache.org>
> >Date: Sun, 09 May 2004 15:15:05 -0500
> >From: Aaron Smuts <aa...@wisc.edu>
> >Subject: RE: Out of memory problem
> >To: 'Turbine JCS Users List' <tu...@jakarta.apache.org>
> >
> >
> >I've been profiling JCS for days.  I still have more work to do, but
I
> >have a few ideas.
> >
> >The shrinker does not maintain references to any objects.  However,
> >there is a potential issue involved in using the shrinker, namely, it
> >causes sudden spikes in memory usage.
> >
> >The main purpose of the shrinker is to find elements that have
exceeded
> >their maximum idle memory time and then to spool these to disk.  It
also
> >checks for expired elements in the memory cache and removes them.
> >
> >When the shrinker finds elements to spool it initiates the spooling
> >process.  This invovles putting the items in purgatory and then they
are
> >queued to be written for disk.  The disk writing can take a while if
you
> >have thousands of items.  The shrinker will try to spool every item
in
> >memory that has exceeded the maxmemory time.  Basically, this can
result
> >in thousands of items in the queue which results in a memory spike
that
> >can exceed the maximum heap size.  All my tests indicate that the
memory
> >is reclaimed.
> >
> >I will add a configuration option to set the maximum number of items
to
> >spool per run of the shrinker.  This way the size of the spike can be
> >minimized.
> >
> >There is a potential memory issue invovled in using the indexed disk
> >cache.  The disk cache stores the keys in memory.  If an expired
element
> >is pulled from the disk cache or an item is removed from the cache,
the
> >key gets removed from the set held in memory by the disk cache.
> >However, if you create millions of entries in the cache that you
never
> >use, many of these get pushed onto the disk, and they don't get
removed,
> >the in memory key size of the disk cache can grow indefinitely.
> >
> >This would most likely be the result of improper cache usage, since
you
> >shouldn't cache something that won't likely be used again.  However,
> >there is a high usage scenario that could result in massive disk key
> >memory usage.  Let's assume that you cache objects that might not get
> >used very frequently, and you set them to expire after a short
duration,
> >let's say 30 seconds.  If that item gets spooled to disk within 30
> >seconds, because either you have a small max memory size or extremely
> >high usage, and there is never a get for that item, then the key will
> >sit in disk memory forever or until a get.  If there is a get it will
be
> >found on disk, checked for expiration, and then removed, but if there
is
> >no get, it will just hang around.
> >
> >For now, the way to prevent this is to increase the memory size and
> >shrink the max life time.  Alternatively, you can disable the disk
cache
> >if most of your items are of this type.
> >
> >In the future some sort of disk cleanup should occur, or the key
sotrage
> >for the disk cache could use a LRU algorithm that would limit it
size.
> >I think this is actually the best option.  I will do something like
this
> >soon.
> >
> >I can't find any actual memory leaks, only the spike from the
shrinker
> >and the potential key bloat in the disk cache.
> >
> >I'm still hunting for more.
> >
> >Aaron
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Estefano Eduardo [mailto:eduardo.estefano@siemens.com]
> >> Sent: Friday, May 07, 2004 2:43 AM
> >> To: Turbine JCS Users List
> >> Subject: RE: Out of memory problem
> >>
> >> We also started getting out of memory errors. Same situation, only
> >after
> >> we started using the jcs library (I have latest copy from main
trunk).
> >>
> >> We did not pin point the error yet though. It did not happen
straight
> >> away, only after the server was running for a few days. The
solution
> >is
> >> to restart the server at the moment. We only use memory cache and
we
> >do
> >> use schrinkers on some regions.
> >>
> >> -----Original Message-----
> >> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> >> Sent: Thursday, 06 May, 2004 19:52
> >> To: 'Turbine JCS Users List'
> >> Subject: RE: Out of memory problem
> >>
> >>
> >> Matthew informs me that the problem seems to go away with the
shrinker
> >> off.
> >>
> >> This suggests a problem with the memory shrinker.  It could be that
> >the
> >> thread is maintaining a reference to the keys, the spooling from
the
> >> shrinker is problematic, or there is a problem generating the list
of
> >> keys.
> >>
> >> I don't think any of these things is happening, so I'm sort of at a
> >loss
> >> for an explanation.
> >>
> >> I think the shrinker just sleeps for an interval.  Perhaps it would
be
> >> better to use a timer and start the shrinker and then let it die.
One
> >> concern with this is that we wouldn't want more than one shrinker
> >> running per region at any given time.  This could happen say if the
> >> number of elements was huge and the interval between shrinks was
very
> >> small.  I can solve that by passing in a lock that will prevent a
> >second
> >> shrinker from running.
> >>
> >> Hmmn.  I'll look into it again.
> >>
> >> Aaron
> >>
> >> > -----Original Message-----
> >> > From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> >> > Sent: Thursday, May 06, 2004 9:43 AM
> >> > To: 'Turbine JCS Users List'
> >> > Subject: RE: Out of memory problem
> >> >
> >> > Send your cache.ccf and let me know how much memory you've
allocated
> >> for
> >> > the jvm.
> >> >
> >> > Aaron
> >> >
> >> > > -----Original Message-----
> >> > > From: Parvathi Rajaraman
[mailto:Parvathi.Rajaraman@metcash.com]
> >> > > Sent: Thursday, May 06, 2004 1:53 AM
> >> > > To: 'turbine-jcs-user@jakarta.apache.org'
> >> > > Subject: Out of memory problem
> >> > >
> >> > > Hi,
> >> > >
> >> > > I have recently started using JCS implementation for Caching in
my
> >> > > project. I have noticed that every few minutes, I keep getting
a
> >> > > out-of
> >> memory
> >> > > error
> >> > > and this has happened only after I have started using JCS. So
is
> >> this
> >> > > something to do with the way I have configured it. Can you help
me
> >> > out.
> >> > >
> >> > > Regds
> >> > > Parvathi
> >> > >
> >> > >
> >>
---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail:
> >> > turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> > > For additional commands, e-mail:
> >> > turbine-jcs-user-help@jakarta.apache.org
> >> >
> >> >
> >> >
> >---------------------------------------------------------------------
> >> > To unsubscribe, e-mail:
> >> turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> > For additional commands, e-mail:
> >> turbine-jcs-user-help@jakarta.apache.org
> >>
> >>
> >>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >> turbine-jcs-user-help@jakarta.apache.org
> >>
> >>
> >>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org