Reducing memory usage

[Charlie Hubbard <ch...@gmail.com>]

Hi,

I have a database that I have been storing binary data in a Blob
column.  I've decided that it was a bad idea and I'd like to remove it
from the DB and store it on the filesystem.  I've written a small
program to extract the contents from the DB and save it to the file
system.  It works except it runs out of memory using the default 64
MB.  While I could increase the -Xmx setting it seems superflous.
I've profiled the application and majority of the memory is being
occupied by derby.  I see lots of classes like
org.apache.derby.impl.services.cache.CachedItem and
org.apache.derby.impl.store.raw.data.StoredPage holding 32KB byte
buffers.

Is there a way to change derby so that it would drop those cached
buffers?  Or a way to turn off caching for blobs?  Or is there a way
to optimize the memory a little more so that it would run in a smaller
footprint?

This program will have to run on DB that could be in the GB size.  I
just need to extract the contents and save it out to disk.  Modify the
DB to drop the Blobs, and re-attach those records to the filesystem
paths of the files.

Thanks
Charlie

Re: Reducing memory usage

[Kristian Waagan <Kr...@Sun.COM>]

Dyre.Tjeldvoll@Sun.COM wrote:
> Charlie Hubbard <ch...@gmail.com> writes:
> 
>> Hi,
>>

[ snip ]

> 
> I think there is a lower limt for the page cache size, but I don't
> remember what it is. The tuning guide has the details.

FYI, the lower limit for the page cache size is 40 pages.


-- 
Kristian
> 

Re: Reducing memory usage

[Dy...@Sun.COM]

Charlie Hubbard <ch...@gmail.com> writes:

> Hi,
>
> I have a database that I have been storing binary data in a Blob
> column.  I've decided that it was a bad idea and I'd like to remove it
> from the DB and store it on the filesystem.  I've written a small
> program to extract the contents from the DB and save it to the file
> system.  It works except it runs out of memory using the default 64
> MB.  While I could increase the -Xmx setting it seems superflous.
> I've profiled the application and majority of the memory is being
> occupied by derby.  I see lots of classes like
> org.apache.derby.impl.services.cache.CachedItem and
> org.apache.derby.impl.store.raw.data.StoredPage holding 32KB byte
> buffers.
> Is there a way to change derby so that it would drop those cached
> buffers?  

The size of the page cache is controlled by the
derby.storage.PageCacheSize property. Its default value is 1000, so
you could try to reduce it. Normally you need to adjust your -Xmx
setting so that you have enough heap for the page cache + some
overhead for the rest of Derby...

Reducing it could affect performance but that depends on the nature of
your application (how much it benefits from caching...)

>Or a way to turn off caching for blobs? 

Don't think so

> Or is there a way
> to optimize the memory a little more so that it would run in a smaller
> footprint?

Well, the page cache is the part of Derby that normally uses the most
memory, so it's probably a good place to start.

> This program will have to run on DB that could be in the GB size.  I
> just need to extract the contents and save it out to disk.  

If you only read each record once before writing it to disk, you
probably won't benfit much from caching anyway, so reducing it is
probably a good idea...

I think there is a lower limt for the page cache size, but I don't
remember what it is. The tuning guide has the details.

-- 
dt