[vfs] Cache size issue

[Béla Hullár <hu...@gmail.com>]

Hi All,

We've been developing a system where we move around a lot of files. We
started to use vfs, because sometimes these files are on a remote
filesystem and vfs is really a great library which enables us to handle
these cases uniform.

As the system was started to be used very heavily recently we experienced
memory issues, OutOfMemoryErrors. After investigating the problem it seems
the cause of the problem was the FilesCaches of the manager which was
growing without limits. We have few questions regarding this:

1, is this infinite growing of a SoftRefFilesCache a bug or a feature?
javadoc says if the jvm needs memory then the cache will free. But this one
by one removal in the maintenance thread of the cache doesn't seem to be
very efficient.

2, we started to use LRUFilesCache, because with that we can limit the
number of the cached entries. Indeed it seems it has much lower memory
usage (according the set limit), but checking a heap dump showed that our
LRUCache contained around ~21k entries however its size was set to 500. Do
you have some idea how it's possible?

I created a small test program which can reproduce the issue:
https://svn.code.sf.net/p/screeningbee/code/trunk/vfs-cache-test/src/main/java/ch/systemsx/bee/vfscachetest/CacheSizeChecker.java

if you set the testroot for a directory contains a lot of files and limit
the heapsize (eg. -Xmx128m) then it will fail with the default SoftRefCache
and with WeekRefCache and it works with LRU an NullFilesCache
Of course in the real application we have lot more memory, but it was also
eaten up after a while.

we use java 7 and vfs 2.0

Any help would be appreciated!
Bela

Re: [vfs] Cache size issue

[Béla Hullár <hu...@gmail.com>]

Hi Again,

After setting the caching strategy to CacheStrategy.ON_CALL it seems our
application can work with NullFilesCache too.
But very likely this will have effect on the performance.
Do you have any idea? For us this growing cache seems clearly to a bug.
(specially with the LRUFilesCache, where we can define size, but the cache
still grows constantly)

Thanks,
Bela




2013/11/14 Béla Hullár <hu...@gmail.com>

> After checking the heap dump it seems that the refReverseMap map in the
> SoftRefFileCache doesn't get cleaned. The LocalFileName objects hold by
> this map were occupying more then 80% of the memory.
>
>
>
> 2013/11/13 Béla Hullár <hu...@gmail.com>
>
>> Hi All,
>>
>> We've been developing a system where we move around a lot of files. We
>> started to use vfs, because sometimes these files are on a remote
>> filesystem and vfs is really a great library which enables us to handle
>> these cases uniform.
>>
>> As the system was started to be used very heavily recently we experienced
>> memory issues, OutOfMemoryErrors. After investigating the problem it seems
>> the cause of the problem was the FilesCaches of the manager which was
>> growing without limits. We have few questions regarding this:
>>
>> 1, is this infinite growing of a SoftRefFilesCache a bug or a feature?
>> javadoc says if the jvm needs memory then the cache will free. But this one
>> by one removal in the maintenance thread of the cache doesn't seem to be
>> very efficient.
>>
>> 2, we started to use LRUFilesCache, because with that we can limit the
>> number of the cached entries. Indeed it seems it has much lower memory
>> usage (according the set limit), but checking a heap dump showed that our
>> LRUCache contained around ~21k entries however its size was set to 500. Do
>> you have some idea how it's possible?
>>
>> I created a small test program which can reproduce the issue:
>>
>> https://svn.code.sf.net/p/screeningbee/code/trunk/vfs-cache-test/src/main/java/ch/systemsx/bee/vfscachetest/CacheSizeChecker.java
>>
>> if you set the testroot for a directory contains a lot of files and limit
>> the heapsize (eg. -Xmx128m) then it will fail with the default
>> SoftRefCache and with WeekRefCache and it works with LRU an NullFilesCache
>> Of course in the real application we have lot more memory, but it was
>> also eaten up after a while.
>>
>> we use java 7 and vfs 2.0
>>
>> Any help would be appreciated!
>> Bela
>>
>>
>>
>
>
> --
> Üdvözlettel:
> Hullár Béla
>



-- 
Üdvözlettel:
Hullár Béla

Re: [vfs] Cache size issue

[Béla Hullár <hu...@gmail.com>]

After checking the heap dump it seems that the refReverseMap map in the
SoftRefFileCache doesn't get cleaned. The LocalFileName objects hold by
this map were occupying more then 80% of the memory.



2013/11/13 Béla Hullár <hu...@gmail.com>

> Hi All,
>
> We've been developing a system where we move around a lot of files. We
> started to use vfs, because sometimes these files are on a remote
> filesystem and vfs is really a great library which enables us to handle
> these cases uniform.
>
> As the system was started to be used very heavily recently we experienced
> memory issues, OutOfMemoryErrors. After investigating the problem it seems
> the cause of the problem was the FilesCaches of the manager which was
> growing without limits. We have few questions regarding this:
>
> 1, is this infinite growing of a SoftRefFilesCache a bug or a feature?
> javadoc says if the jvm needs memory then the cache will free. But this one
> by one removal in the maintenance thread of the cache doesn't seem to be
> very efficient.
>
> 2, we started to use LRUFilesCache, because with that we can limit the
> number of the cached entries. Indeed it seems it has much lower memory
> usage (according the set limit), but checking a heap dump showed that our
> LRUCache contained around ~21k entries however its size was set to 500. Do
> you have some idea how it's possible?
>
> I created a small test program which can reproduce the issue:
>
> https://svn.code.sf.net/p/screeningbee/code/trunk/vfs-cache-test/src/main/java/ch/systemsx/bee/vfscachetest/CacheSizeChecker.java
>
> if you set the testroot for a directory contains a lot of files and limit
> the heapsize (eg. -Xmx128m) then it will fail with the default
> SoftRefCache and with WeekRefCache and it works with LRU an NullFilesCache
> Of course in the real application we have lot more memory, but it was also
> eaten up after a while.
>
> we use java 7 and vfs 2.0
>
> Any help would be appreciated!
> Bela
>
>
>


-- 
Üdvözlettel:
Hullár Béla