fieldcache gives OOM. Deos a LRU-style fieldcache exist?

[Britske <gb...@gmail.com>]

First my question: 

Is there an (experimental / patch-version) lucene-fieldcache available which
uses some kind of eviction-strategy (LRU or whatever) so that OOM's would
never happen in my case, but instead some sorts would simply get evicted?

Now the background for those who are interested: 

I'm having a usecase in which products can have a lot of variants. (Up to
10.000) 
These variants can each have their own price (Which I need to sort and
filter by ) and a bunch of stored values (which I just need to display to
the client). At all times only 1 variant per product needs to be shown in a
search-result. I've modeled this by letting each possible variant have 2
fields: 1 pricefield and 1 storagefield. This means that every entry in the
index is a product with possible more than 20.000 fields. Potentially:
because the actual collection of available variants is sparse per product.
This all sounds a bit exotic I know, but uptil I was pointed to this problem
everything went fine. (index-time is a 'bit' slow but that's no real
problem.)

The problem is that potentially all pricefields of all variants can be used
for sorting, meaning that the fieldcache would have to contain up to 10.000
cached lists. This will give OutOfMemory-errors sooner or later, (products
will grow to about 400.000, so the fieldcache will grow to about 16GB) which
is not what my evironment will be able to handle any time soon.

Thanks in advance, 

Geert-Jan
-- 
View this message in context: http://www.nabble.com/fieldcache-gives-OOM.-Deos-a-LRU-style-fieldcache-exist--tf4791826.html#a13707745
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: fieldcache gives OOM. Deos a LRU-style fieldcache exist?

[Mark Miller <ma...@gmail.com>]

https://issues.apache.org/jira/browse/LUCENE-831

This patch should be a good start for you. Not sure how stale it is at 
this point though.

Britske wrote:
> First my question: 
>
> Is there an (experimental / patch-version) lucene-fieldcache available which
> uses some kind of eviction-strategy (LRU or whatever) so that OOM's would
> never happen in my case, but instead some sorts would simply get evicted?
>
> Now the background for those who are interested: 
>
> I'm having a usecase in which products can have a lot of variants. (Up to
> 10.000) 
> These variants can each have their own price (Which I need to sort and
> filter by ) and a bunch of stored values (which I just need to display to
> the client). At all times only 1 variant per product needs to be shown in a
> search-result. I've modeled this by letting each possible variant have 2
> fields: 1 pricefield and 1 storagefield. This means that every entry in the
> index is a product with possible more than 20.000 fields. Potentially:
> because the actual collection of available variants is sparse per product.
> This all sounds a bit exotic I know, but uptil I was pointed to this problem
> everything went fine. (index-time is a 'bit' slow but that's no real
> problem.)
>
> The problem is that potentially all pricefields of all variants can be used
> for sorting, meaning that the fieldcache would have to contain up to 10.000
> cached lists. This will give OutOfMemory-errors sooner or later, (products
> will grow to about 400.000, so the fieldcache will grow to about 16GB) which
> is not what my evironment will be able to handle any time soon.
>
> Thanks in advance, 
>
> Geert-Jan
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org