Calculating memory consumption FVH

[Markus Jelsma <ma...@openindex.io>]

Hi,

How can i calculate the memory requirements for highlighting documents where 
the variables are number of documents to highlight and the total size of the 
documents in a single request? I can then multiply a worst-case result set 
with the expected concurrency.

For example, with a _limited_ heap highlighting works perfectly well and is 
only CPU-bound, which is clearly visible when highlighting a result set with 
many very large documents. When such a worst-case query is fired a lot of times 
concurrently Solr is guaranteed to run out of memory.

So, now i'm looking for a method to calcualte per-request memory consumption 
for set-ups with FVH and all term* settings enabled on the highlighted field in 
schema.

Thanks,

Re: Calculating memory consumption FVH

[Koji Sekiguchi <ko...@r.email.ne.jp>]

(11/06/12 8:27), Markus Jelsma wrote:
> Hi,
>
> How can i calculate the memory requirements for highlighting documents where
> the variables are number of documents to highlight and the total size of the
> documents in a single request? I can then multiply a worst-case result set
> with the expected concurrency.
>
> For example, with a _limited_ heap highlighting works perfectly well and is
> only CPU-bound, which is clearly visible when highlighting a result set with
> many very large documents. When such a worst-case query is fired a lot of times
> concurrently Solr is guaranteed to run out of memory.
>
> So, now i'm looking for a method to calcualte per-request memory consumption
> for set-ups with FVH and all term* settings enabled on the highlighted field in
> schema.

Though FVH uses some structured objects described at:

http://lucene.apache.org/java/3_2_0/api/all/org/apache/lucene/search/vectorhighlight/package-summary.html#package_description

during its processing, but these should be relatively small.

So, I think huge stored field data to be highlighted consumes a lot of heap.
To limit it, think of use of copyField with maxChars for the highlight field.

koji
-- 
http://www.rondhuit.com/en/