You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Todd Benge <to...@gmail.com> on 2008/10/29 23:03:45 UTC

OutOfMemory Problems Lucene 2.4 / Tomcat

Hi,

I'm the lead engineer for search on a large website using lucene for search.

We're indexing about 300M documents in ~ 100 indices.  The indices add
 up to ~ 60G.

The indices are sorted into 4 different Multisearcher with the largest
handling ~50G.

The code is basically like the following:

private static MultiSearcher searcher;

public void init(File files) {

     IndexSearcer [] searchers = new IndexSearcher[files.length] ();
     int i = 0;
     for ( File file: files ) {
          searchers[i++] = new IndexSearcher(FSDirectory.getDirectory(file);
     }

searcher = new MultiSearcher(searchers);
}

public Searcher getSearcher() {
   return searcher;
}

We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
Performance is good but servers are consistently hanging with
OutOfMemory errors.

We're allocating 4G in the heap to each server.

Is there any way to control the amount of memory Lucene consume for
caching?  Any other suggestions on fixing the memory errors?

Thanks,

Todd

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Mark Miller <ma...@gmail.com>.
How many fields are you sorting on? Lots of unuiqe terms in those  
fields?

- Mark


On Oct 29, 2008, at 6:03 PM, "Todd Benge" <to...@gmail.com> wrote:

> Hi,
>
> I'm the lead engineer for search on a large website using lucene for  
> search.
>
> We're indexing about 300M documents in ~ 100 indices.  The indices add
> up to ~ 60G.
>
> The indices are sorted into 4 different Multisearcher with the largest
> handling ~50G.
>
> The code is basically like the following:
>
> private static MultiSearcher searcher;
>
> public void init(File files) {
>
>     IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>     int i = 0;
>     for ( File file: files ) {
>          searchers[i++] = new  
> IndexSearcher(FSDirectory.getDirectory(file);
>     }
>
> searcher = new MultiSearcher(searchers);
> }
>
> public Searcher getSearcher() {
>   return searcher;
> }
>
> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
> Performance is good but servers are consistently hanging with
> OutOfMemory errors.
>
> We're allocating 4G in the heap to each server.
>
> Is there any way to control the amount of memory Lucene consume for
> caching?  Any other suggestions on fixing the memory errors?
>
> Thanks,
>
> Todd
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Michael McCandless <lu...@mikemccandless.com>.
The terms index (*.tii), which is loaded entirely into RAM, can  
consume an unexpectedly large amount of memory when there are an  
unusually high number of terms.   If you are not using compound file  
format, can you look at the size of *.tii?

If this is what is affecting you, one simple workaround is to call  
IndexReader.setTermInfosIndexDivisor, which subsamples the terms in  
*.tii when loading them into RAM.  EG passing in 2 halves the RAM  
used, but, seeks to a given term will take longer.

I think we can actually improve the memory usage of tii as part of the  
flexible indexing work; I think with a small change to the file format  
we don't need to store a separate TermInfo instance for each indexed  
term, which'd be a big RAM savings for indices with many unique terms.

Another thing to watch out for is norms: if you have 300M docs, then  
each fields which has norms turned on (they are turned on by default)  
will consume 300M of RAM.  If you decide to turn norms off, you have  
to fully re-index since norms "spread" during segment merging.

Mike

Todd Benge wrote:

> Thanks Mark.  I appreciate the help.
>
> I thought our memory may be low but wanted to verify there if there is
> any way to control memory usage.  I think we'll likely upgrade the
> memory on the machines but that may just delay the inevitable.
>
> Wondering if anyone else has encountered similar issues with indices
> if a similar size.  I've been thinking we will need to move to a
> clustered solution and have been reading on hadoop, nutch, solr &
> terracotta for possibilities such as index sharding.
>
> Has anyone implemented a solution using hadoop or terracotta for a
> large scale system?  Just wondering the pro's / con's of the various
> approaches.
>
> Thanks,
>
> Todd
>
> On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <ma...@gmail.com>  
> wrote:
>> The term, terminfo, indexreader internals stuff is prob on the low  
>> end
>> compared to the size of your field caches (needed for sorting). If  
>> you are
>> sorting by String I think the space needed is 32 bits x number of  
>> docs + an
>> array to hold all of the unique terms. So checking 300 million docs  
>> (I know
>> you are actually breaking it up smaller than that, but for example)  
>> and
>> ignoring things like String chars being variable byte lengths and  
>> storing
>> the length, etc and randomly picking 50000 unique terms at 6 chars  
>> per:
>>
>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138  
>> megabytes
>>
>> Thats per field your sorting on. If you are sorting on an int field  
>> it
>> should be closer to 32 bits x num docs - shorts, 32 bits x num  
>> docs, etc.
>>
>> So you have those field caches, plus the IndexReader terminfo, term  
>> stuff,
>> plus whatever RAM your app needs beyond Lucene. 4 gig might just  
>> not *quite*
>> cut it is my guess.
>>
>> Todd Benge wrote:
>>>
>>> There's usually only a couple sort fields and a bunch of terms in  
>>> the
>>> various indices.  The terms are user entered on various media so the
>>> number of terms is very large.
>>>
>>> Thanks for the help.
>>>
>>> Todd
>>>
>>>
>>>
>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I'm the lead engineer for search on a large website using lucene  
>>>> for
>>>> search.
>>>>
>>>> We're indexing about 300M documents in ~ 100 indices.  The  
>>>> indices add
>>>> up to ~ 60G.
>>>>
>>>> The indices are sorted into 4 different Multisearcher with the  
>>>> largest
>>>> handling ~50G.
>>>>
>>>> The code is basically like the following:
>>>>
>>>> private static MultiSearcher searcher;
>>>>
>>>> public void init(File files) {
>>>>
>>>>    IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>    int i = 0;
>>>>    for ( File file: files ) {
>>>>         searchers[i++] = new
>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>    }
>>>>
>>>> searcher = new MultiSearcher(searchers);
>>>> }
>>>>
>>>> public Searcher getSearcher() {
>>>>  return searcher;
>>>> }
>>>>
>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>> Performance is good but servers are consistently hanging with
>>>> OutOfMemory errors.
>>>>
>>>> We're allocating 4G in the heap to each server.
>>>>
>>>> Is there any way to control the amount of memory Lucene consume for
>>>> caching?  Any other suggestions on fixing the memory errors?
>>>>
>>>> Thanks,
>>>>
>>>> Todd
>>>>
>>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Todd Benge <to...@gmail.com>.
I tried the term divisor index prior to the posting and didn't see
much difference in the memory usage.

I don't think we can turn off field norms because we use boosting to
influence some content to the front of the results.

Will definitely spend some time with Solr, Terracotta, and possibly
hadoop to see if any of that helps.

Also planning on running the example code Mark posted to see if we can
see any optimizations.

We really appreciate the help.

Todd

On Thu, Oct 30, 2008 at 5:21 AM, mark harwood <ma...@yahoo.co.uk> wrote:
> One issue with the existing field cache implementation is that it uses int arrays to reference into the list of unique terms where short or even byte arrays may suffice for fields with smaller numbers of unique terms.
> How many unique terms do you have?

I don't know the actual number for unique terms but they're vast since
they're using entered data on 300M media objects.

>
> I posted some code that measures the potential RAM savings to be had, given your particular index.
>    See here: http://www.nabble.com/Re%3A-Question-regarding-sorting-and-memory-consumption-in-lucene-p19997753.html
>
> This code shows what theoretically could be saved given a different approach to field caching. Unfortunately the use of int arrays is not abstracted in Lucene's StringIndex so this optimisation cannot be dropped in without some disruption.
> Shame, as there is a big difference between byte[reader.maxDoc] and int[reader.maxDoc] when you are working with large indexes and are tight on RAM.
>
> If you're prepared to write a more optimized custom field cache then the above code may be a useful start point.
>
> Cheers,
> Mark
>
>
>
>
>
> ----- Original Message ----
> From: Mark Miller <ma...@gmail.com>
> To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
> Sent: Thursday, 30 October, 2008 10:37:48
> Subject: Re: OutOfMemory Problems Lucene 2.4 / Tomcat
>
> Michaels got some great points (he the lucene master), especially possibly turning off norms if you can, but for an index like that i'd reccomwnd solr. Solr sharding can be scaled to billions (min a billion or two anyway) with few limitations (of course there are a few). Plus it has further caching options, indexreader refresh managment, etc etc etc
>
>
> - Mark
>
>
> On Oct 29, 2008, at 10:30 PM, "Todd Benge" <to...@gmail.com> wrote:
>
>> Thanks Mark.  I appreciate the help.
>>
>> I thought our memory may be low but wanted to verify there if there is
>> any way to control memory usage.  I think we'll likely upgrade the
>> memory on the machines but that may just delay the inevitable.
>>
>> Wondering if anyone else has encountered similar issues with indices
>> if a similar size.  I've been thinking we will need to move to a
>> clustered solution and have been reading on hadoop, nutch, solr &
>> terracotta for possibilities such as index sharding.
>>
>> Has anyone implemented a solution using hadoop or terracotta for a
>> large scale system?  Just wondering the pro's / con's of the various
>> approaches.
>>
>> Thanks,
>>
>> Todd
>>
>> On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <ma...@gmail.com> wrote:
>>> The term, terminfo, indexreader internals stuff is prob on the low end
>>> compared to the size of your field caches (needed for sorting). If you are
>>> sorting by String I think the space needed is 32 bits x number of docs + an
>>> array to hold all of the unique terms. So checking 300 million docs (I know
>>> you are actually breaking it up smaller than that, but for example) and
>>> ignoring things like String chars being variable byte lengths and storing
>>> the length, etc and randomly picking 50000 unique terms at 6 chars per:
>>>
>>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
>>>
>>> Thats per field your sorting on. If you are sorting on an int field it
>>> should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.
>>>
>>> So you have those field caches, plus the IndexReader terminfo, term stuff,
>>> plus whatever RAM your app needs beyond Lucene. 4 gig might just not *quite*
>>> cut it is my guess.
>>>
>>> Todd Benge wrote:
>>>>
>>>> There's usually only a couple sort fields and a bunch of terms in the
>>>> various indices.  The terms are user entered on various media so the
>>>> number of terms is very large.
>>>>
>>>> Thanks for the help.
>>>>
>>>> Todd
>>>>
>>>>
>>>>
>>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm the lead engineer for search on a large website using lucene for
>>>>> search.
>>>>>
>>>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>>> up to ~ 60G.
>>>>>
>>>>> The indices are sorted into 4 different Multisearcher with the largest
>>>>> handling ~50G.
>>>>>
>>>>> The code is basically like the following:
>>>>>
>>>>> private static MultiSearcher searcher;
>>>>>
>>>>> public void init(File files) {
>>>>>
>>>>>    IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>>    int i = 0;
>>>>>    for ( File file: files ) {
>>>>>         searchers[i++] = new
>>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>>    }
>>>>>
>>>>> searcher = new MultiSearcher(searchers);
>>>>> }
>>>>>
>>>>> public Searcher getSearcher() {
>>>>>  return searcher;
>>>>> }
>>>>>
>>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>>> Performance is good but servers are consistently hanging with
>>>>> OutOfMemory errors.
>>>>>
>>>>> We're allocating 4G in the heap to each server.
>>>>>
>>>>> Is there any way to control the amount of memory Lucene consume for
>>>>> caching?  Any other suggestions on fixing the memory errors?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Todd
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by mark harwood <ma...@yahoo.co.uk>.
One issue with the existing field cache implementation is that it uses int arrays to reference into the list of unique terms where short or even byte arrays may suffice for fields with smaller numbers of unique terms.
How many unique terms do you have?

I posted some code that measures the potential RAM savings to be had, given your particular index.
    See here: http://www.nabble.com/Re%3A-Question-regarding-sorting-and-memory-consumption-in-lucene-p19997753.html

This code shows what theoretically could be saved given a different approach to field caching. Unfortunately the use of int arrays is not abstracted in Lucene's StringIndex so this optimisation cannot be dropped in without some disruption.
Shame, as there is a big difference between byte[reader.maxDoc] and int[reader.maxDoc] when you are working with large indexes and are tight on RAM.

If you're prepared to write a more optimized custom field cache then the above code may be a useful start point.

Cheers,
Mark





----- Original Message ----
From: Mark Miller <ma...@gmail.com>
To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
Sent: Thursday, 30 October, 2008 10:37:48
Subject: Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Michaels got some great points (he the lucene master), especially possibly turning off norms if you can, but for an index like that i'd reccomwnd solr. Solr sharding can be scaled to billions (min a billion or two anyway) with few limitations (of course there are a few). Plus it has further caching options, indexreader refresh managment, etc etc etc


- Mark


On Oct 29, 2008, at 10:30 PM, "Todd Benge" <to...@gmail.com> wrote:

> Thanks Mark.  I appreciate the help.
> 
> I thought our memory may be low but wanted to verify there if there is
> any way to control memory usage.  I think we'll likely upgrade the
> memory on the machines but that may just delay the inevitable.
> 
> Wondering if anyone else has encountered similar issues with indices
> if a similar size.  I've been thinking we will need to move to a
> clustered solution and have been reading on hadoop, nutch, solr &
> terracotta for possibilities such as index sharding.
> 
> Has anyone implemented a solution using hadoop or terracotta for a
> large scale system?  Just wondering the pro's / con's of the various
> approaches.
> 
> Thanks,
> 
> Todd
> 
> On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <ma...@gmail.com> wrote:
>> The term, terminfo, indexreader internals stuff is prob on the low end
>> compared to the size of your field caches (needed for sorting). If you are
>> sorting by String I think the space needed is 32 bits x number of docs + an
>> array to hold all of the unique terms. So checking 300 million docs (I know
>> you are actually breaking it up smaller than that, but for example) and
>> ignoring things like String chars being variable byte lengths and storing
>> the length, etc and randomly picking 50000 unique terms at 6 chars per:
>> 
>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
>> 
>> Thats per field your sorting on. If you are sorting on an int field it
>> should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.
>> 
>> So you have those field caches, plus the IndexReader terminfo, term stuff,
>> plus whatever RAM your app needs beyond Lucene. 4 gig might just not *quite*
>> cut it is my guess.
>> 
>> Todd Benge wrote:
>>> 
>>> There's usually only a couple sort fields and a bunch of terms in the
>>> various indices.  The terms are user entered on various media so the
>>> number of terms is very large.
>>> 
>>> Thanks for the help.
>>> 
>>> Todd
>>> 
>>> 
>>> 
>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> I'm the lead engineer for search on a large website using lucene for
>>>> search.
>>>> 
>>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>> up to ~ 60G.
>>>> 
>>>> The indices are sorted into 4 different Multisearcher with the largest
>>>> handling ~50G.
>>>> 
>>>> The code is basically like the following:
>>>> 
>>>> private static MultiSearcher searcher;
>>>> 
>>>> public void init(File files) {
>>>> 
>>>>    IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>    int i = 0;
>>>>    for ( File file: files ) {
>>>>         searchers[i++] = new
>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>    }
>>>> 
>>>> searcher = new MultiSearcher(searchers);
>>>> }
>>>> 
>>>> public Searcher getSearcher() {
>>>>  return searcher;
>>>> }
>>>> 
>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>> Performance is good but servers are consistently hanging with
>>>> OutOfMemory errors.
>>>> 
>>>> We're allocating 4G in the heap to each server.
>>>> 
>>>> Is there any way to control the amount of memory Lucene consume for
>>>> caching?  Any other suggestions on fixing the memory errors?
>>>> 
>>>> Thanks,
>>>> 
>>>> Todd
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Mark Miller <ma...@gmail.com>.
Michaels got some great points (he the lucene master), especially  
possibly turning off norms if you can, but for an index like that i'd  
reccomwnd solr. Solr sharding can be scaled to billions (min a billion  
or two anyway) with few limitations (of course there are a few). Plus  
it has further caching options, indexreader refresh managment, etc etc  
etc


- Mark


On Oct 29, 2008, at 10:30 PM, "Todd Benge" <to...@gmail.com> wrote:

> Thanks Mark.  I appreciate the help.
>
> I thought our memory may be low but wanted to verify there if there is
> any way to control memory usage.  I think we'll likely upgrade the
> memory on the machines but that may just delay the inevitable.
>
> Wondering if anyone else has encountered similar issues with indices
> if a similar size.  I've been thinking we will need to move to a
> clustered solution and have been reading on hadoop, nutch, solr &
> terracotta for possibilities such as index sharding.
>
> Has anyone implemented a solution using hadoop or terracotta for a
> large scale system?  Just wondering the pro's / con's of the various
> approaches.
>
> Thanks,
>
> Todd
>
> On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <ma...@gmail.com>  
> wrote:
>> The term, terminfo, indexreader internals stuff is prob on the low  
>> end
>> compared to the size of your field caches (needed for sorting). If  
>> you are
>> sorting by String I think the space needed is 32 bits x number of  
>> docs + an
>> array to hold all of the unique terms. So checking 300 million docs  
>> (I know
>> you are actually breaking it up smaller than that, but for example)  
>> and
>> ignoring things like String chars being variable byte lengths and  
>> storing
>> the length, etc and randomly picking 50000 unique terms at 6 chars  
>> per:
>>
>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138  
>> megabytes
>>
>> Thats per field your sorting on. If you are sorting on an int field  
>> it
>> should be closer to 32 bits x num docs - shorts, 32 bits x num  
>> docs, etc.
>>
>> So you have those field caches, plus the IndexReader terminfo, term  
>> stuff,
>> plus whatever RAM your app needs beyond Lucene. 4 gig might just  
>> not *quite*
>> cut it is my guess.
>>
>> Todd Benge wrote:
>>>
>>> There's usually only a couple sort fields and a bunch of terms in  
>>> the
>>> various indices.  The terms are user entered on various media so the
>>> number of terms is very large.
>>>
>>> Thanks for the help.
>>>
>>> Todd
>>>
>>>
>>>
>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I'm the lead engineer for search on a large website using lucene  
>>>> for
>>>> search.
>>>>
>>>> We're indexing about 300M documents in ~ 100 indices.  The  
>>>> indices add
>>>> up to ~ 60G.
>>>>
>>>> The indices are sorted into 4 different Multisearcher with the  
>>>> largest
>>>> handling ~50G.
>>>>
>>>> The code is basically like the following:
>>>>
>>>> private static MultiSearcher searcher;
>>>>
>>>> public void init(File files) {
>>>>
>>>>    IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>    int i = 0;
>>>>    for ( File file: files ) {
>>>>         searchers[i++] = new
>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>    }
>>>>
>>>> searcher = new MultiSearcher(searchers);
>>>> }
>>>>
>>>> public Searcher getSearcher() {
>>>>  return searcher;
>>>> }
>>>>
>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>> Performance is good but servers are consistently hanging with
>>>> OutOfMemory errors.
>>>>
>>>> We're allocating 4G in the heap to each server.
>>>>
>>>> Is there any way to control the amount of memory Lucene consume for
>>>> caching?  Any other suggestions on fixing the memory errors?
>>>>
>>>> Thanks,
>>>>
>>>> Todd
>>>>
>>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Todd Benge <to...@gmail.com>.
Thanks Mark.  I appreciate the help.

I thought our memory may be low but wanted to verify there if there is
any way to control memory usage.  I think we'll likely upgrade the
memory on the machines but that may just delay the inevitable.

Wondering if anyone else has encountered similar issues with indices
if a similar size.  I've been thinking we will need to move to a
clustered solution and have been reading on hadoop, nutch, solr &
terracotta for possibilities such as index sharding.

Has anyone implemented a solution using hadoop or terracotta for a
large scale system?  Just wondering the pro's / con's of the various
approaches.

Thanks,

Todd

On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <ma...@gmail.com> wrote:
> The term, terminfo, indexreader internals stuff is prob on the low end
> compared to the size of your field caches (needed for sorting). If you are
> sorting by String I think the space needed is 32 bits x number of docs + an
> array to hold all of the unique terms. So checking 300 million docs (I know
> you are actually breaking it up smaller than that, but for example) and
> ignoring things like String chars being variable byte lengths and storing
> the length, etc and randomly picking 50000 unique terms at 6 chars per:
>
> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
>
> Thats per field your sorting on. If you are sorting on an int field it
> should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.
>
> So you have those field caches, plus the IndexReader terminfo, term stuff,
> plus whatever RAM your app needs beyond Lucene. 4 gig might just not *quite*
> cut it is my guess.
>
> Todd Benge wrote:
>>
>> There's usually only a couple sort fields and a bunch of terms in the
>> various indices.  The terms are user entered on various media so the
>> number of terms is very large.
>>
>> Thanks for the help.
>>
>> Todd
>>
>>
>>
>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I'm the lead engineer for search on a large website using lucene for
>>> search.
>>>
>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>  up to ~ 60G.
>>>
>>> The indices are sorted into 4 different Multisearcher with the largest
>>> handling ~50G.
>>>
>>> The code is basically like the following:
>>>
>>> private static MultiSearcher searcher;
>>>
>>> public void init(File files) {
>>>
>>>     IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>     int i = 0;
>>>     for ( File file: files ) {
>>>          searchers[i++] = new
>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>     }
>>>
>>> searcher = new MultiSearcher(searchers);
>>> }
>>>
>>> public Searcher getSearcher() {
>>>   return searcher;
>>> }
>>>
>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>> Performance is good but servers are consistently hanging with
>>> OutOfMemory errors.
>>>
>>> We're allocating 4G in the heap to each server.
>>>
>>> Is there any way to control the amount of memory Lucene consume for
>>> caching?  Any other suggestions on fixing the memory errors?
>>>
>>> Thanks,
>>>
>>> Todd
>>>
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Pablo Saavedra <pa...@gmail.com>.
I hope that helps, if you find anything interesting do post it somewhere.
I'm afraid I'm a little bit far away from New Orleans at the moment.

Regards.

2008/11/4 Todd Benge <to...@gmail.com>

> Thanks Pablo.
>
> I'll be flying to New Orleans tomorrow for ApacheCon and would love
> the opportunity to talk with others about architectures others are
> using.
>
> Todd
>
>
>
> On 11/4/08, PabloS <pa...@gmail.com> wrote:
> >
> > Sure Todd,
> >
> > the idea basically consist in the following:
> >
> > - Subclassing FIeldSortedHitQueue and calling support with an empty
> > SortField array: this disables caching because the comparators are
> retrieved
> > during construction
> > - Creating a new SortComparatorSource that creates the sort comparators
> > similarly to FieldCacheImpl but without performing any caching
> > - Creating a new HitCollector that behaves just like TopFieldDocCollector
> > but uses my custom FieldSortedHitQueue. The tricky part was creating a
> > TopFieldDoc instance (which has default visibility constructor. I managed
> to
> > do it creating an empty TopFieldDocCollector, asking the TopFieldDocs and
> > modifiying that instance.
> > - Overriding the Searcher's search(Weight, Filter, int, Sort) so it uses
> my
> > HitCollector (I use the Hits class, which internally calls that method)
> >
> > That way, you completely override field caching and you'll suffer the
> > performance consequences. What I did was making the SortComparatorSource
> a
> > spring bean and adding a caching interceptor to it. You an plug your
> > favorite cache library in, in my case ehcache.
> >
> > I hope you find this useful.
> > Regards.
> >
> >
> > tbenge wrote:
> >>
> >> Pablo,
> >>
> >> Would you mind adding a little more detail about how you're working
> >> around the problem?
> >>
> >> I'm still evaluating our different options so am interested in what you
> >> did.
> >>
> >> Todd
> >>
> >> On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pa...@gmail.com>
> wrote:
> >>>
> >>> Thanks hossman, but I've already 'solved' the problem without the need
> to
> >>> patch lucene. I had to code a bit around Lucene's visibility
> restrictions
> >>> but I've managed to completely skip the field caching mechanism and add
> >>> ehcache to it.
> >>>
> >>> At the moment it seems to be working quite well, although not as fast
> as
> >>> it
> >>> was when lucene performed the caching.
> >>>
> >>> Thanks a lot for the info anyway.
> >>> Regards.
> >>>
> >>> hossman wrote:
> >>>>
> >>>>
> >>>> : I'm having a similar problem with my application, although we are
> >>>> using
> >>>> : lucene 2.3.2. The problem we have is that we are required to sort
> on
> >>>> most of
> >>>> : the fields (20 at least). Is there any way of changing the cache
> being
> >>>> used?
> >>>>
> >>>> there is a patch in Jira that takes a completley different approach
> >>>> towards dealing with sorting by using the *stored* values of the
> >>>> documents
> >>>> that match...
> >>>>
> >>>> https://issues.apache.org/jira/browse/LUCENE-769
> >>>>
> >>>> ...in theory it's "better" for use cases where:
> >>>>
> >>>>   1) you are confident you are going to get back a "small" result set
> in
> >>>>      proportion to the total size of your index.
> >>>>   2) you need to support sorting on "lots" of fields and don't have
> >>>> enough
> >>>>      ram for all of the FieldCaches of those fields
> >>>>
> >>>> So far the only person to test it out (and post comments) is the patch
> >>>> submitter, but if other people report success it might be worth adding
> >>>> as
> >>>> a contrib.
> >>>>
> >>>> (Disclaimer: it looks like the patch was updated after my last
> >>>> review/comments.  i have not read, nor claim any opinion about the
> >>>> currently attached patch)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> -Hoss
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
> >>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> >
> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20326783.html
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> --
> Sent from Gmail for mobile | mobile.google.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Todd Benge <to...@gmail.com>.
Thanks Pablo.

I'll be flying to New Orleans tomorrow for ApacheCon and would love
the opportunity to talk with others about architectures others are
using.

Todd



On 11/4/08, PabloS <pa...@gmail.com> wrote:
>
> Sure Todd,
>
> the idea basically consist in the following:
>
> - Subclassing FIeldSortedHitQueue and calling support with an empty
> SortField array: this disables caching because the comparators are retrieved
> during construction
> - Creating a new SortComparatorSource that creates the sort comparators
> similarly to FieldCacheImpl but without performing any caching
> - Creating a new HitCollector that behaves just like TopFieldDocCollector
> but uses my custom FieldSortedHitQueue. The tricky part was creating a
> TopFieldDoc instance (which has default visibility constructor. I managed to
> do it creating an empty TopFieldDocCollector, asking the TopFieldDocs and
> modifiying that instance.
> - Overriding the Searcher's search(Weight, Filter, int, Sort) so it uses my
> HitCollector (I use the Hits class, which internally calls that method)
>
> That way, you completely override field caching and you'll suffer the
> performance consequences. What I did was making the SortComparatorSource a
> spring bean and adding a caching interceptor to it. You an plug your
> favorite cache library in, in my case ehcache.
>
> I hope you find this useful.
> Regards.
>
>
> tbenge wrote:
>>
>> Pablo,
>>
>> Would you mind adding a little more detail about how you're working
>> around the problem?
>>
>> I'm still evaluating our different options so am interested in what you
>> did.
>>
>> Todd
>>
>> On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pa...@gmail.com> wrote:
>>>
>>> Thanks hossman, but I've already 'solved' the problem without the need to
>>> patch lucene. I had to code a bit around Lucene's visibility restrictions
>>> but I've managed to completely skip the field caching mechanism and add
>>> ehcache to it.
>>>
>>> At the moment it seems to be working quite well, although not as fast as
>>> it
>>> was when lucene performed the caching.
>>>
>>> Thanks a lot for the info anyway.
>>> Regards.
>>>
>>> hossman wrote:
>>>>
>>>>
>>>> : I'm having a similar problem with my application, although we are
>>>> using
>>>> : lucene 2.3.2. The problem we have is that we are required to sort on
>>>> most of
>>>> : the fields (20 at least). Is there any way of changing the cache being
>>>> used?
>>>>
>>>> there is a patch in Jira that takes a completley different approach
>>>> towards dealing with sorting by using the *stored* values of the
>>>> documents
>>>> that match...
>>>>
>>>> https://issues.apache.org/jira/browse/LUCENE-769
>>>>
>>>> ...in theory it's "better" for use cases where:
>>>>
>>>>   1) you are confident you are going to get back a "small" result set in
>>>>      proportion to the total size of your index.
>>>>   2) you need to support sorting on "lots" of fields and don't have
>>>> enough
>>>>      ram for all of the FieldCaches of those fields
>>>>
>>>> So far the only person to test it out (and post comments) is the patch
>>>> submitter, but if other people report success it might be worth adding
>>>> as
>>>> a contrib.
>>>>
>>>> (Disclaimer: it looks like the patch was updated after my last
>>>> review/comments.  i have not read, nor claim any opinion about the
>>>> currently attached patch)
>>>>
>>>>
>>>>
>>>>
>>>> -Hoss
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20326783.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-- 
Sent from Gmail for mobile | mobile.google.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by PabloS <pa...@gmail.com>.
Sure Todd,

the idea basically consist in the following:

- Subclassing FIeldSortedHitQueue and calling support with an empty
SortField array: this disables caching because the comparators are retrieved
during construction
- Creating a new SortComparatorSource that creates the sort comparators
similarly to FieldCacheImpl but without performing any caching
- Creating a new HitCollector that behaves just like TopFieldDocCollector
but uses my custom FieldSortedHitQueue. The tricky part was creating a
TopFieldDoc instance (which has default visibility constructor. I managed to
do it creating an empty TopFieldDocCollector, asking the TopFieldDocs and
modifiying that instance.
- Overriding the Searcher's search(Weight, Filter, int, Sort) so it uses my
HitCollector (I use the Hits class, which internally calls that method)

That way, you completely override field caching and you'll suffer the
performance consequences. What I did was making the SortComparatorSource a
spring bean and adding a caching interceptor to it. You an plug your
favorite cache library in, in my case ehcache.

I hope you find this useful.
Regards.


tbenge wrote:
> 
> Pablo,
> 
> Would you mind adding a little more detail about how you're working
> around the problem?
> 
> I'm still evaluating our different options so am interested in what you
> did.
> 
> Todd
> 
> On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pa...@gmail.com> wrote:
>>
>> Thanks hossman, but I've already 'solved' the problem without the need to
>> patch lucene. I had to code a bit around Lucene's visibility restrictions
>> but I've managed to completely skip the field caching mechanism and add
>> ehcache to it.
>>
>> At the moment it seems to be working quite well, although not as fast as
>> it
>> was when lucene performed the caching.
>>
>> Thanks a lot for the info anyway.
>> Regards.
>>
>> hossman wrote:
>>>
>>>
>>> : I'm having a similar problem with my application, although we are
>>> using
>>> : lucene 2.3.2. The problem we have is that we are required to sort on
>>> most of
>>> : the fields (20 at least). Is there any way of changing the cache being
>>> used?
>>>
>>> there is a patch in Jira that takes a completley different approach
>>> towards dealing with sorting by using the *stored* values of the
>>> documents
>>> that match...
>>>
>>> https://issues.apache.org/jira/browse/LUCENE-769
>>>
>>> ...in theory it's "better" for use cases where:
>>>
>>>   1) you are confident you are going to get back a "small" result set in
>>>      proportion to the total size of your index.
>>>   2) you need to support sorting on "lots" of fields and don't have
>>> enough
>>>      ram for all of the FieldCaches of those fields
>>>
>>> So far the only person to test it out (and post comments) is the patch
>>> submitter, but if other people report success it might be worth adding
>>> as
>>> a contrib.
>>>
>>> (Disclaimer: it looks like the patch was updated after my last
>>> review/comments.  i have not read, nor claim any opinion about the
>>> currently attached patch)
>>>
>>>
>>>
>>>
>>> -Hoss
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20326783.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Todd Benge <to...@gmail.com>.
Pablo,

Would you mind adding a little more detail about how you're working
around the problem?

I'm still evaluating our different options so am interested in what you did.

Todd

On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pa...@gmail.com> wrote:
>
> Thanks hossman, but I've already 'solved' the problem without the need to
> patch lucene. I had to code a bit around Lucene's visibility restrictions
> but I've managed to completely skip the field caching mechanism and add
> ehcache to it.
>
> At the moment it seems to be working quite well, although not as fast as it
> was when lucene performed the caching.
>
> Thanks a lot for the info anyway.
> Regards.
>
> hossman wrote:
>>
>>
>> : I'm having a similar problem with my application, although we are using
>> : lucene 2.3.2. The problem we have is that we are required to sort on
>> most of
>> : the fields (20 at least). Is there any way of changing the cache being
>> used?
>>
>> there is a patch in Jira that takes a completley different approach
>> towards dealing with sorting by using the *stored* values of the documents
>> that match...
>>
>> https://issues.apache.org/jira/browse/LUCENE-769
>>
>> ...in theory it's "better" for use cases where:
>>
>>   1) you are confident you are going to get back a "small" result set in
>>      proportion to the total size of your index.
>>   2) you need to support sorting on "lots" of fields and don't have enough
>>      ram for all of the FieldCaches of those fields
>>
>> So far the only person to test it out (and post comments) is the patch
>> submitter, but if other people report success it might be worth adding as
>> a contrib.
>>
>> (Disclaimer: it looks like the patch was updated after my last
>> review/comments.  i have not read, nor claim any opinion about the
>> currently attached patch)
>>
>>
>>
>>
>> -Hoss
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by PabloS <pa...@gmail.com>.
Thanks hossman, but I've already 'solved' the problem without the need to
patch lucene. I had to code a bit around Lucene's visibility restrictions
but I've managed to completely skip the field caching mechanism and add
ehcache to it.

At the moment it seems to be working quite well, although not as fast as it
was when lucene performed the caching.

Thanks a lot for the info anyway.
Regards.

hossman wrote:
> 
> 
> : I'm having a similar problem with my application, although we are using
> : lucene 2.3.2. The problem we have is that we are required to sort on
> most of
> : the fields (20 at least). Is there any way of changing the cache being
> used?
> 
> there is a patch in Jira that takes a completley different approach
> towards dealing with sorting by using the *stored* values of the documents
> that match...
> 
> https://issues.apache.org/jira/browse/LUCENE-769
> 
> ...in theory it's "better" for use cases where:
> 
>   1) you are confident you are going to get back a "small" result set in
>      proportion to the total size of your index.
>   2) you need to support sorting on "lots" of fields and don't have enough
>      ram for all of the FieldCaches of those fields
> 
> So far the only person to test it out (and post comments) is the patch
> submitter, but if other people report success it might be worth adding as
> a contrib.
> 
> (Disclaimer: it looks like the patch was updated after my last 
> review/comments.  i have not read, nor claim any opinion about the 
> currently attached patch)
> 
> 
> 
> 
> -Hoss
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Chris Hostetter <ho...@fucit.org>.
: I'm having a similar problem with my application, although we are using
: lucene 2.3.2. The problem we have is that we are required to sort on most of
: the fields (20 at least). Is there any way of changing the cache being used?

there is a patch in Jira that takes a completley different approach
towards dealing with sorting by using the *stored* values of the documents
that match...

https://issues.apache.org/jira/browse/LUCENE-769

...in theory it's "better" for use cases where:

  1) you are confident you are going to get back a "small" result set in
     proportion to the total size of your index.
  2) you need to support sorting on "lots" of fields and don't have enough
     ram for all of the FieldCaches of those fields

So far the only person to test it out (and post comments) is the patch
submitter, but if other people report success it might be worth adding as
a contrib.

(Disclaimer: it looks like the patch was updated after my last 
review/comments.  i have not read, nor claim any opinion about the 
currently attached patch)




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by PabloS <pa...@gmail.com>.
Thanks for the quick reply :). For now, I'd settle with just storing cache
values in soft references so at least the GC would be able to free up some
space when it needs to.

I think I'll just try to override the default sorting mechanism by
subclassing FieldSortedHitQueue. I'll let you know how it goes.

If anyone else has any idea please do let me know.
Regards.
Pablo.



markrmiller wrote:
> 
> 20 fields on a huge index? Wow - not sure there is a ton you can do with 
> that...anyone have any suggestions for that one? Distributed should help 
> I suppose, but thats a lot of sort fields for a large index.
> 
> If LUCENE-831 ever gets off the ground you will be able to change the 
> cache used, and possibly use something that spills over to disk.
> 
> PabloS wrote:
>> Hi,
>>
>> I'm having a similar problem with my application, although we are using
>> lucene 2.3.2. The problem we have is that we are required to sort on most
>> of
>> the fields (20 at least). Is there any way of changing the cache being
>> used?
>> I can't seem to find a way, since the cache is being accessed using the
>> FieldCache.DEFAULT static field..
>>
>> Any tip would be appreciated, otherwise I'll have to start looking for a
>> clustered solution like Todd.
>>
>> Thanks in advance.
>> Pablo
>>
>>
>>
>>
>> markrmiller wrote:
>>   
>>> The term, terminfo, indexreader internals stuff is prob on the low end 
>>> compared to the size of your field caches (needed for sorting). If you 
>>> are sorting by String I think the space needed is 32 bits x number of 
>>> docs + an array to hold all of the unique terms. So checking 300 million 
>>> docs (I know you are actually breaking it up smaller than that, but for 
>>> example) and ignoring things like String chars being variable byte 
>>> lengths and storing the length, etc and randomly picking 50000 unique 
>>> terms at 6 chars per:
>>>
>>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
>>>
>>> Thats per field your sorting on. If you are sorting on an int field it 
>>> should be closer to 32 bits x num docs - shorts, 32 bits x num docs,
>>> etc.
>>>
>>> So you have those field caches, plus the IndexReader terminfo, term 
>>> stuff, plus whatever RAM your app needs beyond Lucene. 4 gig might just 
>>> not *quite* cut it is my guess.
>>>
>>> Todd Benge wrote:
>>>     
>>>> There's usually only a couple sort fields and a bunch of terms in the
>>>> various indices.  The terms are user entered on various media so the
>>>> number of terms is very large.
>>>>
>>>> Thanks for the help.
>>>>
>>>> Todd
>>>>
>>>>
>>>>
>>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>>>   
>>>>       
>>>>> Hi,
>>>>>
>>>>> I'm the lead engineer for search on a large website using lucene for
>>>>> search.
>>>>>
>>>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>>>  up to ~ 60G.
>>>>>
>>>>> The indices are sorted into 4 different Multisearcher with the largest
>>>>> handling ~50G.
>>>>>
>>>>> The code is basically like the following:
>>>>>
>>>>> private static MultiSearcher searcher;
>>>>>
>>>>> public void init(File files) {
>>>>>
>>>>>      IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>>      int i = 0;
>>>>>      for ( File file: files ) {
>>>>>           searchers[i++] = new
>>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>>      }
>>>>>
>>>>> searcher = new MultiSearcher(searchers);
>>>>> }
>>>>>
>>>>> public Searcher getSearcher() {
>>>>>    return searcher;
>>>>> }
>>>>>
>>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>>> Performance is good but servers are consistently hanging with
>>>>> OutOfMemory errors.
>>>>>
>>>>> We're allocating 4G in the heap to each server.
>>>>>
>>>>> Is there any way to control the amount of memory Lucene consume for
>>>>> caching?  Any other suggestions on fixing the memory errors?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Todd
>>>>>
>>>>>     
>>>>>         
>>>>   
>>>>       
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>     
>>
>>   
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20264511.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Mark Miller <ma...@gmail.com>.
20 fields on a huge index? Wow - not sure there is a ton you can do with 
that...anyone have any suggestions for that one? Distributed should help 
I suppose, but thats a lot of sort fields for a large index.

If LUCENE-831 ever gets off the ground you will be able to change the 
cache used, and possibly use something that spills over to disk.

PabloS wrote:
> Hi,
>
> I'm having a similar problem with my application, although we are using
> lucene 2.3.2. The problem we have is that we are required to sort on most of
> the fields (20 at least). Is there any way of changing the cache being used?
> I can't seem to find a way, since the cache is being accessed using the
> FieldCache.DEFAULT static field..
>
> Any tip would be appreciated, otherwise I'll have to start looking for a
> clustered solution like Todd.
>
> Thanks in advance.
> Pablo
>
>
>
>
> markrmiller wrote:
>   
>> The term, terminfo, indexreader internals stuff is prob on the low end 
>> compared to the size of your field caches (needed for sorting). If you 
>> are sorting by String I think the space needed is 32 bits x number of 
>> docs + an array to hold all of the unique terms. So checking 300 million 
>> docs (I know you are actually breaking it up smaller than that, but for 
>> example) and ignoring things like String chars being variable byte 
>> lengths and storing the length, etc and randomly picking 50000 unique 
>> terms at 6 chars per:
>>
>> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
>>
>> Thats per field your sorting on. If you are sorting on an int field it 
>> should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.
>>
>> So you have those field caches, plus the IndexReader terminfo, term 
>> stuff, plus whatever RAM your app needs beyond Lucene. 4 gig might just 
>> not *quite* cut it is my guess.
>>
>> Todd Benge wrote:
>>     
>>> There's usually only a couple sort fields and a bunch of terms in the
>>> various indices.  The terms are user entered on various media so the
>>> number of terms is very large.
>>>
>>> Thanks for the help.
>>>
>>> Todd
>>>
>>>
>>>
>>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>>   
>>>       
>>>> Hi,
>>>>
>>>> I'm the lead engineer for search on a large website using lucene for
>>>> search.
>>>>
>>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>>  up to ~ 60G.
>>>>
>>>> The indices are sorted into 4 different Multisearcher with the largest
>>>> handling ~50G.
>>>>
>>>> The code is basically like the following:
>>>>
>>>> private static MultiSearcher searcher;
>>>>
>>>> public void init(File files) {
>>>>
>>>>      IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>>      int i = 0;
>>>>      for ( File file: files ) {
>>>>           searchers[i++] = new
>>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>>      }
>>>>
>>>> searcher = new MultiSearcher(searchers);
>>>> }
>>>>
>>>> public Searcher getSearcher() {
>>>>    return searcher;
>>>> }
>>>>
>>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>>> Performance is good but servers are consistently hanging with
>>>> OutOfMemory errors.
>>>>
>>>> We're allocating 4G in the heap to each server.
>>>>
>>>> Is there any way to control the amount of memory Lucene consume for
>>>> caching?  Any other suggestions on fixing the memory errors?
>>>>
>>>> Thanks,
>>>>
>>>> Todd
>>>>
>>>>     
>>>>         
>>>   
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by PabloS <pa...@gmail.com>.
Hi,

I'm having a similar problem with my application, although we are using
lucene 2.3.2. The problem we have is that we are required to sort on most of
the fields (20 at least). Is there any way of changing the cache being used?
I can't seem to find a way, since the cache is being accessed using the
FieldCache.DEFAULT static field..

Any tip would be appreciated, otherwise I'll have to start looking for a
clustered solution like Todd.

Thanks in advance.
Pablo




markrmiller wrote:
> 
> The term, terminfo, indexreader internals stuff is prob on the low end 
> compared to the size of your field caches (needed for sorting). If you 
> are sorting by String I think the space needed is 32 bits x number of 
> docs + an array to hold all of the unique terms. So checking 300 million 
> docs (I know you are actually breaking it up smaller than that, but for 
> example) and ignoring things like String chars being variable byte 
> lengths and storing the length, etc and randomly picking 50000 unique 
> terms at 6 chars per:
> 
> 32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes
> 
> Thats per field your sorting on. If you are sorting on an int field it 
> should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.
> 
> So you have those field caches, plus the IndexReader terminfo, term 
> stuff, plus whatever RAM your app needs beyond Lucene. 4 gig might just 
> not *quite* cut it is my guess.
> 
> Todd Benge wrote:
>> There's usually only a couple sort fields and a bunch of terms in the
>> various indices.  The terms are user entered on various media so the
>> number of terms is very large.
>>
>> Thanks for the help.
>>
>> Todd
>>
>>
>>
>> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>>   
>>> Hi,
>>>
>>> I'm the lead engineer for search on a large website using lucene for
>>> search.
>>>
>>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>>  up to ~ 60G.
>>>
>>> The indices are sorted into 4 different Multisearcher with the largest
>>> handling ~50G.
>>>
>>> The code is basically like the following:
>>>
>>> private static MultiSearcher searcher;
>>>
>>> public void init(File files) {
>>>
>>>      IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>>      int i = 0;
>>>      for ( File file: files ) {
>>>           searchers[i++] = new
>>> IndexSearcher(FSDirectory.getDirectory(file);
>>>      }
>>>
>>> searcher = new MultiSearcher(searchers);
>>> }
>>>
>>> public Searcher getSearcher() {
>>>    return searcher;
>>> }
>>>
>>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>>> Performance is good but servers are consistently hanging with
>>> OutOfMemory errors.
>>>
>>> We're allocating 4G in the heap to each server.
>>>
>>> Is there any way to control the amount of memory Lucene consume for
>>> caching?  Any other suggestions on fixing the memory errors?
>>>
>>> Thanks,
>>>
>>> Todd
>>>
>>>     
>>
>>   
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20263342.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Mark Miller <ma...@gmail.com>.
The term, terminfo, indexreader internals stuff is prob on the low end 
compared to the size of your field caches (needed for sorting). If you 
are sorting by String I think the space needed is 32 bits x number of 
docs + an array to hold all of the unique terms. So checking 300 million 
docs (I know you are actually breaking it up smaller than that, but for 
example) and ignoring things like String chars being variable byte 
lengths and storing the length, etc and randomly picking 50000 unique 
terms at 6 chars per:

32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138 megabytes

Thats per field your sorting on. If you are sorting on an int field it 
should be closer to 32 bits x num docs - shorts, 32 bits x num docs, etc.

So you have those field caches, plus the IndexReader terminfo, term 
stuff, plus whatever RAM your app needs beyond Lucene. 4 gig might just 
not *quite* cut it is my guess.

Todd Benge wrote:
> There's usually only a couple sort fields and a bunch of terms in the
> various indices.  The terms are user entered on various media so the
> number of terms is very large.
>
> Thanks for the help.
>
> Todd
>
>
>
> On 10/29/08, Todd Benge <to...@gmail.com> wrote:
>   
>> Hi,
>>
>> I'm the lead engineer for search on a large website using lucene for search.
>>
>> We're indexing about 300M documents in ~ 100 indices.  The indices add
>>  up to ~ 60G.
>>
>> The indices are sorted into 4 different Multisearcher with the largest
>> handling ~50G.
>>
>> The code is basically like the following:
>>
>> private static MultiSearcher searcher;
>>
>> public void init(File files) {
>>
>>      IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>>      int i = 0;
>>      for ( File file: files ) {
>>           searchers[i++] = new IndexSearcher(FSDirectory.getDirectory(file);
>>      }
>>
>> searcher = new MultiSearcher(searchers);
>> }
>>
>> public Searcher getSearcher() {
>>    return searcher;
>> }
>>
>> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
>> Performance is good but servers are consistently hanging with
>> OutOfMemory errors.
>>
>> We're allocating 4G in the heap to each server.
>>
>> Is there any way to control the amount of memory Lucene consume for
>> caching?  Any other suggestions on fixing the memory errors?
>>
>> Thanks,
>>
>> Todd
>>
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: OutOfMemory Problems Lucene 2.4 / Tomcat

Posted by Todd Benge <to...@gmail.com>.
There's usually only a couple sort fields and a bunch of terms in the
various indices.  The terms are user entered on various media so the
number of terms is very large.

Thanks for the help.

Todd



On 10/29/08, Todd Benge <to...@gmail.com> wrote:
> Hi,
>
> I'm the lead engineer for search on a large website using lucene for search.
>
> We're indexing about 300M documents in ~ 100 indices.  The indices add
>  up to ~ 60G.
>
> The indices are sorted into 4 different Multisearcher with the largest
> handling ~50G.
>
> The code is basically like the following:
>
> private static MultiSearcher searcher;
>
> public void init(File files) {
>
>      IndexSearcer [] searchers = new IndexSearcher[files.length] ();
>      int i = 0;
>      for ( File file: files ) {
>           searchers[i++] = new IndexSearcher(FSDirectory.getDirectory(file);
>      }
>
> searcher = new MultiSearcher(searchers);
> }
>
> public Searcher getSearcher() {
>    return searcher;
> }
>
> We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
> Performance is good but servers are consistently hanging with
> OutOfMemory errors.
>
> We're allocating 4G in the heap to each server.
>
> Is there any way to control the amount of memory Lucene consume for
> caching?  Any other suggestions on fixing the memory errors?
>
> Thanks,
>
> Todd
>

-- 
Sent from Gmail for mobile | mobile.google.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org