You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jawahar Lal <jl...@chambal.com> on 2010/12/28 08:00:02 UTC
Outof memory exception on using Integer.MaxValue
Hi,
I am using multiSearcher as
objMultiSearcher.search(query,null,Integer.MaxValue);
I searched a query, which should & does not return any hit. As I run the
query, the above statement throw OutofMemory exception. I change the
Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
Why its thowing memory exception, even there is not hit found for the query
?
Thanks
Re: Outof memory exception on using Integer.MaxValue
Posted by Simon Willnauer <si...@googlemail.com>.
MultiSearcher uses the given integer as the expected number of results
and creates a hitqueue for it. Since you are asking for INT_MAX hits
Lucene creates a hitQueue for either INT_MAX or maxDoc() documents and
preallocates that memory. So in the worst case it creates an Object
array with INTEGER_MAX elements - each taking 8 bytes as the object
reference on a 64 bit VM - unless you have lots of heap space this
will make your vm barf.
you should only specify the number of documents you really want no
blindly get all of them.
why your query returns 0 docs is a different question.
simon
On Tue, Dec 28, 2010 at 8:00 AM, Jawahar Lal <jl...@chambal.com> wrote:
> Hi,
>
> I am using multiSearcher as
>
> objMultiSearcher.search(query,null,Integer.MaxValue);
>
> I searched a query, which should & does not return any hit. As I run the
> query, the above statement throw OutofMemory exception. I change the
> Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
>
> Why its thowing memory exception, even there is not hit found for the query
> ?
>
> Thanks
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Outof memory exception on using Integer.MaxValue
Posted by Jawahar Lal <jl...@chambal.com>.
Thanks all of you.
On 28 December 2010 15:54, Paul Libbrecht <pa...@hoplahup.net> wrote:
> I also not that this is a fundamental characteristic of the great
> performance of Lucene and its related products since it allows cleanly
> managed resources. "this" is generally called paging.
>
> paul
>
>
> Le 28 déc. 2010 à 10:32, Uwe Schindler a écrit :
>
> > The TopDocs returning methods are not intended to get all results, so Max
> Int is not a valid value. The maximum hit count allocates a PriorityQueue
> with so many slots. To retrieve all hits (unsorted) you have to write your
> own collector. But that is not the natural use case for a full text engine,
> which is to return the top n ranking results. For this reason its not
> allowed to go beyond page 50 or like that in Google (just as example). So
> choose n as big as how many top ranking results you are interested in. This
> number is normally not so big, as most users are only interested in top
> ranking results.
> >
> > Uwe
> >
> >
> >
> > "Jawahar Lal" <jl...@chambal.com> schrieb:
> >
> >> I am using lucene 3.0.3
> >>
> >> On 28 December 2010 14:06, Laxmilal Menariya <lm...@chambal.com>
> >> wrote:
> >>
> >>> Which version you are trying of lucene ?
> >>>
> >>> On Tue, Dec 28, 2010 at 12:30 PM, Jawahar Lal <jl...@chambal.com>
> >> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I am using multiSearcher as
> >>>>
> >>>> objMultiSearcher.search(query,null,Integer.MaxValue);
> >>>>
> >>>> I searched a query, which should & does not return any hit. As I
> >> run the
> >>>> query, the above statement throw OutofMemory exception. I change
> >> the
> >>>> Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
> >>>>
> >>>> Why its thowing memory exception, even there is not hit found for
> >> the
> >>> query
> >>>> ?
> >>>>
> >>>> Thanks
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Laxmilal Menariya
> >>>
> >
> > --
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, 28213 Bremen
> > http://www.thetaphi.de
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Outof memory exception on using Integer.MaxValue
Posted by Paul Libbrecht <pa...@hoplahup.net>.
I also not that this is a fundamental characteristic of the great performance of Lucene and its related products since it allows cleanly managed resources. "this" is generally called paging.
paul
Le 28 déc. 2010 à 10:32, Uwe Schindler a écrit :
> The TopDocs returning methods are not intended to get all results, so Max Int is not a valid value. The maximum hit count allocates a PriorityQueue with so many slots. To retrieve all hits (unsorted) you have to write your own collector. But that is not the natural use case for a full text engine, which is to return the top n ranking results. For this reason its not allowed to go beyond page 50 or like that in Google (just as example). So choose n as big as how many top ranking results you are interested in. This number is normally not so big, as most users are only interested in top ranking results.
>
> Uwe
>
>
>
> "Jawahar Lal" <jl...@chambal.com> schrieb:
>
>> I am using lucene 3.0.3
>>
>> On 28 December 2010 14:06, Laxmilal Menariya <lm...@chambal.com>
>> wrote:
>>
>>> Which version you are trying of lucene ?
>>>
>>> On Tue, Dec 28, 2010 at 12:30 PM, Jawahar Lal <jl...@chambal.com>
>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using multiSearcher as
>>>>
>>>> objMultiSearcher.search(query,null,Integer.MaxValue);
>>>>
>>>> I searched a query, which should & does not return any hit. As I
>> run the
>>>> query, the above statement throw OutofMemory exception. I change
>> the
>>>> Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
>>>>
>>>> Why its thowing memory exception, even there is not hit found for
>> the
>>> query
>>>> ?
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Laxmilal Menariya
>>>
>
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Outof memory exception on using Integer.MaxValue
Posted by Uwe Schindler <uw...@thetaphi.de>.
The TopDocs returning methods are not intended to get all results, so Max Int is not a valid value. The maximum hit count allocates a PriorityQueue with so many slots. To retrieve all hits (unsorted) you have to write your own collector. But that is not the natural use case for a full text engine, which is to return the top n ranking results. For this reason its not allowed to go beyond page 50 or like that in Google (just as example). So choose n as big as how many top ranking results you are interested in. This number is normally not so big, as most users are only interested in top ranking results.
Uwe
"Jawahar Lal" <jl...@chambal.com> schrieb:
>I am using lucene 3.0.3
>
>On 28 December 2010 14:06, Laxmilal Menariya <lm...@chambal.com>
>wrote:
>
>> Which version you are trying of lucene ?
>>
>> On Tue, Dec 28, 2010 at 12:30 PM, Jawahar Lal <jl...@chambal.com>
>wrote:
>>
>> > Hi,
>> >
>> > I am using multiSearcher as
>> >
>> > objMultiSearcher.search(query,null,Integer.MaxValue);
>> >
>> > I searched a query, which should & does not return any hit. As I
>run the
>> > query, the above statement throw OutofMemory exception. I change
>the
>> > Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
>> >
>> > Why its thowing memory exception, even there is not hit found for
>the
>> query
>> > ?
>> >
>> > Thanks
>> >
>>
>>
>>
>> --
>> Thanks,
>> Laxmilal Menariya
>>
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Outof memory exception on using Integer.MaxValue
Posted by Jawahar Lal <jl...@chambal.com>.
I am using lucene 3.0.3
On 28 December 2010 14:06, Laxmilal Menariya <lm...@chambal.com> wrote:
> Which version you are trying of lucene ?
>
> On Tue, Dec 28, 2010 at 12:30 PM, Jawahar Lal <jl...@chambal.com> wrote:
>
> > Hi,
> >
> > I am using multiSearcher as
> >
> > objMultiSearcher.search(query,null,Integer.MaxValue);
> >
> > I searched a query, which should & does not return any hit. As I run the
> > query, the above statement throw OutofMemory exception. I change the
> > Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
> >
> > Why its thowing memory exception, even there is not hit found for the
> query
> > ?
> >
> > Thanks
> >
>
>
>
> --
> Thanks,
> Laxmilal Menariya
>
Re: Outof memory exception on using Integer.MaxValue
Posted by Laxmilal Menariya <lm...@chambal.com>.
Which version you are trying of lucene ?
On Tue, Dec 28, 2010 at 12:30 PM, Jawahar Lal <jl...@chambal.com> wrote:
> Hi,
>
> I am using multiSearcher as
>
> objMultiSearcher.search(query,null,Integer.MaxValue);
>
> I searched a query, which should & does not return any hit. As I run the
> query, the above statement throw OutofMemory exception. I change the
> Integer.MaxValue TO 100, then I again searched, it returned 0 hit.
>
> Why its thowing memory exception, even there is not hit found for the query
> ?
>
> Thanks
>
--
Thanks,
Laxmilal Menariya