You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by olivier sallou <ol...@gmail.com> on 2010/06/29 20:04:19 UTC
Faceted search outofmemory
Hi,
I try to make a faceted search on a very large index (around 200GB with 200M
doc).
I have an out of memory error. With no facet it works fine.
There are quite many questions around this but I could not find the answer.
How can we know the required memory when facets are used so that I try to
scale my server/index correctly to handle it.
Thanks
Olivier
Re: Re: Faceted search outofmemory
Posted by olivier sallou <ol...@gmail.com>.
I already use facet.limit in my query. I tried however facet.method=enum and
though it does not seem to fix everything, I have some requests without the
outofmemory error.
Best would be to have a calculation rule of required memory for such type of
query.
2010/6/29 Markus Jelsma <ma...@buyways.nl>
> http://wiki.apache.org/solr/SimpleFacetParameters#facet.limit
>
> -----Original message-----
> From: olivier sallou <ol...@gmail.com>
> Sent: Tue 29-06-2010 20:11
> To: solr-user@lucene.apache.org;
> Subject: Re: Faceted search outofmemory
>
> How do make paging over facets?
>
> 2010/6/29 Ankit Bhatnagar <ab...@vantage.com>
>
> >
> > Did you trying paging them?
> >
> >
> > -----Original Message-----
> > From: olivier sallou [mailto:olivier.sallou@gmail.com]
> > Sent: Tuesday, June 29, 2010 2:04 PM
> > To: solr-user@lucene.apache.org
> > Subject: Faceted search outofmemory
> >
> > Hi,
> > I try to make a faceted search on a very large index (around 200GB with
> > 200M
> > doc).
> > I have an out of memory error. With no facet it works fine.
> >
> > There are quite many questions around this but I could not find the
> answer.
> > How can we know the required memory when facets are used so that I try to
> > scale my server/index correctly to handle it.
> >
> > Thanks
> >
> > Olivier
> >
>
RE: Re: Faceted search outofmemory
Posted by Markus Jelsma <ma...@buyways.nl>.
http://wiki.apache.org/solr/SimpleFacetParameters#facet.limit
-----Original message-----
From: olivier sallou <ol...@gmail.com>
Sent: Tue 29-06-2010 20:11
To: solr-user@lucene.apache.org;
Subject: Re: Faceted search outofmemory
How do make paging over facets?
2010/6/29 Ankit Bhatnagar <ab...@vantage.com>
>
> Did you trying paging them?
>
>
> -----Original Message-----
> From: olivier sallou [mailto:olivier.sallou@gmail.com]
> Sent: Tuesday, June 29, 2010 2:04 PM
> To: solr-user@lucene.apache.org
> Subject: Faceted search outofmemory
>
> Hi,
> I try to make a faceted search on a very large index (around 200GB with
> 200M
> doc).
> I have an out of memory error. With no facet it works fine.
>
> There are quite many questions around this but I could not find the answer.
> How can we know the required memory when facets are used so that I try to
> scale my server/index correctly to handle it.
>
> Thanks
>
> Olivier
>
Re: Faceted search outofmemory
Posted by olivier sallou <ol...@gmail.com>.
How do make paging over facets?
2010/6/29 Ankit Bhatnagar <ab...@vantage.com>
>
> Did you trying paging them?
>
>
> -----Original Message-----
> From: olivier sallou [mailto:olivier.sallou@gmail.com]
> Sent: Tuesday, June 29, 2010 2:04 PM
> To: solr-user@lucene.apache.org
> Subject: Faceted search outofmemory
>
> Hi,
> I try to make a faceted search on a very large index (around 200GB with
> 200M
> doc).
> I have an out of memory error. With no facet it works fine.
>
> There are quite many questions around this but I could not find the answer.
> How can we know the required memory when facets are used so that I try to
> scale my server/index correctly to handle it.
>
> Thanks
>
> Olivier
>
RE: Faceted search outofmemory
Posted by Ankit Bhatnagar <ab...@vantage.com>.
Did you trying paging them?
-----Original Message-----
From: olivier sallou [mailto:olivier.sallou@gmail.com]
Sent: Tuesday, June 29, 2010 2:04 PM
To: solr-user@lucene.apache.org
Subject: Faceted search outofmemory
Hi,
I try to make a faceted search on a very large index (around 200GB with 200M
doc).
I have an out of memory error. With no facet it works fine.
There are quite many questions around this but I could not find the answer.
How can we know the required memory when facets are used so that I try to
scale my server/index correctly to handle it.
Thanks
Olivier
Re: Faceted search outofmemory
Posted by Lance Norskog <go...@gmail.com>.
There is memory used for each facet. All of the facets are loaded for
any facet query. Your best shot is to limit the number of facets.
On Tue, Jun 29, 2010 at 11:42 AM, olivier sallou
<ol...@gmail.com> wrote:
> I have given 6G to Tomcat. Using facet.method=enum and facet.limit seems to
> fix the issue with a few tests, but I do know that it is not a "final"
> solution. Will work under certain configurations.
>
> Real "issue" is to be able to know what is the required RAM for an index...
>
> 2010/6/29 Nagelberg, Kallin <KN...@globeandmail.com>
>
>> How much memory have you given the solr jvm? Many servlet containers have
>> small amount by default.
>>
>> -Kal
>>
>> -----Original Message-----
>> From: olivier sallou [mailto:olivier.sallou@gmail.com]
>> Sent: Tuesday, June 29, 2010 2:04 PM
>> To: solr-user@lucene.apache.org
>> Subject: Faceted search outofmemory
>>
>> Hi,
>> I try to make a faceted search on a very large index (around 200GB with
>> 200M
>> doc).
>> I have an out of memory error. With no facet it works fine.
>>
>> There are quite many questions around this but I could not find the answer.
>> How can we know the required memory when facets are used so that I try to
>> scale my server/index correctly to handle it.
>>
>> Thanks
>>
>> Olivier
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Faceted search outofmemory
Posted by olivier sallou <ol...@gmail.com>.
I have given 6G to Tomcat. Using facet.method=enum and facet.limit seems to
fix the issue with a few tests, but I do know that it is not a "final"
solution. Will work under certain configurations.
Real "issue" is to be able to know what is the required RAM for an index...
2010/6/29 Nagelberg, Kallin <KN...@globeandmail.com>
> How much memory have you given the solr jvm? Many servlet containers have
> small amount by default.
>
> -Kal
>
> -----Original Message-----
> From: olivier sallou [mailto:olivier.sallou@gmail.com]
> Sent: Tuesday, June 29, 2010 2:04 PM
> To: solr-user@lucene.apache.org
> Subject: Faceted search outofmemory
>
> Hi,
> I try to make a faceted search on a very large index (around 200GB with
> 200M
> doc).
> I have an out of memory error. With no facet it works fine.
>
> There are quite many questions around this but I could not find the answer.
> How can we know the required memory when facets are used so that I try to
> scale my server/index correctly to handle it.
>
> Thanks
>
> Olivier
>
RE: Faceted search outofmemory
Posted by "Nagelberg, Kallin" <KN...@globeandmail.com>.
How much memory have you given the solr jvm? Many servlet containers have small amount by default.
-Kal
-----Original Message-----
From: olivier sallou [mailto:olivier.sallou@gmail.com]
Sent: Tuesday, June 29, 2010 2:04 PM
To: solr-user@lucene.apache.org
Subject: Faceted search outofmemory
Hi,
I try to make a faceted search on a very large index (around 200GB with 200M
doc).
I have an out of memory error. With no facet it works fine.
There are quite many questions around this but I could not find the answer.
How can we know the required memory when facets are used so that I try to
scale my server/index correctly to handle it.
Thanks
Olivier