You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by gururaj kosuru <gu...@gmail.com> on 2013/06/11 06:52:51 UTC
reg: efficient querying using solr
Hello,
I have recently started using solr 3.4 and have a standalone
system deployed that has 40,000,000 data rows with 3 indexed fields
totalling around 9 GB. I have given a Heap size of 2GB and I run the
instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
searching among the indexed fields using the keyword 'OR' to search for
multiple words in a single field and the keyword 'AND' to get the
intersection of multiple matches from different fields. The problem I face
is an out of memory exception that happens when the solr core has been
queried for a long time and I am forced to restart the solr instance. My
primary questions are :
1. Can I make any changes in my query as I have noticed that if I divide my
query into parts eg- if my query has "A and B and C", executing only A
gives out 75,000 results. However, If I run the whole query, I get an out
of memory error in SegmentNorms.java at line 156 which allocates a new byte
array of size count.
2. Does my index need to be able to fit into the RAM in one go?
3. Will moving to Solr Cloud solve the out of memory exception issue and if
so, what will be the ratio of my RAM size to the shard size that should be
used?
Thanks a lot,
Gururaj
Re: reg: efficient querying using solr
Posted by Otis Gospodnetic <ot...@gmail.com>.
You can't calculate it precisely, but you can test it. This is where EC2 is
handy.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jun 11, 2013 2:16 AM, "gururaj kosuru" <gu...@gmail.com> wrote:
> How can one calculate an ideal max shard size for a solr core instance if I
> am running a cloud with multiple systems of 4GB?
>
> Thanks
>
>
> On 11 June 2013 11:18, Walter Underwood <wu...@wunderwood.org> wrote:
>
> > An index does not need to fit into the heap. But a 4GB machine is almost
> > certainly too small to run Solr with 40 million documents.
> >
> > wunder
> >
> > On Jun 10, 2013, at 10:36 PM, gururaj kosuru wrote:
> >
> > > Hi Walter,
> > > thanks for replying. Do you mean that it is necessary
> for
> > > the index to fit into the heap? if so, will a heap size that is greater
> > > than the actual RAM size slow down the queries?
> > >
> > > Thanks,
> > > Gururaj
> > >
> > >
> > > On 11 June 2013 10:36, Walter Underwood <wu...@wunderwood.org> wrote:
> > >
> > >> 2GB is a rather small heap. Our production systems run with 8GB and
> > >> smaller indexes than that. Our dev and test systems run with 6GB
> heaps.
> > >>
> > >> wunder
> > >>
> > >> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote:
> > >>
> > >>> Hello,
> > >>> I have recently started using solr 3.4 and have a standalone
> > >>> system deployed that has 40,000,000 data rows with 3 indexed fields
> > >>> totalling around 9 GB. I have given a Heap size of 2GB and I run the
> > >>> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
> > >>> searching among the indexed fields using the keyword 'OR' to search
> for
> > >>> multiple words in a single field and the keyword 'AND' to get the
> > >>> intersection of multiple matches from different fields. The problem I
> > >> face
> > >>> is an out of memory exception that happens when the solr core has
> been
> > >>> queried for a long time and I am forced to restart the solr instance.
> > My
> > >>> primary questions are :
> > >>>
> > >>> 1. Can I make any changes in my query as I have noticed that if I
> > divide
> > >> my
> > >>> query into parts eg- if my query has "A and B and C", executing only
> A
> > >>> gives out 75,000 results. However, If I run the whole query, I get an
> > out
> > >>> of memory error in SegmentNorms.java at line 156 which allocates a
> new
> > >> byte
> > >>> array of size count.
> > >>>
> > >>> 2. Does my index need to be able to fit into the RAM in one go?
> > >>>
> > >>> 3. Will moving to Solr Cloud solve the out of memory exception issue
> > and
> > >> if
> > >>> so, what will be the ratio of my RAM size to the shard size that
> should
> > >> be
> > >>> used?
> > >>>
> > >>> Thanks a lot,
> > >>> Gururaj
> > >>
> > >> --
> > >> Walter Underwood
> > >> wunder@wunderwood.org
> > >>
> > >>
> > >>
> > >>
> >
> > --
> > Walter Underwood
> > wunder@wunderwood.org
> >
> >
> >
> >
>
Re: reg: efficient querying using solr
Posted by gururaj kosuru <gu...@gmail.com>.
Thanks Walter, Shawn and Otis for the assistance, I will look into tuning
the parameters by experimenting as seems to be the only way to go.
On 11 June 2013 19:17, Shawn Heisey <so...@elyograg.org> wrote:
> On 6/11/2013 12:15 AM, gururaj kosuru wrote:
> > How can one calculate an ideal max shard size for a solr core instance
> if I
> > am running a cloud with multiple systems of 4GB?
>
> That question is impossible to answer without experimentation, but
> here's a good starting point. That's all it is, a starting point:
>
> http://wiki.apache.org/solr/SolrPerformanceProblems
>
> Thanks,
> Shawn
>
>
Re: reg: efficient querying using solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/11/2013 12:15 AM, gururaj kosuru wrote:
> How can one calculate an ideal max shard size for a solr core instance if I
> am running a cloud with multiple systems of 4GB?
That question is impossible to answer without experimentation, but
here's a good starting point. That's all it is, a starting point:
http://wiki.apache.org/solr/SolrPerformanceProblems
Thanks,
Shawn
Re: reg: efficient querying using solr
Posted by gururaj kosuru <gu...@gmail.com>.
How can one calculate an ideal max shard size for a solr core instance if I
am running a cloud with multiple systems of 4GB?
Thanks
On 11 June 2013 11:18, Walter Underwood <wu...@wunderwood.org> wrote:
> An index does not need to fit into the heap. But a 4GB machine is almost
> certainly too small to run Solr with 40 million documents.
>
> wunder
>
> On Jun 10, 2013, at 10:36 PM, gururaj kosuru wrote:
>
> > Hi Walter,
> > thanks for replying. Do you mean that it is necessary for
> > the index to fit into the heap? if so, will a heap size that is greater
> > than the actual RAM size slow down the queries?
> >
> > Thanks,
> > Gururaj
> >
> >
> > On 11 June 2013 10:36, Walter Underwood <wu...@wunderwood.org> wrote:
> >
> >> 2GB is a rather small heap. Our production systems run with 8GB and
> >> smaller indexes than that. Our dev and test systems run with 6GB heaps.
> >>
> >> wunder
> >>
> >> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote:
> >>
> >>> Hello,
> >>> I have recently started using solr 3.4 and have a standalone
> >>> system deployed that has 40,000,000 data rows with 3 indexed fields
> >>> totalling around 9 GB. I have given a Heap size of 2GB and I run the
> >>> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
> >>> searching among the indexed fields using the keyword 'OR' to search for
> >>> multiple words in a single field and the keyword 'AND' to get the
> >>> intersection of multiple matches from different fields. The problem I
> >> face
> >>> is an out of memory exception that happens when the solr core has been
> >>> queried for a long time and I am forced to restart the solr instance.
> My
> >>> primary questions are :
> >>>
> >>> 1. Can I make any changes in my query as I have noticed that if I
> divide
> >> my
> >>> query into parts eg- if my query has "A and B and C", executing only A
> >>> gives out 75,000 results. However, If I run the whole query, I get an
> out
> >>> of memory error in SegmentNorms.java at line 156 which allocates a new
> >> byte
> >>> array of size count.
> >>>
> >>> 2. Does my index need to be able to fit into the RAM in one go?
> >>>
> >>> 3. Will moving to Solr Cloud solve the out of memory exception issue
> and
> >> if
> >>> so, what will be the ratio of my RAM size to the shard size that should
> >> be
> >>> used?
> >>>
> >>> Thanks a lot,
> >>> Gururaj
> >>
> >> --
> >> Walter Underwood
> >> wunder@wunderwood.org
> >>
> >>
> >>
> >>
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
>
Re: reg: efficient querying using solr
Posted by Walter Underwood <wu...@wunderwood.org>.
An index does not need to fit into the heap. But a 4GB machine is almost certainly too small to run Solr with 40 million documents.
wunder
On Jun 10, 2013, at 10:36 PM, gururaj kosuru wrote:
> Hi Walter,
> thanks for replying. Do you mean that it is necessary for
> the index to fit into the heap? if so, will a heap size that is greater
> than the actual RAM size slow down the queries?
>
> Thanks,
> Gururaj
>
>
> On 11 June 2013 10:36, Walter Underwood <wu...@wunderwood.org> wrote:
>
>> 2GB is a rather small heap. Our production systems run with 8GB and
>> smaller indexes than that. Our dev and test systems run with 6GB heaps.
>>
>> wunder
>>
>> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote:
>>
>>> Hello,
>>> I have recently started using solr 3.4 and have a standalone
>>> system deployed that has 40,000,000 data rows with 3 indexed fields
>>> totalling around 9 GB. I have given a Heap size of 2GB and I run the
>>> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
>>> searching among the indexed fields using the keyword 'OR' to search for
>>> multiple words in a single field and the keyword 'AND' to get the
>>> intersection of multiple matches from different fields. The problem I
>> face
>>> is an out of memory exception that happens when the solr core has been
>>> queried for a long time and I am forced to restart the solr instance. My
>>> primary questions are :
>>>
>>> 1. Can I make any changes in my query as I have noticed that if I divide
>> my
>>> query into parts eg- if my query has "A and B and C", executing only A
>>> gives out 75,000 results. However, If I run the whole query, I get an out
>>> of memory error in SegmentNorms.java at line 156 which allocates a new
>> byte
>>> array of size count.
>>>
>>> 2. Does my index need to be able to fit into the RAM in one go?
>>>
>>> 3. Will moving to Solr Cloud solve the out of memory exception issue and
>> if
>>> so, what will be the ratio of my RAM size to the shard size that should
>> be
>>> used?
>>>
>>> Thanks a lot,
>>> Gururaj
>>
>> --
>> Walter Underwood
>> wunder@wunderwood.org
>>
>>
>>
>>
--
Walter Underwood
wunder@wunderwood.org
Re: reg: efficient querying using solr
Posted by gururaj kosuru <gu...@gmail.com>.
Hi Walter,
thanks for replying. Do you mean that it is necessary for
the index to fit into the heap? if so, will a heap size that is greater
than the actual RAM size slow down the queries?
Thanks,
Gururaj
On 11 June 2013 10:36, Walter Underwood <wu...@wunderwood.org> wrote:
> 2GB is a rather small heap. Our production systems run with 8GB and
> smaller indexes than that. Our dev and test systems run with 6GB heaps.
>
> wunder
>
> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote:
>
> > Hello,
> > I have recently started using solr 3.4 and have a standalone
> > system deployed that has 40,000,000 data rows with 3 indexed fields
> > totalling around 9 GB. I have given a Heap size of 2GB and I run the
> > instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
> > searching among the indexed fields using the keyword 'OR' to search for
> > multiple words in a single field and the keyword 'AND' to get the
> > intersection of multiple matches from different fields. The problem I
> face
> > is an out of memory exception that happens when the solr core has been
> > queried for a long time and I am forced to restart the solr instance. My
> > primary questions are :
> >
> > 1. Can I make any changes in my query as I have noticed that if I divide
> my
> > query into parts eg- if my query has "A and B and C", executing only A
> > gives out 75,000 results. However, If I run the whole query, I get an out
> > of memory error in SegmentNorms.java at line 156 which allocates a new
> byte
> > array of size count.
> >
> > 2. Does my index need to be able to fit into the RAM in one go?
> >
> > 3. Will moving to Solr Cloud solve the out of memory exception issue and
> if
> > so, what will be the ratio of my RAM size to the shard size that should
> be
> > used?
> >
> > Thanks a lot,
> > Gururaj
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
>
Re: reg: efficient querying using solr
Posted by Walter Underwood <wu...@wunderwood.org>.
2GB is a rather small heap. Our production systems run with 8GB and smaller indexes than that. Our dev and test systems run with 6GB heaps.
wunder
On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote:
> Hello,
> I have recently started using solr 3.4 and have a standalone
> system deployed that has 40,000,000 data rows with 3 indexed fields
> totalling around 9 GB. I have given a Heap size of 2GB and I run the
> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve
> searching among the indexed fields using the keyword 'OR' to search for
> multiple words in a single field and the keyword 'AND' to get the
> intersection of multiple matches from different fields. The problem I face
> is an out of memory exception that happens when the solr core has been
> queried for a long time and I am forced to restart the solr instance. My
> primary questions are :
>
> 1. Can I make any changes in my query as I have noticed that if I divide my
> query into parts eg- if my query has "A and B and C", executing only A
> gives out 75,000 results. However, If I run the whole query, I get an out
> of memory error in SegmentNorms.java at line 156 which allocates a new byte
> array of size count.
>
> 2. Does my index need to be able to fit into the RAM in one go?
>
> 3. Will moving to Solr Cloud solve the out of memory exception issue and if
> so, what will be the ratio of my RAM size to the shard size that should be
> used?
>
> Thanks a lot,
> Gururaj
--
Walter Underwood
wunder@wunderwood.org