You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sujatha Arun <su...@gmail.com> on 2012/08/17 08:21:12 UTC
Muticore Sharding
Hello,
One of the Index in a multicore set up has a 3GB+ index ,and it seems to
take around 5000ms+ to return simple boolean queries . The Index is
not optimized
Would it make sense to shard the index as cores in the same server to
expect better response time?
do I have to re index all over again to distribute the documents between 2
shards ?
Regards
Sujatha
Re: Muticore Sharding
Posted by Erick Erickson <er...@gmail.com>.
Well, as Eric said, you're apparently calculating
the scores on over 3M documents since you're
querying on very common words ('if', 'a', etc).
Have you tried simply using a stopword list?
Best
Erick
On Fri, Aug 17, 2012 at 10:53 AM, Sujatha Arun <su...@gmail.com> wrote:
> Hi,
>
> This is the parsed query string which *returns 3331234 documents*
>
> <str name="parsedquery_toString">content:elena content:read content:a
> content:rate content:r content:page content:per content:minut content:a
> content:total content:m content:minut</str> re
>
> and a Boolean "AND " query of the above query takes 410 ms and *returns
> just 1 result*
>
> <str name="parsedquery_toString">+content:elena +content:read +content:a
> +content:rate +content:r +content:page +content:per +content:minut
> +content:a +content:total +content:m +content:minut</str>
>
> so the main difference being the number of results returned ,so is this
> i/0 issue or is this time expected given the number of documents returned.
>
> Also what is the difference between Qtime and timing shown as a part of
> Debugquery?
>
> Regards
> Sujatha
>
> Regards
> Sujatha
>
>
>
> On Fri, Aug 17, 2012 at 8:35 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
>
>> First, provide us with the additional info Erik requested in his last
>> reply.
>>
>> Then, can you provide a snippet from your Solr log file that shows a
>> couple of queries. Maybe there is something else going on or some
>> exceptions, or something. Or at least to show us the quey times in context.
>>
>> Finally, has this been happening from the very beginning, or did it begin
>> to grow gradually and steadily, or did it just suddenly start to happen
>> even though it had been fine just a few minutes earlier? If the latter, had
>> there been and system or JVM or Solr configuration changes between the time
>> queries were fast and when they became slow?
>>
>> Thanks.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Sujatha Arun
>> Sent: Friday, August 17, 2012 10:50 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Muticore Sharding
>>
>>
>> Sorry typo ,I meant upwards of 20s at any time. What should I be looking
>> at?
>>
>> Regards
>> Sujatha
>>
>> On Fri, Aug 17, 2012 at 8:09 PM, Sujatha Arun <su...@gmail.com> wrote:
>>
>> Erik,
>>>
>>> What could be the issue Load / I/O ? It seems to shows upwards of 20 ms
>>> at any time
>>>
>>> Regards
>>> Sujatha
>>>
>>>
>>> On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher <erik.hatcher@gmail.com
>>> >wrote:
>>>
>>> Sujatha - that query debug output shows only 218ms, so it isn't
>>>> representative of the issue you're reporting.
>>>>
>>>> Also, what's the query parse output? I imagine you're doing a boolean
>>>> OR query across all those terms (include "a"), yes? Maybe you'd rather
>>>> the
>>>> operator be AND?
>>>>
>>>> Erik
>>>>
>>>>
>>>> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>>>>
>>>> > No customization,its the default standard request handler.Solr Version
>>>> is
>>>> > 1.3
>>>> >
>>>> > "a" is not there in stop words
>>>> >
>>>> > Server Load ,i presume is not there , but not too sure ,not checked.
>>>> >
>>>> > RAM :
>>>> >
>>>> > TOTAL RAM :48GB
>>>> > RAM to JVM :18 GB ,Permgen =2GB
>>>> > TOTAL INDEX SIZE of all the multicore Instances =23GB
>>>> >
>>>> > Timing [Cut & paste ] , I have never looked at this before
>>>> >
>>>> > <str name="QParser">**OldLuceneQParser</str><lst name="timing"><double
>>>> > name="time">218.0</double><lst name="prepare"><double
>>>> > name="time">6.0</double><lst
>>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>>> > name="time">5.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.**
>>>> MoreLikeThisComponent"><double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>>> <double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>>> > name="time">0.0</double></lst>**</lst><lst name="process"><double
>>>> > name="time">211.0</double><lst
>>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.**
>>>> MoreLikeThisComponent"><double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>>> <double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>>> > name="time">211.0</double></**lst></lst></lst></lst>
>>>> >
>>>> > Regards
>>>> > Sujatha
>>>> >
>>>> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatcher@gmail.com
>>>> >wrote:
>>>> >
>>>> >> Just over 4M docs... no need to shard.
>>>> >>
>>>> >> Both your queries contain what is likely very common terms (content:a
>>>> in
>>>> >> the first one and content:1 in the second). Generally these are "stop
>>>> >> words" and removed either during indexing or querying, but I guess not
>>>> in
>>>> >> your case.
>>>> >>
>>>> >> What's your "standard" request handler look like? Doing anything
>>>> custom
>>>> >> in there?
>>>> >>
>>>> >> What are the timings of the components in the debugQuery=true >>
>>>> response?
>>>> >>
>>>> >> Is the server under load when you're issuing these queries? What
>>>> about
>>>> >> RAM?
>>>> >>
>>>> >> Again, what version of Solr?
>>>> >>
>>>> >> Erik
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>>>> >>
>>>> >>> Hi Erick,
>>>> >>>
>>>> >>> The number of documents is : 4389048
>>>> >>>
>>>> >>> I have given 2 queries below with timing and the number of hits
>>>> >>>
>>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>> >>>
>>>> >>
>>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>>> fq=&rows=100&start=0&q=**content:if+content:elena+**
>>>> content:reads+content:at+**content:a+content:rate+**
>>>> content:of+content:r+content:**pages+content:per+content:**
>>>> minute+content:for+content:a+**content:total+content:of+**
>>>> content:m+content:minutes}
>>>> >>> hits=3331109 status=0 QTime=22677
>>>> >>>
>>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>> >>>
>>>> >>
>>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>>> fq=&rows=100&start=0&q=**content:chapter+content:1}
>>>> >>> hits=1919445 status=0 QTime=6677
>>>> >>>
>>>> >>>
>>>> >>> Any time I execute the time taken to execute the first query is >
>>>> than
>>>> >>> 22secs ,now it took 33 secs to execute.
>>>> >>>
>>>> >>> Regards
>>>> >>> Sujatha
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <
>>>> erik.hatcher@gmail.com
>>>> >>> wrote:
>>>> >>>
>>>> >>>> How many documents do you have? What are the queries? I'd guess
>>>> your
>>>> >>>> query complexity (or load) is to blame here, not index size. What
>>>> >> version
>>>> >>>> of Solr?
>>>> >>>>
>>>> >>>> Until you know what is causing the slow queries, sharding is not
>>>> >> something
>>>> >>>> to consider I'd say. But yes, you would want to reindex to
>>>> distribute
>>>> >> the
>>>> >>>> documents.
>>>> >>>>
>>>> >>>> Erik
>>>> >>>>
>>>> >>>>
>>>> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>>>> >>>>
>>>> >>>>> Hello,
>>>> >>>>>
>>>> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it
>>>> seems
>>>> >> to
>>>> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
>>>> >>>>> not optimized
>>>> >>>>> Would it make sense to shard the index as cores in the same server
>>>> to
>>>> >>>>> expect better response time?
>>>> >>>>>
>>>> >>>>> do I have to re index all over again to distribute the documents
>>>> >> between
>>>> >>>> 2
>>>> >>>>> shards ?
>>>> >>>>>
>>>> >>>>> Regards
>>>> >>>>> Sujatha
>>>> >>>>
>>>> >>>>
>>>> >>
>>>> >>
>>>>
>>>>
>>>>
>>>
>>
Re: Muticore Sharding
Posted by Sujatha Arun <su...@gmail.com>.
Hi,
This is the parsed query string which *returns 3331234 documents*
<str name="parsedquery_toString">content:elena content:read content:a
content:rate content:r content:page content:per content:minut content:a
content:total content:m content:minut</str> re
and a Boolean "AND " query of the above query takes 410 ms and *returns
just 1 result*
<str name="parsedquery_toString">+content:elena +content:read +content:a
+content:rate +content:r +content:page +content:per +content:minut
+content:a +content:total +content:m +content:minut</str>
so the main difference being the number of results returned ,so is this
i/0 issue or is this time expected given the number of documents returned.
Also what is the difference between Qtime and timing shown as a part of
Debugquery?
Regards
Sujatha
Regards
Sujatha
On Fri, Aug 17, 2012 at 8:35 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> First, provide us with the additional info Erik requested in his last
> reply.
>
> Then, can you provide a snippet from your Solr log file that shows a
> couple of queries. Maybe there is something else going on or some
> exceptions, or something. Or at least to show us the quey times in context.
>
> Finally, has this been happening from the very beginning, or did it begin
> to grow gradually and steadily, or did it just suddenly start to happen
> even though it had been fine just a few minutes earlier? If the latter, had
> there been and system or JVM or Solr configuration changes between the time
> queries were fast and when they became slow?
>
> Thanks.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Sujatha Arun
> Sent: Friday, August 17, 2012 10:50 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Muticore Sharding
>
>
> Sorry typo ,I meant upwards of 20s at any time. What should I be looking
> at?
>
> Regards
> Sujatha
>
> On Fri, Aug 17, 2012 at 8:09 PM, Sujatha Arun <su...@gmail.com> wrote:
>
> Erik,
>>
>> What could be the issue Load / I/O ? It seems to shows upwards of 20 ms
>> at any time
>>
>> Regards
>> Sujatha
>>
>>
>> On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher <erik.hatcher@gmail.com
>> >wrote:
>>
>> Sujatha - that query debug output shows only 218ms, so it isn't
>>> representative of the issue you're reporting.
>>>
>>> Also, what's the query parse output? I imagine you're doing a boolean
>>> OR query across all those terms (include "a"), yes? Maybe you'd rather
>>> the
>>> operator be AND?
>>>
>>> Erik
>>>
>>>
>>> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>>>
>>> > No customization,its the default standard request handler.Solr Version
>>> is
>>> > 1.3
>>> >
>>> > "a" is not there in stop words
>>> >
>>> > Server Load ,i presume is not there , but not too sure ,not checked.
>>> >
>>> > RAM :
>>> >
>>> > TOTAL RAM :48GB
>>> > RAM to JVM :18 GB ,Permgen =2GB
>>> > TOTAL INDEX SIZE of all the multicore Instances =23GB
>>> >
>>> > Timing [Cut & paste ] , I have never looked at this before
>>> >
>>> > <str name="QParser">**OldLuceneQParser</str><lst name="timing"><double
>>> > name="time">218.0</double><lst name="prepare"><double
>>> > name="time">6.0</double><lst
>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>> > name="time">5.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.**
>>> MoreLikeThisComponent"><double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>> <double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>> > name="time">0.0</double></lst>**</lst><lst name="process"><double
>>> > name="time">211.0</double><lst
>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.**
>>> MoreLikeThisComponent"><double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>> <double
>>> > name="time">0.0</double></lst>**<lst
>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>> > name="time">211.0</double></**lst></lst></lst></lst>
>>> >
>>> > Regards
>>> > Sujatha
>>> >
>>> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatcher@gmail.com
>>> >wrote:
>>> >
>>> >> Just over 4M docs... no need to shard.
>>> >>
>>> >> Both your queries contain what is likely very common terms (content:a
>>> in
>>> >> the first one and content:1 in the second). Generally these are "stop
>>> >> words" and removed either during indexing or querying, but I guess not
>>> in
>>> >> your case.
>>> >>
>>> >> What's your "standard" request handler look like? Doing anything
>>> custom
>>> >> in there?
>>> >>
>>> >> What are the timings of the components in the debugQuery=true >>
>>> response?
>>> >>
>>> >> Is the server under load when you're issuing these queries? What
>>> about
>>> >> RAM?
>>> >>
>>> >> Again, what version of Solr?
>>> >>
>>> >> Erik
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>>> >>
>>> >>> Hi Erick,
>>> >>>
>>> >>> The number of documents is : 4389048
>>> >>>
>>> >>> I have given 2 queries below with timing and the number of hits
>>> >>>
>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>> >>>
>>> >>
>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>> fq=&rows=100&start=0&q=**content:if+content:elena+**
>>> content:reads+content:at+**content:a+content:rate+**
>>> content:of+content:r+content:**pages+content:per+content:**
>>> minute+content:for+content:a+**content:total+content:of+**
>>> content:m+content:minutes}
>>> >>> hits=3331109 status=0 QTime=22677
>>> >>>
>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>> >>>
>>> >>
>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>> fq=&rows=100&start=0&q=**content:chapter+content:1}
>>> >>> hits=1919445 status=0 QTime=6677
>>> >>>
>>> >>>
>>> >>> Any time I execute the time taken to execute the first query is >
>>> than
>>> >>> 22secs ,now it took 33 secs to execute.
>>> >>>
>>> >>> Regards
>>> >>> Sujatha
>>> >>>
>>> >>>
>>> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <
>>> erik.hatcher@gmail.com
>>> >>> wrote:
>>> >>>
>>> >>>> How many documents do you have? What are the queries? I'd guess
>>> your
>>> >>>> query complexity (or load) is to blame here, not index size. What
>>> >> version
>>> >>>> of Solr?
>>> >>>>
>>> >>>> Until you know what is causing the slow queries, sharding is not
>>> >> something
>>> >>>> to consider I'd say. But yes, you would want to reindex to
>>> distribute
>>> >> the
>>> >>>> documents.
>>> >>>>
>>> >>>> Erik
>>> >>>>
>>> >>>>
>>> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>>> >>>>
>>> >>>>> Hello,
>>> >>>>>
>>> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it
>>> seems
>>> >> to
>>> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
>>> >>>>> not optimized
>>> >>>>> Would it make sense to shard the index as cores in the same server
>>> to
>>> >>>>> expect better response time?
>>> >>>>>
>>> >>>>> do I have to re index all over again to distribute the documents
>>> >> between
>>> >>>> 2
>>> >>>>> shards ?
>>> >>>>>
>>> >>>>> Regards
>>> >>>>> Sujatha
>>> >>>>
>>> >>>>
>>> >>
>>> >>
>>>
>>>
>>>
>>
>
Re: Muticore Sharding
Posted by Jack Krupansky <ja...@basetechnology.com>.
First, provide us with the additional info Erik requested in his last reply.
Then, can you provide a snippet from your Solr log file that shows a couple
of queries. Maybe there is something else going on or some exceptions, or
something. Or at least to show us the quey times in context.
Finally, has this been happening from the very beginning, or did it begin to
grow gradually and steadily, or did it just suddenly start to happen even
though it had been fine just a few minutes earlier? If the latter, had there
been and system or JVM or Solr configuration changes between the time
queries were fast and when they became slow?
Thanks.
-- Jack Krupansky
-----Original Message-----
From: Sujatha Arun
Sent: Friday, August 17, 2012 10:50 AM
To: solr-user@lucene.apache.org
Subject: Re: Muticore Sharding
Sorry typo ,I meant upwards of 20s at any time. What should I be looking at?
Regards
Sujatha
On Fri, Aug 17, 2012 at 8:09 PM, Sujatha Arun <su...@gmail.com> wrote:
> Erik,
>
> What could be the issue Load / I/O ? It seems to shows upwards of 20 ms
> at any time
>
> Regards
> Sujatha
>
>
> On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher
> <er...@gmail.com>wrote:
>
>> Sujatha - that query debug output shows only 218ms, so it isn't
>> representative of the issue you're reporting.
>>
>> Also, what's the query parse output? I imagine you're doing a boolean
>> OR query across all those terms (include "a"), yes? Maybe you'd rather
>> the
>> operator be AND?
>>
>> Erik
>>
>>
>> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>>
>> > No customization,its the default standard request handler.Solr Version
>> is
>> > 1.3
>> >
>> > "a" is not there in stop words
>> >
>> > Server Load ,i presume is not there , but not too sure ,not checked.
>> >
>> > RAM :
>> >
>> > TOTAL RAM :48GB
>> > RAM to JVM :18 GB ,Permgen =2GB
>> > TOTAL INDEX SIZE of all the multicore Instances =23GB
>> >
>> > Timing [Cut & paste ] , I have never looked at this before
>> >
>> > <str name="QParser">OldLuceneQParser</str><lst name="timing"><double
>> > name="time">218.0</double><lst name="prepare"><double
>> > name="time">6.0</double><lst
>> > name="org.apache.solr.handler.component.QueryComponent"><double
>> > name="time">5.0</double></lst><lst
>> > name="org.apache.solr.handler.component.FacetComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.HighlightComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.DebugComponent"><double
>> > name="time">0.0</double></lst></lst><lst name="process"><double
>> > name="time">211.0</double><lst
>> > name="org.apache.solr.handler.component.QueryComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.FacetComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.HighlightComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.DebugComponent"><double
>> > name="time">211.0</double></lst></lst></lst></lst>
>> >
>> > Regards
>> > Sujatha
>> >
>> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatcher@gmail.com
>> >wrote:
>> >
>> >> Just over 4M docs... no need to shard.
>> >>
>> >> Both your queries contain what is likely very common terms (content:a
>> in
>> >> the first one and content:1 in the second). Generally these are "stop
>> >> words" and removed either during indexing or querying, but I guess not
>> in
>> >> your case.
>> >>
>> >> What's your "standard" request handler look like? Doing anything
>> custom
>> >> in there?
>> >>
>> >> What are the timings of the components in the debugQuery=true
>> >> response?
>> >>
>> >> Is the server under load when you're issuing these queries? What
>> about
>> >> RAM?
>> >>
>> >> Again, what version of Solr?
>> >>
>> >> Erik
>> >>
>> >>
>> >>
>> >>
>> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>> >>
>> >>> Hi Erick,
>> >>>
>> >>> The number of documents is : 4389048
>> >>>
>> >>> I have given 2 queries below with timing and the number of hits
>> >>>
>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>> >>>
>> >>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
>> >>> hits=3331109 status=0 QTime=22677
>> >>>
>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>> >>>
>> >>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
>> >>> hits=1919445 status=0 QTime=6677
>> >>>
>> >>>
>> >>> Any time I execute the time taken to execute the first query is >
>> than
>> >>> 22secs ,now it took 33 secs to execute.
>> >>>
>> >>> Regards
>> >>> Sujatha
>> >>>
>> >>>
>> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <erik.hatcher@gmail.com
>> >>> wrote:
>> >>>
>> >>>> How many documents do you have? What are the queries? I'd guess
>> your
>> >>>> query complexity (or load) is to blame here, not index size. What
>> >> version
>> >>>> of Solr?
>> >>>>
>> >>>> Until you know what is causing the slow queries, sharding is not
>> >> something
>> >>>> to consider I'd say. But yes, you would want to reindex to
>> distribute
>> >> the
>> >>>> documents.
>> >>>>
>> >>>> Erik
>> >>>>
>> >>>>
>> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>> >>>>
>> >>>>> Hello,
>> >>>>>
>> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it
>> seems
>> >> to
>> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
>> >>>>> not optimized
>> >>>>> Would it make sense to shard the index as cores in the same server
>> to
>> >>>>> expect better response time?
>> >>>>>
>> >>>>> do I have to re index all over again to distribute the documents
>> >> between
>> >>>> 2
>> >>>>> shards ?
>> >>>>>
>> >>>>> Regards
>> >>>>> Sujatha
>> >>>>
>> >>>>
>> >>
>> >>
>>
>>
>
Re: Muticore Sharding
Posted by Sujatha Arun <su...@gmail.com>.
Sorry typo ,I meant upwards of 20s at any time. What should I be looking at?
Regards
Sujatha
On Fri, Aug 17, 2012 at 8:09 PM, Sujatha Arun <su...@gmail.com> wrote:
> Erik,
>
> What could be the issue Load / I/O ? It seems to shows upwards of 20 ms
> at any time
>
> Regards
> Sujatha
>
>
> On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher <er...@gmail.com>wrote:
>
>> Sujatha - that query debug output shows only 218ms, so it isn't
>> representative of the issue you're reporting.
>>
>> Also, what's the query parse output? I imagine you're doing a boolean
>> OR query across all those terms (include "a"), yes? Maybe you'd rather the
>> operator be AND?
>>
>> Erik
>>
>>
>> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>>
>> > No customization,its the default standard request handler.Solr Version
>> is
>> > 1.3
>> >
>> > "a" is not there in stop words
>> >
>> > Server Load ,i presume is not there , but not too sure ,not checked.
>> >
>> > RAM :
>> >
>> > TOTAL RAM :48GB
>> > RAM to JVM :18 GB ,Permgen =2GB
>> > TOTAL INDEX SIZE of all the multicore Instances =23GB
>> >
>> > Timing [Cut & paste ] , I have never looked at this before
>> >
>> > <str name="QParser">OldLuceneQParser</str><lst name="timing"><double
>> > name="time">218.0</double><lst name="prepare"><double
>> > name="time">6.0</double><lst
>> > name="org.apache.solr.handler.component.QueryComponent"><double
>> > name="time">5.0</double></lst><lst
>> > name="org.apache.solr.handler.component.FacetComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.HighlightComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.DebugComponent"><double
>> > name="time">0.0</double></lst></lst><lst name="process"><double
>> > name="time">211.0</double><lst
>> > name="org.apache.solr.handler.component.QueryComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.FacetComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.HighlightComponent"><double
>> > name="time">0.0</double></lst><lst
>> > name="org.apache.solr.handler.component.DebugComponent"><double
>> > name="time">211.0</double></lst></lst></lst></lst>
>> >
>> > Regards
>> > Sujatha
>> >
>> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatcher@gmail.com
>> >wrote:
>> >
>> >> Just over 4M docs... no need to shard.
>> >>
>> >> Both your queries contain what is likely very common terms (content:a
>> in
>> >> the first one and content:1 in the second). Generally these are "stop
>> >> words" and removed either during indexing or querying, but I guess not
>> in
>> >> your case.
>> >>
>> >> What's your "standard" request handler look like? Doing anything
>> custom
>> >> in there?
>> >>
>> >> What are the timings of the components in the debugQuery=true response?
>> >>
>> >> Is the server under load when you're issuing these queries? What
>> about
>> >> RAM?
>> >>
>> >> Again, what version of Solr?
>> >>
>> >> Erik
>> >>
>> >>
>> >>
>> >>
>> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>> >>
>> >>> Hi Erick,
>> >>>
>> >>> The number of documents is : 4389048
>> >>>
>> >>> I have given 2 queries below with timing and the number of hits
>> >>>
>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>> >>>
>> >>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
>> >>> hits=3331109 status=0 QTime=22677
>> >>>
>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>> >>>
>> >>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
>> >>> hits=1919445 status=0 QTime=6677
>> >>>
>> >>>
>> >>> Any time I execute the time taken to execute the first query is >
>> than
>> >>> 22secs ,now it took 33 secs to execute.
>> >>>
>> >>> Regards
>> >>> Sujatha
>> >>>
>> >>>
>> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <erik.hatcher@gmail.com
>> >>> wrote:
>> >>>
>> >>>> How many documents do you have? What are the queries? I'd guess
>> your
>> >>>> query complexity (or load) is to blame here, not index size. What
>> >> version
>> >>>> of Solr?
>> >>>>
>> >>>> Until you know what is causing the slow queries, sharding is not
>> >> something
>> >>>> to consider I'd say. But yes, you would want to reindex to
>> distribute
>> >> the
>> >>>> documents.
>> >>>>
>> >>>> Erik
>> >>>>
>> >>>>
>> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>> >>>>
>> >>>>> Hello,
>> >>>>>
>> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it
>> seems
>> >> to
>> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
>> >>>>> not optimized
>> >>>>> Would it make sense to shard the index as cores in the same server
>> to
>> >>>>> expect better response time?
>> >>>>>
>> >>>>> do I have to re index all over again to distribute the documents
>> >> between
>> >>>> 2
>> >>>>> shards ?
>> >>>>>
>> >>>>> Regards
>> >>>>> Sujatha
>> >>>>
>> >>>>
>> >>
>> >>
>>
>>
>
Re: Muticore Sharding
Posted by Sujatha Arun <su...@gmail.com>.
Erik,
What could be the issue Load / I/O ? It seems to shows upwards of 20 ms at
any time
Regards
Sujatha
On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher <er...@gmail.com>wrote:
> Sujatha - that query debug output shows only 218ms, so it isn't
> representative of the issue you're reporting.
>
> Also, what's the query parse output? I imagine you're doing a boolean OR
> query across all those terms (include "a"), yes? Maybe you'd rather the
> operator be AND?
>
> Erik
>
>
> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>
> > No customization,its the default standard request handler.Solr Version is
> > 1.3
> >
> > "a" is not there in stop words
> >
> > Server Load ,i presume is not there , but not too sure ,not checked.
> >
> > RAM :
> >
> > TOTAL RAM :48GB
> > RAM to JVM :18 GB ,Permgen =2GB
> > TOTAL INDEX SIZE of all the multicore Instances =23GB
> >
> > Timing [Cut & paste ] , I have never looked at this before
> >
> > <str name="QParser">OldLuceneQParser</str><lst name="timing"><double
> > name="time">218.0</double><lst name="prepare"><double
> > name="time">6.0</double><lst
> > name="org.apache.solr.handler.component.QueryComponent"><double
> > name="time">5.0</double></lst><lst
> > name="org.apache.solr.handler.component.FacetComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.HighlightComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.DebugComponent"><double
> > name="time">0.0</double></lst></lst><lst name="process"><double
> > name="time">211.0</double><lst
> > name="org.apache.solr.handler.component.QueryComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.FacetComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.HighlightComponent"><double
> > name="time">0.0</double></lst><lst
> > name="org.apache.solr.handler.component.DebugComponent"><double
> > name="time">211.0</double></lst></lst></lst></lst>
> >
> > Regards
> > Sujatha
> >
> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatcher@gmail.com
> >wrote:
> >
> >> Just over 4M docs... no need to shard.
> >>
> >> Both your queries contain what is likely very common terms (content:a in
> >> the first one and content:1 in the second). Generally these are "stop
> >> words" and removed either during indexing or querying, but I guess not
> in
> >> your case.
> >>
> >> What's your "standard" request handler look like? Doing anything custom
> >> in there?
> >>
> >> What are the timings of the components in the debugQuery=true response?
> >>
> >> Is the server under load when you're issuing these queries? What about
> >> RAM?
> >>
> >> Again, what version of Solr?
> >>
> >> Erik
> >>
> >>
> >>
> >>
> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
> >>
> >>> Hi Erick,
> >>>
> >>> The number of documents is : 4389048
> >>>
> >>> I have given 2 queries below with timing and the number of hits
> >>>
> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> >>>
> >>
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
> >>> hits=3331109 status=0 QTime=22677
> >>>
> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> >>>
> >>
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
> >>> hits=1919445 status=0 QTime=6677
> >>>
> >>>
> >>> Any time I execute the time taken to execute the first query is > than
> >>> 22secs ,now it took 33 secs to execute.
> >>>
> >>> Regards
> >>> Sujatha
> >>>
> >>>
> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <erik.hatcher@gmail.com
> >>> wrote:
> >>>
> >>>> How many documents do you have? What are the queries? I'd guess
> your
> >>>> query complexity (or load) is to blame here, not index size. What
> >> version
> >>>> of Solr?
> >>>>
> >>>> Until you know what is causing the slow queries, sharding is not
> >> something
> >>>> to consider I'd say. But yes, you would want to reindex to distribute
> >> the
> >>>> documents.
> >>>>
> >>>> Erik
> >>>>
> >>>>
> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it seems
> >> to
> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
> >>>>> not optimized
> >>>>> Would it make sense to shard the index as cores in the same server to
> >>>>> expect better response time?
> >>>>>
> >>>>> do I have to re index all over again to distribute the documents
> >> between
> >>>> 2
> >>>>> shards ?
> >>>>>
> >>>>> Regards
> >>>>> Sujatha
> >>>>
> >>>>
> >>
> >>
>
>
Re: Muticore Sharding
Posted by Erik Hatcher <er...@gmail.com>.
Sujatha - that query debug output shows only 218ms, so it isn't representative of the issue you're reporting.
Also, what's the query parse output? I imagine you're doing a boolean OR query across all those terms (include "a"), yes? Maybe you'd rather the operator be AND?
Erik
On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
> No customization,its the default standard request handler.Solr Version is
> 1.3
>
> "a" is not there in stop words
>
> Server Load ,i presume is not there , but not too sure ,not checked.
>
> RAM :
>
> TOTAL RAM :48GB
> RAM to JVM :18 GB ,Permgen =2GB
> TOTAL INDEX SIZE of all the multicore Instances =23GB
>
> Timing [Cut & paste ] , I have never looked at this before
>
> <str name="QParser">OldLuceneQParser</str><lst name="timing"><double
> name="time">218.0</double><lst name="prepare"><double
> name="time">6.0</double><lst
> name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">5.0</double></lst><lst
> name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">0.0</double></lst></lst><lst name="process"><double
> name="time">211.0</double><lst
> name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">211.0</double></lst></lst></lst></lst>
>
> Regards
> Sujatha
>
> On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <er...@gmail.com>wrote:
>
>> Just over 4M docs... no need to shard.
>>
>> Both your queries contain what is likely very common terms (content:a in
>> the first one and content:1 in the second). Generally these are "stop
>> words" and removed either during indexing or querying, but I guess not in
>> your case.
>>
>> What's your "standard" request handler look like? Doing anything custom
>> in there?
>>
>> What are the timings of the components in the debugQuery=true response?
>>
>> Is the server under load when you're issuing these queries? What about
>> RAM?
>>
>> Again, what version of Solr?
>>
>> Erik
>>
>>
>>
>>
>> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>>
>>> Hi Erick,
>>>
>>> The number of documents is : 4389048
>>>
>>> I have given 2 queries below with timing and the number of hits
>>>
>>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
>>> hits=3331109 status=0 QTime=22677
>>>
>>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>
>> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
>>> hits=1919445 status=0 QTime=6677
>>>
>>>
>>> Any time I execute the time taken to execute the first query is > than
>>> 22secs ,now it took 33 secs to execute.
>>>
>>> Regards
>>> Sujatha
>>>
>>>
>>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <erik.hatcher@gmail.com
>>> wrote:
>>>
>>>> How many documents do you have? What are the queries? I'd guess your
>>>> query complexity (or load) is to blame here, not index size. What
>> version
>>>> of Solr?
>>>>
>>>> Until you know what is causing the slow queries, sharding is not
>> something
>>>> to consider I'd say. But yes, you would want to reindex to distribute
>> the
>>>> documents.
>>>>
>>>> Erik
>>>>
>>>>
>>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> One of the Index in a multicore set up has a 3GB+ index ,and it seems
>> to
>>>>> take around 5000ms+ to return simple boolean queries . The Index is
>>>>> not optimized
>>>>> Would it make sense to shard the index as cores in the same server to
>>>>> expect better response time?
>>>>>
>>>>> do I have to re index all over again to distribute the documents
>> between
>>>> 2
>>>>> shards ?
>>>>>
>>>>> Regards
>>>>> Sujatha
>>>>
>>>>
>>
>>
Re: Muticore Sharding
Posted by Sujatha Arun <su...@gmail.com>.
No customization,its the default standard request handler.Solr Version is
1.3
"a" is not there in stop words
Server Load ,i presume is not there , but not too sure ,not checked.
RAM :
TOTAL RAM :48GB
RAM to JVM :18 GB ,Permgen =2GB
TOTAL INDEX SIZE of all the multicore Instances =23GB
Timing [Cut & paste ] , I have never looked at this before
<str name="QParser">OldLuceneQParser</str><lst name="timing"><double
name="time">218.0</double><lst name="prepare"><double
name="time">6.0</double><lst
name="org.apache.solr.handler.component.QueryComponent"><double
name="time">5.0</double></lst><lst
name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">0.0</double></lst></lst><lst name="process"><double
name="time">211.0</double><lst
name="org.apache.solr.handler.component.QueryComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">211.0</double></lst></lst></lst></lst>
Regards
Sujatha
On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <er...@gmail.com>wrote:
> Just over 4M docs... no need to shard.
>
> Both your queries contain what is likely very common terms (content:a in
> the first one and content:1 in the second). Generally these are "stop
> words" and removed either during indexing or querying, but I guess not in
> your case.
>
> What's your "standard" request handler look like? Doing anything custom
> in there?
>
> What are the timings of the components in the debugQuery=true response?
>
> Is the server under load when you're issuing these queries? What about
> RAM?
>
> Again, what version of Solr?
>
> Erik
>
>
>
>
> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>
> > Hi Erick,
> >
> > The number of documents is : 4389048
> >
> > I have given 2 queries below with timing and the number of hits
> >
> > INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> >
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
> > hits=3331109 status=0 QTime=22677
> >
> > INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> >
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
> > hits=1919445 status=0 QTime=6677
> >
> >
> > Any time I execute the time taken to execute the first query is > than
> > 22secs ,now it took 33 secs to execute.
> >
> > Regards
> > Sujatha
> >
> >
> > On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <erik.hatcher@gmail.com
> >wrote:
> >
> >> How many documents do you have? What are the queries? I'd guess your
> >> query complexity (or load) is to blame here, not index size. What
> version
> >> of Solr?
> >>
> >> Until you know what is causing the slow queries, sharding is not
> something
> >> to consider I'd say. But yes, you would want to reindex to distribute
> the
> >> documents.
> >>
> >> Erik
> >>
> >>
> >> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
> >>
> >>> Hello,
> >>>
> >>> One of the Index in a multicore set up has a 3GB+ index ,and it seems
> to
> >>> take around 5000ms+ to return simple boolean queries . The Index is
> >>> not optimized
> >>> Would it make sense to shard the index as cores in the same server to
> >>> expect better response time?
> >>>
> >>> do I have to re index all over again to distribute the documents
> between
> >> 2
> >>> shards ?
> >>>
> >>> Regards
> >>> Sujatha
> >>
> >>
>
>
Re: Muticore Sharding
Posted by Erik Hatcher <er...@gmail.com>.
Just over 4M docs... no need to shard.
Both your queries contain what is likely very common terms (content:a in the first one and content:1 in the second). Generally these are "stop words" and removed either during indexing or querying, but I guess not in your case.
What's your "standard" request handler look like? Doing anything custom in there?
What are the timings of the components in the debugQuery=true response?
Is the server under load when you're issuing these queries? What about RAM?
Again, what version of Solr?
Erik
On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
> Hi Erick,
>
> The number of documents is : 4389048
>
> I have given 2 queries below with timing and the number of hits
>
> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
> hits=3331109 status=0 QTime=22677
>
> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
> params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
> hits=1919445 status=0 QTime=6677
>
>
> Any time I execute the time taken to execute the first query is > than
> 22secs ,now it took 33 secs to execute.
>
> Regards
> Sujatha
>
>
> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <er...@gmail.com>wrote:
>
>> How many documents do you have? What are the queries? I'd guess your
>> query complexity (or load) is to blame here, not index size. What version
>> of Solr?
>>
>> Until you know what is causing the slow queries, sharding is not something
>> to consider I'd say. But yes, you would want to reindex to distribute the
>> documents.
>>
>> Erik
>>
>>
>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>>
>>> Hello,
>>>
>>> One of the Index in a multicore set up has a 3GB+ index ,and it seems to
>>> take around 5000ms+ to return simple boolean queries . The Index is
>>> not optimized
>>> Would it make sense to shard the index as cores in the same server to
>>> expect better response time?
>>>
>>> do I have to re index all over again to distribute the documents between
>> 2
>>> shards ?
>>>
>>> Regards
>>> Sujatha
>>
>>
Re: Muticore Sharding
Posted by Sujatha Arun <su...@gmail.com>.
Hi Erick,
The number of documents is : 4389048
I have given 2 queries below with timing and the number of hits
INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:if+content:elena+content:reads+content:at+content:a+content:rate+content:of+content:r+content:pages+content:per+content:minute+content:for+content:a+content:total+content:of+content:m+content:minutes}
hits=3331109 status=0 QTime=22677
INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
params={version=2.1&fl=*+score&stylesheet=&qt=standard&fq=&rows=100&start=0&q=content:chapter+content:1}
hits=1919445 status=0 QTime=6677
Any time I execute the time taken to execute the first query is > than
22secs ,now it took 33 secs to execute.
Regards
Sujatha
On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <er...@gmail.com>wrote:
> How many documents do you have? What are the queries? I'd guess your
> query complexity (or load) is to blame here, not index size. What version
> of Solr?
>
> Until you know what is causing the slow queries, sharding is not something
> to consider I'd say. But yes, you would want to reindex to distribute the
> documents.
>
> Erik
>
>
> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>
> > Hello,
> >
> > One of the Index in a multicore set up has a 3GB+ index ,and it seems to
> > take around 5000ms+ to return simple boolean queries . The Index is
> > not optimized
> > Would it make sense to shard the index as cores in the same server to
> > expect better response time?
> >
> > do I have to re index all over again to distribute the documents between
> 2
> > shards ?
> >
> > Regards
> > Sujatha
>
>
Re: Muticore Sharding
Posted by Erik Hatcher <er...@gmail.com>.
How many documents do you have? What are the queries? I'd guess your query complexity (or load) is to blame here, not index size. What version of Solr?
Until you know what is causing the slow queries, sharding is not something to consider I'd say. But yes, you would want to reindex to distribute the documents.
Erik
On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
> Hello,
>
> One of the Index in a multicore set up has a 3GB+ index ,and it seems to
> take around 5000ms+ to return simple boolean queries . The Index is
> not optimized
> Would it make sense to shard the index as cores in the same server to
> expect better response time?
>
> do I have to re index all over again to distribute the documents between 2
> shards ?
>
> Regards
> Sujatha