You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by johnnyisrael <jo...@gmail.com> on 2010/12/03 08:47:25 UTC
Query performance very slow even after autowarming
Hi,
I am using edgeNgramFilterfactory on SOLR 1.4.1 [<filter
class="solr.EdgeNGramFilterFactory" maxGramSize="100" minGramSize="1" />]
for my indexing.
Each document will have about 5 fields in it and only one field is indexed
with EdgeNGramFilterFactory.
I have about 1.4 million documents in my index now and my index size is
approx 296MB.
I made the field that is indexed with EdgeNGramFilterFactory as default
search field. All my query responses are very slow, some of them taking more
than 10seconds to respond.
All my query responses are very slow, Queries with single letters are still
very slow.
/select/?q=m
So I tried query warming as follows.
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst><str name="q">a</str></lst>
<lst><str name="q">b</str></lst>
<lst><str name="q">c</str></lst>
<lst><str name="q">d</str></lst>
<lst><str name="q">e</str></lst>
<lst><str name="q">f</str></lst>
<lst><str name="q">g</str></lst>
<lst><str name="q">h</str></lst>
<lst><str name="q">i</str></lst>
<lst><str name="q">j</str></lst>
<lst><str name="q">k</str></lst>
<lst><str name="q">l</str></lst>
<lst><str name="q">m</str></lst>
<lst><str name="q">n</str></lst>
<lst><str name="q">o</str></lst>
<lst><str name="q">p</str></lst>
<lst><str name="q">q</str></lst>
<lst><str name="q">r</str></lst>
<lst><str name="q">s</str></lst>
<lst><str name="q">t</str></lst>
<lst><str name="q">u</str></lst>
<lst><str name="q">v</str></lst>
<lst><str name="q">w</str></lst>
<lst><str name="q">x</str></lst>
<lst><str name="q">y</str></lst>
<lst><str name="q">z</str></lst>
</arr>
</listener>
The same above is done for firstSearcher as well.
My cache settings are as follows.
<filterCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
autowarmCount="4096"/>
<queryResultCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
autowarmCount="1024"/>
<documentCache
class="solr.LRUCache"
size="16384"
initialSize="16384"
/>
Still after query warming, few single character search is taking up to 3
seconds to respond.
Am i doing anything wrong in my cache setting or autowarm setting or am i
missing anything here?
Thanks,
Johnny
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2010384.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query performance very slow even after autowarming
Posted by johnnyisrael <jo...@gmail.com>.
No Wease,
We got the performance improvement after doing the following stuff
--> Reduced the merge factor from 10 to 3.
--> Auto-warming queries as I mentioned in my initial thread.
Thanks,
Johnny
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2545451.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query performance very slow even after autowarming
Posted by linkedLetter <bi...@live.com>.
Johnny:
Are you still looking for help?
Wease
-----
NOT always nice. When are we getting better releases?
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2542789.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query performance very slow even after autowarming
Posted by johnnyisrael <jo...@gmail.com>.
Hi Chris,
Thanks for looking into it.
Here is the sample query.
http://localhost:8080/solr/core0/select/?qt=autosuggest&q=a
I am using a request handler with a name autosuggest with the following
configuration.
<requestHandler name="autosuggest" class="solr.SearchHandler">
<lst name="defaults">
<str name="wt">json</str>
<str name="fl">name,score</str>
<str name="sort">score desc</str>
<str name="indent">true</str>
</lst>
</requestHandler>
The debug timing for the above query is as follows.
"timing":{
"time":3333.0,
"prepare":{
"time":0.0,
"org.apache.solr.handler.component.QueryComponent":{
"time":0.0},
"org.apache.solr.handler.component.FacetComponent":{
"time":0.0},
"org.apache.solr.handler.component.MoreLikeThisComponent":{
"time":0.0},
"org.apache.solr.handler.component.HighlightComponent":{
"time":0.0},
"org.apache.solr.handler.component.StatsComponent":{
"time":0.0},
"org.apache.solr.handler.component.DebugComponent":{
"time":0.0}},
"process":{
"time":3333.0,
"org.apache.solr.handler.component.QueryComponent":{
"time":3332.0},
"org.apache.solr.handler.component.FacetComponent":{
"time":0.0},
"org.apache.solr.handler.component.MoreLikeThisComponent":{
"time":0.0},
"org.apache.solr.handler.component.HighlightComponent":{
"time":0.0},
"org.apache.solr.handler.component.StatsComponent":{
"time":0.0},
"org.apache.solr.handler.component.DebugComponent":{
"time":1.0}}}}}
This is not at all consistent, sometimes it is happening and sometimes not.
Thanks,
Johnny
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2084263.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query performance very slow even after autowarming
Posted by Chris Hostetter <ho...@fucit.org>.
: I made the field that is indexed with EdgeNGramFilterFactory as default
: search field. All my query responses are very slow, some of them taking more
: than 10seconds to respond.
based on the info you've given, there's dozens of posisble reasons why you
might see slow queries -- it's hard to know because you haven't given us
examples of the fully query urls you are using, and info on how your
request handler is configured.
it could be you are asking for a really large number of rows, it
could be expensive query highlighting, it could faceting .. we have no
idea with out more details.
it would be helpful if you could show us what debug=timing returns for
your slow queries as well.
-Hoss
Re: Query performance very slow even after autowarming
Posted by johnnyisrael <jo...@gmail.com>.
Alexey,
1) I am using EdgeNGramFilter only in index analyzer alone.
2) Sometimes even multiple characters also creating problems,
What i told was an example for a performance problem, And moreover I am
trying out the autowarming and sometime it is not working ideally [100%
guaranteed performance improvement], Let say If I am warming all the first
letters, ideally all the first letter queries should perform good, I am not
seeing that guaranteed performance improvement. Even after query warming
still some queries are taking more than 3 seconds to respond.
Is this the ideal behavior? Or my autowarming configuration is wrong can you
please clarify it?
Thanks,
Johnny
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2039335.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query performance very slow even after autowarming
Posted by Alexey Serba <as...@gmail.com>.
* Do you use EdgeNGramFilter in index analyzer only? Or you also use
it on query side as well?
* What if you create additional field first_letter (string) and put
first character/characters (multivalued?) there in your external
processing code. And then during search you can filter all documents
that start with letter "a" using fq=a filter query. Would that solve
your performance problems?
* It makes sense to specify what are you trying to achieve and
probably more people can help you with that.
On Fri, Dec 3, 2010 at 10:47 AM, johnnyisrael <jo...@gmail.com> wrote:
>
> Hi,
>
> I am using edgeNgramFilterfactory on SOLR 1.4.1 [<filter
> class="solr.EdgeNGramFilterFactory" maxGramSize="100" minGramSize="1" />]
> for my indexing.
>
> Each document will have about 5 fields in it and only one field is indexed
> with EdgeNGramFilterFactory.
>
> I have about 1.4 million documents in my index now and my index size is
> approx 296MB.
>
> I made the field that is indexed with EdgeNGramFilterFactory as default
> search field. All my query responses are very slow, some of them taking more
> than 10seconds to respond.
>
> All my query responses are very slow, Queries with single letters are still
> very slow.
>
> /select/?q=m
>
> So I tried query warming as follows.
>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst><str name="q">a</str></lst>
> <lst><str name="q">b</str></lst>
> <lst><str name="q">c</str></lst>
> <lst><str name="q">d</str></lst>
> <lst><str name="q">e</str></lst>
> <lst><str name="q">f</str></lst>
> <lst><str name="q">g</str></lst>
> <lst><str name="q">h</str></lst>
> <lst><str name="q">i</str></lst>
> <lst><str name="q">j</str></lst>
> <lst><str name="q">k</str></lst>
> <lst><str name="q">l</str></lst>
> <lst><str name="q">m</str></lst>
> <lst><str name="q">n</str></lst>
> <lst><str name="q">o</str></lst>
> <lst><str name="q">p</str></lst>
> <lst><str name="q">q</str></lst>
> <lst><str name="q">r</str></lst>
> <lst><str name="q">s</str></lst>
> <lst><str name="q">t</str></lst>
> <lst><str name="q">u</str></lst>
> <lst><str name="q">v</str></lst>
> <lst><str name="q">w</str></lst>
> <lst><str name="q">x</str></lst>
> <lst><str name="q">y</str></lst>
> <lst><str name="q">z</str></lst>
> </arr>
> </listener>
>
> The same above is done for firstSearcher as well.
>
> My cache settings are as follows.
>
> <filterCache
> class="solr.LRUCache"
> size="16384"
> initialSize="4096"
> autowarmCount="4096"/>
>
> <queryResultCache
> class="solr.LRUCache"
> size="16384"
> initialSize="4096"
> autowarmCount="1024"/>
>
> <documentCache
> class="solr.LRUCache"
> size="16384"
> initialSize="16384"
> />
>
> Still after query warming, few single character search is taking up to 3
> seconds to respond.
>
> Am i doing anything wrong in my cache setting or autowarm setting or am i
> missing anything here?
>
> Thanks,
>
> Johnny
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-performance-very-slow-even-after-autowarming-tp2010384p2010384.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>