You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sammy Yu <te...@gmail.com> on 2008/10/11 00:25:12 UTC
SOLR query times
Hi,
I'm using SOLR 1.3 on a index with approximately 8 million
documents. I would like to disable SOLR's cache so that it is easier
for me to test the scenario when there is a small likelihood of cache
hits. I've disabled caching by commenting out the filterCache,
queryResultCache, and documentCache section in solrconfig.xml as
suggested by the Wiki. It seems disabled because the admin interface
no longer shows any entries in the Cache section.
However, it appears that there is still some sort caching taking
place. The first time I make specific query it would take around 100
msec, subsequent queries would take around 15 msec. Is there some
sort of caching happening at Lucene level?
Thanks for your help,
Sammy Yu
Re: qf boosting - unique count?
Posted by Chris Hostetter <ho...@fucit.org>.
: &q=test&qf=title^1
:
: Title = "test, taking test, exam"
:
: Solr boosts 2 points for 2 occurrences of the word "test" in the title. Is
: there a way to configure solr to just give 1 point, regardless the number of
: occurrences?
the "tf" value of the term "test" is what comes into play here ... if you
want to change how significant the tf is in the scoring function you need
to provide a custom similarity and make the tf() function return a flat
value.
There is ongoing work in Jira to add a new "omitTf" option to fields to
give an easy way to eliminate tf completley (since there are internal
otimizations that can be made in that case) but you'd have to patch Solr
to try that out.
BTW: the term "boost" isn't quite right in this context. a boost is a
numbe that comes from the client to affect scores ... there is a "query
boost" in your example which is "1" because of the "^1" in the qf param
but what you are talking about is just the score.
-Hoss
qf boosting - unique count?
Posted by Hung Huynh <hh...@vinaconsulting.com>.
&q=test&qf=title^1
Title = "test, taking test, exam"
Solr boosts 2 points for 2 occurrences of the word "test" in the title. Is
there a way to configure solr to just give 1 point, regardless the number of
occurrences?
Thanks,
HH
Re: SOLR query times
Posted by Chris Hostetter <ho...@fucit.org>.
: If the latency is caused by loading and caching of Lucene's segments,
: is there a way to force Lucene's index to preload this? This seems to
configure some static queries in newSearcher and firstSearcher listeners
... it doesn't really matter what the queries are as long as they force
some access to the new index ... but if you do any sorting you should
definitely make sure that is represented in the queries as well.
-Hoss
Re: SOLR query times
Posted by Sammy Yu <te...@gmail.com>.
Hi Grant,
Thanks for your response. I'm trying to simulate our production
environment's search traffic which has very low cache hit rate.
Turning off the caches can help us better understand query times and
the load of the slave's when distribution occurs with a small list of
pre-canned queries.
If the latency is caused by loading and caching of Lucene's segments,
is there a way to force Lucene's index to preload this? This seems to
be the case in our production environment, when SOLR restarts the load
spikes and it takes a couple of hours before it settles down.
Also, are there general acceptable ways of doing scalability and
performance characterization?
Thanks,
Sammy.
On Sun, Oct 12, 2008 at 8:17 AM, Grant Ingersoll <gs...@apache.org> wrote:
> This is pretty typical. The first query is always more expensive, as Lucene
> lazily loads some pieces of the index into memory and you may see the
> FieldCache in action, depending on sorting, not to mention you are also
> seeing operating system caching take place.
>
> Is there some reason you don't want these or are you just trying to
> understand the why?
>
> -Grant
>
> On Oct 10, 2008, at 6:25 PM, Sammy Yu wrote:
>
>> Hi,
>> I'm using SOLR 1.3 on a index with approximately 8 million
>> documents. I would like to disable SOLR's cache so that it is easier
>> for me to test the scenario when there is a small likelihood of cache
>> hits. I've disabled caching by commenting out the filterCache,
>> queryResultCache, and documentCache section in solrconfig.xml as
>> suggested by the Wiki. It seems disabled because the admin interface
>> no longer shows any entries in the Cache section.
>>
>> However, it appears that there is still some sort caching taking
>> place. The first time I make specific query it would take around 100
>> msec, subsequent queries would take around 15 msec. Is there some
>> sort of caching happening at Lucene level?
>>
>> Thanks for your help,
>> Sammy Yu
>
> --------------------------
> Grant Ingersoll
> Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
> http://www.lucenebootcamp.com
>
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
>
>
Re: SOLR query times
Posted by Grant Ingersoll <gs...@apache.org>.
This is pretty typical. The first query is always more expensive, as
Lucene lazily loads some pieces of the index into memory and you may
see the FieldCache in action, depending on sorting, not to mention you
are also seeing operating system caching take place.
Is there some reason you don't want these or are you just trying to
understand the why?
-Grant
On Oct 10, 2008, at 6:25 PM, Sammy Yu wrote:
> Hi,
> I'm using SOLR 1.3 on a index with approximately 8 million
> documents. I would like to disable SOLR's cache so that it is easier
> for me to test the scenario when there is a small likelihood of cache
> hits. I've disabled caching by commenting out the filterCache,
> queryResultCache, and documentCache section in solrconfig.xml as
> suggested by the Wiki. It seems disabled because the admin interface
> no longer shows any entries in the Cache section.
>
> However, it appears that there is still some sort caching taking
> place. The first time I make specific query it would take around 100
> msec, subsequent queries would take around 15 msec. Is there some
> sort of caching happening at Lucene level?
>
> Thanks for your help,
> Sammy Yu
--------------------------
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ