Query speed decreased dramatically, not sure why though?

[C0re <bl...@hotmail.co.uk>]

We have a query which takes the form of

".../select?q=*&sort=evalDate+desc,score+desc&start=0&rows=10"

This query takes around 5 seconds to complete.

I changed the query to the following;

".../select?q=[* TO NOW]&sort=evalDate+desc,score+desc&start=0&rows=10"

The query now returns in around 600 milliseconds.

Can any one help with explaining why [* TO NOW] has had such a big effect?
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Query-speed-decreased-dramatically-not-sure-why-though-tp1307636p1307636.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Query speed decreased dramatically, not sure why though?

[Erick Erickson <er...@gmail.com>]

If you attach &debugQuery=on to your query, you'll often
get pointers as to what's actually happening under the covers...

Best
Erick

On Tue, Aug 24, 2010 at 2:26 AM, C0re <bl...@hotmail.co.uk> wrote:

>
> We have a query which takes the form of
>
> ".../select?q=*&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> This query takes around 5 seconds to complete.
>
> I changed the query to the following;
>
> ".../select?q=[* TO NOW]&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> The query now returns in around 600 milliseconds.
>
> Can any one help with explaining why [* TO NOW] has had such a big effect?
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Query-speed-decreased-dramatically-not-sure-why-though-tp1307636p1307636.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Query speed decreased dramatically, not sure why though?

[Chris Hostetter <ho...@fucit.org>]

: ".../select?q=*&sort=evalDate+desc,score+desc&start=0&rows=10"
: 
: This query takes around 5 seconds to complete.
:
: I changed the query to the following;
: 
: ".../select?q=[* TO NOW]&sort=evalDate+desc,score+desc&start=0&rows=10"
: 
: The query now returns in around 600 milliseconds.
: 
: Can any one help with explaining why [* TO NOW] has had such a big effect?

Short answer: because these are compleltey differnet queries, and match 
completley differnet sets of documents, using completley different logic.

"q=*" means "a prefix query against the default search field where the 
prefix is the empty string" and it -- so assuming your default search 
field is called "text" q=* is equivilent to q=text:*

that will match all docs that have some value indexed in the text field.

finding all of those docs means iterating over every term in that field, 
and tracking every document associated with those values -- the default 
search field typically has *lots* of indexed terms 

conversly, "q=[* TO NOW]" means "a range query of the default search 
field for all terms that are less then 'NOW'".   for a date field "NOW" 
is converted to the current time, but unless your default search field is 
a date field that's not relevant.  more then likely "NOW" is just 
interpreted as the string "NOW" (or maybe "now" if your default serach 
field is lowercased) and that query just matches all the terms that are 
alphabetically before it.

all terms alphabeticly sorted before "NOW" are less then all terms, so the 
second query is basically garunteed to be faster then the first ... and 
most likely matches a much smaller set of documents.


-Hoss

--
http://lucenerevolution.org/  ...  October 7-8, Boston
http://bit.ly/stump-hoss      ...  Stump The Chump!

Re: Query speed decreased dramatically, not sure why though?

[Constantijn Visinescu <ba...@gmail.com>]

What happens to your performance if you query for *:* instead of * ?
(probably have to url encode the colon)

On Tue, Aug 24, 2010 at 11:26 AM, C0re <bl...@hotmail.co.uk> wrote:
>
> We have a query which takes the form of
>
> ".../select?q=*&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> This query takes around 5 seconds to complete.
>
> I changed the query to the following;
>
> ".../select?q=[* TO NOW]&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> The query now returns in around 600 milliseconds.
>
> Can any one help with explaining why [* TO NOW] has had such a big effect?
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-speed-decreased-dramatically-not-sure-why-though-tp1307636p1307636.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>