You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Simon Wistow <si...@thegestalt.org> on 2010/02/04 00:40:17 UTC
Slow QueryComponent.process() when queries have numbers in them
According to my logs
org.apache.solr.handler.component.QueryComponent.process()
takes a significant amount of time (>5s but I've seen up to 15s) when a
query has an odd pattern of numbers in e.g
"neodymium megagauss-oersteds (MGOe) (1 MG·Oe = 7,958·10³ T·A/m = 7,958
kJ/m³"
"myers 8e psychology chapter 9"
"JOHN PIPER 1 TIMOTEO 3:1?
"lab 2.6.2: using wireshark to view protocol data units"
"malha de aço 3x3 6mm - peso m2"
or even looks like it could be a query
"An experiment has two outcomes, A and A. If A is three time as likely to occur as , what is P(A)?"
other params were
fl:
*,score
fq:
+num_pages:[2 TO *] AND +language:1
hl:
true
hl.fl:
content title description
hl.simple.post:
</strong>
hl.simple.pre:
<strong>
hl.snippets:
2
qf:
title^1.5 content^0.8
qs:
2
qt:
dismax
rows:
10
sort:
score desc
start:
0
wt:
json
is this just something I'm going to have to put up with? Or is there
something I can do to mitigate it. If it's a bug any suggestions on how
to start patching it?
Re: Slow QueryComponent.process() when queries have numbers in them
Posted by Lance Norskog <go...@gmail.com>.
The single-digit numbers are probably in all of docs. You might want
to rip them out with a SynonymFilter. The more docs that a query
finds, the longer the query takes.
On Fri, Feb 5, 2010 at 1:23 PM, Simon Wistow <si...@thegestalt.org> wrote:
> On Wed, Feb 03, 2010 at 07:38:13PM -0800, Lance Norskog said:
>> The debugQuery parameter shows you how the query is parsed into a tree
>> of Lucene query objects.
>
> Well, that's kind of what I'm asking - I know how the query is being
> parsed:
>
> <str name="rawquerystring">myers 8e psychology chapter 9</str>
>
> <str name="querystring">myers 8e psychology chapter 9</str>
>
> <str name="parsedquery">
> +((DisjunctionMaxQuery((content:myer^0.8 | title:myer^1.5)~0.01)
> DisjunctionMaxQuery((content:"8 e"~2^0.8 | title:"8 e"~2^1.5)~0.01)
> DisjunctionMaxQuery((content:psycholog^0.8 | title:psycholog^1.5)~0.01)
> DisjunctionMaxQuery((content:chapter^0.8 | title:chapter^1.5)~0.01)
> DisjunctionMaxQuery((content:9^0.8 | title:9^1.5)~0.01))~4) ()
> </str>
>
> <str name="parsedquery_toString">
> +(((content:myer^0.8 | title:myer^1.5)~0.01 (content:"8 e"~2^0.8 |
> title:"8 e"~2^1.5)~0.01 (content:psycholog^0.8 |
> title:psycholog^1.5)~0.01 (content:chapter^0.8 | title:chapter^1.5)~0.01
> (content:9^0.8 | title:9^1.5)~0.01)~4) ()
> </str>
>
> But that's sort of besides the point - I was really asking if this is a
> known issue (i.e queries with numbers in them can be very slow) and
> whether there are any workarounds
>
>
>
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Slow QueryComponent.process() when queries have numbers in them
Posted by Simon Wistow <si...@thegestalt.org>.
On Wed, Feb 03, 2010 at 07:38:13PM -0800, Lance Norskog said:
> The debugQuery parameter shows you how the query is parsed into a tree
> of Lucene query objects.
Well, that's kind of what I'm asking - I know how the query is being
parsed:
<str name="rawquerystring">myers 8e psychology chapter 9</str>
<str name="querystring">myers 8e psychology chapter 9</str>
<str name="parsedquery">
+((DisjunctionMaxQuery((content:myer^0.8 | title:myer^1.5)~0.01)
DisjunctionMaxQuery((content:"8 e"~2^0.8 | title:"8 e"~2^1.5)~0.01)
DisjunctionMaxQuery((content:psycholog^0.8 | title:psycholog^1.5)~0.01)
DisjunctionMaxQuery((content:chapter^0.8 | title:chapter^1.5)~0.01)
DisjunctionMaxQuery((content:9^0.8 | title:9^1.5)~0.01))~4) ()
</str>
<str name="parsedquery_toString">
+(((content:myer^0.8 | title:myer^1.5)~0.01 (content:"8 e"~2^0.8 |
title:"8 e"~2^1.5)~0.01 (content:psycholog^0.8 |
title:psycholog^1.5)~0.01 (content:chapter^0.8 | title:chapter^1.5)~0.01
(content:9^0.8 | title:9^1.5)~0.01)~4) ()
</str>
But that's sort of besides the point - I was really asking if this is a
known issue (i.e queries with numbers in them can be very slow) and
whether there are any workarounds
Re: Slow QueryComponent.process() when queries have numbers in them
Posted by Lance Norskog <go...@gmail.com>.
The debugQuery parameter shows you how the query is parsed into a tree
of Lucene query objects.
http://wiki.apache.org/solr/CommonQueryParameters#debugQuery
On Wed, Feb 3, 2010 at 3:40 PM, Simon Wistow <si...@thegestalt.org> wrote:
> According to my logs
>
> org.apache.solr.handler.component.QueryComponent.process()
>
> takes a significant amount of time (>5s but I've seen up to 15s) when a
> query has an odd pattern of numbers in e.g
>
> "neodymium megagauss-oersteds (MGOe) (1 MG·Oe = 7,958·10³ T·A/m = 7,958
> kJ/m³"
>
> "myers 8e psychology chapter 9"
>
> "JOHN PIPER 1 TIMOTEO 3:1?
>
> "lab 2.6.2: using wireshark to view protocol data units"
>
> "malha de aço 3x3 6mm - peso m2"
>
> or even looks like it could be a query
>
> "An experiment has two outcomes, A and A. If A is three time as likely to occur as , what is P(A)?"
>
> other params were
>
> fl:
> *,score
> fq:
> +num_pages:[2 TO *] AND +language:1
> hl:
> true
> hl.fl:
> content title description
> hl.simple.post:
> </strong>
> hl.simple.pre:
> <strong>
> hl.snippets:
> 2
> qf:
> title^1.5 content^0.8
> qs:
> 2
> qt:
> dismax
> rows:
> 10
> sort:
> score desc
> start:
> 0
> wt:
> json
>
>
> is this just something I'm going to have to put up with? Or is there
> something I can do to mitigate it. If it's a bug any suggestions on how
> to start patching it?
>
>
>
--
Lance Norskog
goksron@gmail.com