You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Martin Grotzke <ma...@googlemail.com> on 2010/11/29 13:14:46 UTC
ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Hi,
after an upgrade from solr-1.3 to 1.4.1 we're getting an
ArrayIndexOutOfBoundsException for a query with rows=0 and a sort
param specified:
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660)
at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84)
at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
The query is e.g.:
/select/?sort=popularity+desc&rows=0&start=0&q=foo
When this is changed to rows=1 or when the sort param is removed the
exception is gone and everything's fine.
With a clean 1.4.1 installation (unzipped, started example and posted
two documents as described in the tutorial) this issue is not
reproducable.
Does anyone have a clue what might be the reason for this and how we
could fix this on the solr side?
Of course - for a quick fix - I'll change our app so that there's no
sort param specified when rows=0.
Thanx && cheers,
Martin
--
Martin Grotzke
http://twitter.com/martin_grotzke
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Posted by Martin Grotzke <ma...@googlemail.com>.
On Tue, Nov 30, 2010 at 7:51 PM, Martin Grotzke
<ma...@googlemail.com> wrote:
> On Tue, Nov 30, 2010 at 3:09 PM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>> On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke
>> <ma...@googlemail.com> wrote:
>>> Still I'm wondering, why this issue does not occur with the plain
>>> example solr setup with 2 indexed docs. Any explanation?
>>
>> It's an old option you have in your solrconfig.xml that causes a
>> different code path to be followed in Solr:
>>
>> <!-- An optimization that attempts to use a filter to satisfy a search.
>> If the requested sort does not include score, then the filterCache
>> will be checked for a filter matching the query. If found, the filter
>> will be used as the source of document ids, and then the sort will be
>> applied to that. -->
>> <useFilterForSortedQuery>true</useFilterForSortedQuery>
>>
>> Most apps would be better off commenting that out or setting it to
>> false. It only makes sense when a high number of queries will be
>> duplicated, but with different sorts.
>
> Great, this sounds really promising, would be a very easy fix. I need
> to check this tomorrow on our test/integration server if changing this
> does the trick for us.
I just verified this fix on our test/integration system and it works - cool!
Thanx a lot for this hint,
cheers,
Martin
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Posted by Martin Grotzke <ma...@googlemail.com>.
On Tue, Nov 30, 2010 at 3:09 PM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke
> <ma...@googlemail.com> wrote:
>> Still I'm wondering, why this issue does not occur with the plain
>> example solr setup with 2 indexed docs. Any explanation?
>
> It's an old option you have in your solrconfig.xml that causes a
> different code path to be followed in Solr:
>
> <!-- An optimization that attempts to use a filter to satisfy a search.
> If the requested sort does not include score, then the filterCache
> will be checked for a filter matching the query. If found, the filter
> will be used as the source of document ids, and then the sort will be
> applied to that. -->
> <useFilterForSortedQuery>true</useFilterForSortedQuery>
>
> Most apps would be better off commenting that out or setting it to
> false. It only makes sense when a high number of queries will be
> duplicated, but with different sorts.
Great, this sounds really promising, would be a very easy fix. I need
to check this tomorrow on our test/integration server if changing this
does the trick for us.
Though, I just enabled useFilterForSortedQuery in the solr 1.4.1
example and tested rows=0 with a sort param, which doesn't fail - a
correct/valid result is returned.
Is there any condition that has to be met additionally to produce the error?
>> One question: as leaving out sorting leads to better performance, this
>> should also be true for rows=0. Or is lucene/solr already that clever
>> that it makes this optimization (ignoring sort) automatically?
>
> Solr has always special-cased this case and avoided sorting altogether
Great, good to know!
Cheers,
Martin
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke
<ma...@googlemail.com> wrote:
> Still I'm wondering, why this issue does not occur with the plain
> example solr setup with 2 indexed docs. Any explanation?
It's an old option you have in your solrconfig.xml that causes a
different code path to be followed in Solr:
<!-- An optimization that attempts to use a filter to satisfy a search.
If the requested sort does not include score, then the filterCache
will be checked for a filter matching the query. If found, the filter
will be used as the source of document ids, and then the sort will be
applied to that. -->
<useFilterForSortedQuery>true</useFilterForSortedQuery>
Most apps would be better off commenting that out or setting it to
false. It only makes sense when a high number of queries will be
duplicated, but with different sorts.
>>
>> But: why is your app doing this? Ie, if numHits (rows) is 0, the only
>> useful thing you can get is totalHits?
>
> Actually I don't know this (yet). Normally our search logic should
> optimize this and ignore a requested sorting with rows=0, but there
> seems to be a case that circumvents this - still figuring out.
>
>>
>> Still I think we should fix it in Lucene -- it's a nuisance to push
>> such corner case checks up into the apps. I'll open an issue...
>
> Just for the record, this is https://issues.apache.org/jira/browse/LUCENE-2785
>
> One question: as leaving out sorting leads to better performance, this
> should also be true for rows=0. Or is lucene/solr already that clever
> that it makes this optimization (ignoring sort) automatically?
Solr has always special-cased this case and avoided sorting altogether
(for the normal code path... but overlooked it when
useFilterForSortedQuery=true.
-Yonik
http://www.lucidimagination.com
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Posted by Martin Grotzke <ma...@googlemail.com>.
On Tue, Nov 30, 2010 at 10:29 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Hmm this is in fact a regression.
>
> TopFieldCollector expects (but does not verify) that numHits is > 0.
>
> I guess to fix this we could fix TopFieldCollector.create to return a
> NullCollector when numHits is 0.
Fixing this in lucene/solr sounds good :-)
Still I'm wondering, why this issue does not occur with the plain
example solr setup with 2 indexed docs. Any explanation?
>
> But: why is your app doing this? Ie, if numHits (rows) is 0, the only
> useful thing you can get is totalHits?
Actually I don't know this (yet). Normally our search logic should
optimize this and ignore a requested sorting with rows=0, but there
seems to be a case that circumvents this - still figuring out.
>
> Still I think we should fix it in Lucene -- it's a nuisance to push
> such corner case checks up into the apps. I'll open an issue...
Just for the record, this is https://issues.apache.org/jira/browse/LUCENE-2785
One question: as leaving out sorting leads to better performance, this
should also be true for rows=0. Or is lucene/solr already that clever
that it makes this optimization (ignoring sort) automatically? Do I
understand it correctly, that the solution with the null collector
would make this optimiztion?
We're just asking ourselves if we should go ahead and analyze and fix
this in our app or wait for a patch for solr/lucene.
What do you think? Is there s.th. like a timeframe when there's an
agreement on the correct solution and a patch available?
Thanx && cheers,
Martin
>
> Mike
>
> On Mon, Nov 29, 2010 at 7:14 AM, Martin Grotzke
> <ma...@googlemail.com> wrote:
>> Hi,
>>
>> after an upgrade from solr-1.3 to 1.4.1 we're getting an
>> ArrayIndexOutOfBoundsException for a query with rows=0 and a sort
>> param specified:
>>
>> java.lang.ArrayIndexOutOfBoundsException: 0
>> at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660)
>> at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84)
>> at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391)
>> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872)
>> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
>> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
>> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>
>> The query is e.g.:
>> /select/?sort=popularity+desc&rows=0&start=0&q=foo
>>
>> When this is changed to rows=1 or when the sort param is removed the
>> exception is gone and everything's fine.
>>
>> With a clean 1.4.1 installation (unzipped, started example and posted
>> two documents as described in the tutorial) this issue is not
>> reproducable.
>>
>> Does anyone have a clue what might be the reason for this and how we
>> could fix this on the solr side?
>> Of course - for a quick fix - I'll change our app so that there's no
>> sort param specified when rows=0.
>>
>> Thanx && cheers,
>> Martin
>>
>> --
>> Martin Grotzke
>> http://twitter.com/martin_grotzke
>>
>
--
Martin Grotzke
http://www.javakaffee.de/blog/
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm this is in fact a regression.
TopFieldCollector expects (but does not verify) that numHits is > 0.
I guess to fix this we could fix TopFieldCollector.create to return a
NullCollector when numHits is 0.
But: why is your app doing this? Ie, if numHits (rows) is 0, the only
useful thing you can get is totalHits?
Still I think we should fix it in Lucene -- it's a nuisance to push
such corner case checks up into the apps. I'll open an issue...
Mike
On Mon, Nov 29, 2010 at 7:14 AM, Martin Grotzke
<ma...@googlemail.com> wrote:
> Hi,
>
> after an upgrade from solr-1.3 to 1.4.1 we're getting an
> ArrayIndexOutOfBoundsException for a query with rows=0 and a sort
> param specified:
>
> java.lang.ArrayIndexOutOfBoundsException: 0
> at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660)
> at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84)
> at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391)
> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872)
> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>
> The query is e.g.:
> /select/?sort=popularity+desc&rows=0&start=0&q=foo
>
> When this is changed to rows=1 or when the sort param is removed the
> exception is gone and everything's fine.
>
> With a clean 1.4.1 installation (unzipped, started example and posted
> two documents as described in the tutorial) this issue is not
> reproducable.
>
> Does anyone have a clue what might be the reason for this and how we
> could fix this on the solr side?
> Of course - for a quick fix - I'll change our app so that there's no
> sort param specified when rows=0.
>
> Thanx && cheers,
> Martin
>
> --
> Martin Grotzke
> http://twitter.com/martin_grotzke
>