You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Burton-West, Tom" <tb...@umich.edu> on 2011/04/08 22:53:10 UTC
ArrayIndexOutOfBoundsException with facet query
The query below results in an array out of bounds exception:
select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
Here is the exception:
Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
We are using a dev version of Solr/Lucene:
Solr Specification Version: 3.0.0.2010.11.19.16.00.54
Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19 16:00:54
Lucene Specification Version: 3.1-SNAPSHOT
Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
Just before the exception we see this entry in our tomcat logs:
Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
INFO: UnInverted multi-valued field {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
Is this a known bug? Can anyone provide a clue as to how we can determine what the problem is?
Tom Burton-West
Appended Below is the exception stack trace:
SEVERE: Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
at org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
at org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
at org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
at org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
RE: ArrayIndexOutOfBoundsException with facet query
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Thanks Mike,
With the unpatched version, the first time I run the facet query on topicStr it works fine, but the second time I get the ArrayIndexOutOfBoundsException. If I try different facets such as language, I don't see the same symptoms. Maybe the number of facet values needs to exceed some number to trigger the bug?
I rebuilt lucene-core-3.1-SNAPSHOT.jar with your patch and it fixes the problem.
Tom
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Monday, April 11, 2011 1:00 PM
To: Burton-West, Tom
Cc: solr-user@lucene.apache.org
Subject: Re: ArrayIndexOutOfBoundsException with facet query
Right, it's the total number of terms across all fields... unfortunately.
This class is used to enroll a term into the terms cache that wraps
the terms dictionary, so in theory you could also hit this issue
during normal searching when a term is looked up once, and then
looked up again (the 2nd time will pull from the cache).
I've mod'd Test2BTerms and am running it now...
Mike
http://blog.mikemccandless.com
On Mon, Apr 11, 2011 at 12:51 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Thanks Mike,
>
> At first I thought this couldn't be related to the 2.1 Billion terms issue since the only place we have tons of terms is in the OCR field and this is not the OCR field. But then I remembered that the total number of terms in all fields is what matters. We've had no problems with regular searches against the index or with other facet queries. Only with this facet. Is TermInfoAndOrd only used for faceting?
>
> I'll go ahead and build the patch and let you know.
>
>
> Tom
>
> p.s. Here is the field definition:
> <field name="topicStr" type="string" indexed="true" stored="false" multiValued="true"/>
> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, April 11, 2011 8:40 AM
> To: solr-user@lucene.apache.org
> Cc: Burton-West, Tom
> Subject: Re: ArrayIndexOutOfBoundsException with facet query
>
> Tom,
>
> I think I see where this may be -- it looks like another > 2B terms
> bug in Lucene (we are using an int instead of a long in the
> TermInfoAndOrd class inside TermInfosReader.java), only present in
> 3.1.
>
> I'm also mad that Test2BTerms fails to catch this!! I will go fix
> that test and confirm it sees this bug.
>
> Can you build from source? If so, try this patch:
>
> Index: lucene/src/java/org/apache/lucene/index/TermInfosReader.java
> ===================================================================
> --- lucene/src/java/org/apache/lucene/index/TermInfosReader.java (revision
> 1089906)
> +++ lucene/src/java/org/apache/lucene/index/TermInfosReader.java (working copy)
> @@ -46,8 +46,8 @@
>
> // Just adds term's ord to TermInfo
> private final static class TermInfoAndOrd extends TermInfo {
> - final int termOrd;
> - public TermInfoAndOrd(TermInfo ti, int termOrd) {
> + final long termOrd;
> + public TermInfoAndOrd(TermInfo ti, long termOrd) {
> super(ti);
> this.termOrd = termOrd;
> }
> @@ -245,7 +245,7 @@
> // wipe out the cache when they iterate over a large numbers
> // of terms in order
> if (tiOrd == null) {
> - termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
> enumerator.position));
> + termsCache.put(cacheKey, new TermInfoAndOrd(ti,
> enumerator.position));
> } else {
> assert sameTermInfo(ti, tiOrd, enumerator);
> assert (int) enumerator.position == tiOrd.termOrd;
> @@ -262,7 +262,7 @@
> // random-access: must seek
> final int indexPos;
> if (tiOrd != null) {
> - indexPos = tiOrd.termOrd / totalIndexInterval;
> + indexPos = (int) (tiOrd.termOrd / totalIndexInterval);
> } else {
> // Must do binary search:
> indexPos = getIndexOffset(term);
> @@ -274,7 +274,7 @@
> if (enumerator.term() != null && term.compareTo(enumerator.term()) == 0) {
> ti = enumerator.termInfo();
> if (tiOrd == null) {
> - termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
> enumerator.position));
> + termsCache.put(cacheKey, new TermInfoAndOrd(ti, enumerator.position));
> } else {
> assert sameTermInfo(ti, tiOrd, enumerator);
> assert (int) enumerator.position == tiOrd.termOrd;
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Fri, Apr 8, 2011 at 4:53 PM, Burton-West, Tom <tb...@umich.edu> wrote:
>> The query below results in an array out of bounds exception:
>> select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
>>
>> Here is the exception:
>> Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
>> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>>
>> We are using a dev version of Solr/Lucene:
>>
>> Solr Specification Version: 3.0.0.2010.11.19.16.00.54
>> Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19 16:00:54
>> Lucene Specification Version: 3.1-SNAPSHOT
>> Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
>>
>> Just before the exception we see this entry in our tomcat logs:
>>
>> Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
>> INFO: UnInverted multi-valued field {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
>> Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
>>
>> Is this a known bug? Can anyone provide a clue as to how we can determine what the problem is?
>>
>> Tom Burton-West
>>
>>
>> Appended Below is the exception stack trace:
>>
>> SEVERE: Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
>> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
>> at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
>> at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
>> at org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
>> at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
>> at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
>> at org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
>> at org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
>> at org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
>> at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
>> at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
>> at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
>> at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
>> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
>>
>>
>
Re: ArrayIndexOutOfBoundsException with facet query
Posted by Michael McCandless <lu...@mikemccandless.com>.
Right, it's the total number of terms across all fields... unfortunately.
This class is used to enroll a term into the terms cache that wraps
the terms dictionary, so in theory you could also hit this issue
during normal searching when a term is looked up once, and then
looked up again (the 2nd time will pull from the cache).
I've mod'd Test2BTerms and am running it now...
Mike
http://blog.mikemccandless.com
On Mon, Apr 11, 2011 at 12:51 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Thanks Mike,
>
> At first I thought this couldn't be related to the 2.1 Billion terms issue since the only place we have tons of terms is in the OCR field and this is not the OCR field. But then I remembered that the total number of terms in all fields is what matters. We've had no problems with regular searches against the index or with other facet queries. Only with this facet. Is TermInfoAndOrd only used for faceting?
>
> I'll go ahead and build the patch and let you know.
>
>
> Tom
>
> p.s. Here is the field definition:
> <field name="topicStr" type="string" indexed="true" stored="false" multiValued="true"/>
> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, April 11, 2011 8:40 AM
> To: solr-user@lucene.apache.org
> Cc: Burton-West, Tom
> Subject: Re: ArrayIndexOutOfBoundsException with facet query
>
> Tom,
>
> I think I see where this may be -- it looks like another > 2B terms
> bug in Lucene (we are using an int instead of a long in the
> TermInfoAndOrd class inside TermInfosReader.java), only present in
> 3.1.
>
> I'm also mad that Test2BTerms fails to catch this!! I will go fix
> that test and confirm it sees this bug.
>
> Can you build from source? If so, try this patch:
>
> Index: lucene/src/java/org/apache/lucene/index/TermInfosReader.java
> ===================================================================
> --- lucene/src/java/org/apache/lucene/index/TermInfosReader.java (revision
> 1089906)
> +++ lucene/src/java/org/apache/lucene/index/TermInfosReader.java (working copy)
> @@ -46,8 +46,8 @@
>
> // Just adds term's ord to TermInfo
> private final static class TermInfoAndOrd extends TermInfo {
> - final int termOrd;
> - public TermInfoAndOrd(TermInfo ti, int termOrd) {
> + final long termOrd;
> + public TermInfoAndOrd(TermInfo ti, long termOrd) {
> super(ti);
> this.termOrd = termOrd;
> }
> @@ -245,7 +245,7 @@
> // wipe out the cache when they iterate over a large numbers
> // of terms in order
> if (tiOrd == null) {
> - termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
> enumerator.position));
> + termsCache.put(cacheKey, new TermInfoAndOrd(ti,
> enumerator.position));
> } else {
> assert sameTermInfo(ti, tiOrd, enumerator);
> assert (int) enumerator.position == tiOrd.termOrd;
> @@ -262,7 +262,7 @@
> // random-access: must seek
> final int indexPos;
> if (tiOrd != null) {
> - indexPos = tiOrd.termOrd / totalIndexInterval;
> + indexPos = (int) (tiOrd.termOrd / totalIndexInterval);
> } else {
> // Must do binary search:
> indexPos = getIndexOffset(term);
> @@ -274,7 +274,7 @@
> if (enumerator.term() != null && term.compareTo(enumerator.term()) == 0) {
> ti = enumerator.termInfo();
> if (tiOrd == null) {
> - termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
> enumerator.position));
> + termsCache.put(cacheKey, new TermInfoAndOrd(ti, enumerator.position));
> } else {
> assert sameTermInfo(ti, tiOrd, enumerator);
> assert (int) enumerator.position == tiOrd.termOrd;
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Fri, Apr 8, 2011 at 4:53 PM, Burton-West, Tom <tb...@umich.edu> wrote:
>> The query below results in an array out of bounds exception:
>> select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
>>
>> Here is the exception:
>> Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
>> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>>
>> We are using a dev version of Solr/Lucene:
>>
>> Solr Specification Version: 3.0.0.2010.11.19.16.00.54
>> Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19 16:00:54
>> Lucene Specification Version: 3.1-SNAPSHOT
>> Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
>>
>> Just before the exception we see this entry in our tomcat logs:
>>
>> Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
>> INFO: UnInverted multi-valued field {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
>> Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
>>
>> Is this a known bug? Can anyone provide a clue as to how we can determine what the problem is?
>>
>> Tom Burton-West
>>
>>
>> Appended Below is the exception stack trace:
>>
>> SEVERE: Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
>> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
>> at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
>> at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
>> at org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
>> at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
>> at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
>> at org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
>> at org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
>> at org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
>> at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
>> at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
>> at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
>> at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
>> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
>>
>>
>
RE: ArrayIndexOutOfBoundsException with facet query
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Thanks Mike,
At first I thought this couldn't be related to the 2.1 Billion terms issue since the only place we have tons of terms is in the OCR field and this is not the OCR field. But then I remembered that the total number of terms in all fields is what matters. We've had no problems with regular searches against the index or with other facet queries. Only with this facet. Is TermInfoAndOrd only used for faceting?
I'll go ahead and build the patch and let you know.
Tom
p.s. Here is the field definition:
<field name="topicStr" type="string" indexed="true" stored="false" multiValued="true"/>
<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Monday, April 11, 2011 8:40 AM
To: solr-user@lucene.apache.org
Cc: Burton-West, Tom
Subject: Re: ArrayIndexOutOfBoundsException with facet query
Tom,
I think I see where this may be -- it looks like another > 2B terms
bug in Lucene (we are using an int instead of a long in the
TermInfoAndOrd class inside TermInfosReader.java), only present in
3.1.
I'm also mad that Test2BTerms fails to catch this!! I will go fix
that test and confirm it sees this bug.
Can you build from source? If so, try this patch:
Index: lucene/src/java/org/apache/lucene/index/TermInfosReader.java
===================================================================
--- lucene/src/java/org/apache/lucene/index/TermInfosReader.java (revision
1089906)
+++ lucene/src/java/org/apache/lucene/index/TermInfosReader.java (working copy)
@@ -46,8 +46,8 @@
// Just adds term's ord to TermInfo
private final static class TermInfoAndOrd extends TermInfo {
- final int termOrd;
- public TermInfoAndOrd(TermInfo ti, int termOrd) {
+ final long termOrd;
+ public TermInfoAndOrd(TermInfo ti, long termOrd) {
super(ti);
this.termOrd = termOrd;
}
@@ -245,7 +245,7 @@
// wipe out the cache when they iterate over a large numbers
// of terms in order
if (tiOrd == null) {
- termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
enumerator.position));
+ termsCache.put(cacheKey, new TermInfoAndOrd(ti,
enumerator.position));
} else {
assert sameTermInfo(ti, tiOrd, enumerator);
assert (int) enumerator.position == tiOrd.termOrd;
@@ -262,7 +262,7 @@
// random-access: must seek
final int indexPos;
if (tiOrd != null) {
- indexPos = tiOrd.termOrd / totalIndexInterval;
+ indexPos = (int) (tiOrd.termOrd / totalIndexInterval);
} else {
// Must do binary search:
indexPos = getIndexOffset(term);
@@ -274,7 +274,7 @@
if (enumerator.term() != null && term.compareTo(enumerator.term()) == 0) {
ti = enumerator.termInfo();
if (tiOrd == null) {
- termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
enumerator.position));
+ termsCache.put(cacheKey, new TermInfoAndOrd(ti, enumerator.position));
} else {
assert sameTermInfo(ti, tiOrd, enumerator);
assert (int) enumerator.position == tiOrd.termOrd;
Mike
http://blog.mikemccandless.com
On Fri, Apr 8, 2011 at 4:53 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> The query below results in an array out of bounds exception:
> select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
>
> Here is the exception:
> Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>
> We are using a dev version of Solr/Lucene:
>
> Solr Specification Version: 3.0.0.2010.11.19.16.00.54
> Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19 16:00:54
> Lucene Specification Version: 3.1-SNAPSHOT
> Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
>
> Just before the exception we see this entry in our tomcat logs:
>
> Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
> Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
>
> Is this a known bug? Can anyone provide a clue as to how we can determine what the problem is?
>
> Tom Burton-West
>
>
> Appended Below is the exception stack trace:
>
> SEVERE: Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
> at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
> at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
> at org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
> at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
> at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
> at org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
> at org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
> at org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
> at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
> at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
> at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
> at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
>
>
Re: ArrayIndexOutOfBoundsException with facet query
Posted by Michael McCandless <lu...@mikemccandless.com>.
Tom,
I think I see where this may be -- it looks like another > 2B terms
bug in Lucene (we are using an int instead of a long in the
TermInfoAndOrd class inside TermInfosReader.java), only present in
3.1.
I'm also mad that Test2BTerms fails to catch this!! I will go fix
that test and confirm it sees this bug.
Can you build from source? If so, try this patch:
Index: lucene/src/java/org/apache/lucene/index/TermInfosReader.java
===================================================================
--- lucene/src/java/org/apache/lucene/index/TermInfosReader.java (revision
1089906)
+++ lucene/src/java/org/apache/lucene/index/TermInfosReader.java (working copy)
@@ -46,8 +46,8 @@
// Just adds term's ord to TermInfo
private final static class TermInfoAndOrd extends TermInfo {
- final int termOrd;
- public TermInfoAndOrd(TermInfo ti, int termOrd) {
+ final long termOrd;
+ public TermInfoAndOrd(TermInfo ti, long termOrd) {
super(ti);
this.termOrd = termOrd;
}
@@ -245,7 +245,7 @@
// wipe out the cache when they iterate over a large numbers
// of terms in order
if (tiOrd == null) {
- termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
enumerator.position));
+ termsCache.put(cacheKey, new TermInfoAndOrd(ti,
enumerator.position));
} else {
assert sameTermInfo(ti, tiOrd, enumerator);
assert (int) enumerator.position == tiOrd.termOrd;
@@ -262,7 +262,7 @@
// random-access: must seek
final int indexPos;
if (tiOrd != null) {
- indexPos = tiOrd.termOrd / totalIndexInterval;
+ indexPos = (int) (tiOrd.termOrd / totalIndexInterval);
} else {
// Must do binary search:
indexPos = getIndexOffset(term);
@@ -274,7 +274,7 @@
if (enumerator.term() != null && term.compareTo(enumerator.term()) == 0) {
ti = enumerator.termInfo();
if (tiOrd == null) {
- termsCache.put(cacheKey, new TermInfoAndOrd(ti, (int)
enumerator.position));
+ termsCache.put(cacheKey, new TermInfoAndOrd(ti, enumerator.position));
} else {
assert sameTermInfo(ti, tiOrd, enumerator);
assert (int) enumerator.position == tiOrd.termOrd;
Mike
http://blog.mikemccandless.com
On Fri, Apr 8, 2011 at 4:53 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> The query below results in an array out of bounds exception:
> select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
>
> Here is the exception:
> Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>
> We are using a dev version of Solr/Lucene:
>
> Solr Specification Version: 3.0.0.2010.11.19.16.00.54
> Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19 16:00:54
> Lucene Specification Version: 3.1-SNAPSHOT
> Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
>
> Just before the exception we see this entry in our tomcat logs:
>
> Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
> Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
>
> Is this a known bug? Can anyone provide a clue as to how we can determine what the problem is?
>
> Tom Burton-West
>
>
> Appended Below is the exception stack trace:
>
> SEVERE: Exception during facet.field of topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
> at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
> at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
> at org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
> at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
> at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
> at org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
> at org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
> at org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
> at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
> at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
> at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
> at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
>
>
Re: ArrayIndexOutOfBoundsException with facet query
Posted by Edgar Espina <es...@gmail.com>.
Hi Tom,
Can you show us the field definition for topicStr?
Also, faceting is done on indexed rather than stored values. Can you check
if the field is indexed?
Thanks,
On Fri, Apr 8, 2011 at 5:53 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> The query below results in an array out of bounds exception:
> select/?q=solr&version=2.2&start=0&rows=0&facet=true&facet.field=topicStr
>
> Here is the exception:
> Exception during facet.field of
> topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at
> org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
>
> We are using a dev version of Solr/Lucene:
>
> Solr Specification Version: 3.0.0.2010.11.19.16.00.54
> Solr Implementation Version: 3.1-SNAPSHOT 1036094 - root - 2010-11-19
> 16:00:54
> Lucene Specification Version: 3.1-SNAPSHOT
> Lucene Implementation Version: 3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10
>
> Just before the exception we see this entry in our tomcat logs:
>
> Apr 8, 2011 2:01:58 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
> {field=topicStr,memSize=7675174,tindexSize=289102,time=2577,phase1=2537,nTerms=498975,bigTerms=0,termInstances=1368694,uses=0}
> Apr 8, 2011 2:01:58 PM org.apache.solr.core.SolrCore execute
>
> Is this a known bug? Can anyone provide a clue as to how we can determine
> what the problem is?
>
> Tom Burton-West
>
>
> Appended Below is the exception stack trace:
>
> SEVERE: Exception during facet.field of
> topicStr:java.lang.ArrayIndexOutOfBoundsException: -1931149
> at
> org.apache.lucene.index.TermInfosReader.seekEnum(TermInfosReader.java:201)
> at
> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:271)
> at
> org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:338)
> at
> org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:928)
> at
> org.apache.lucene.index.DirectoryReader$MultiTermEnum.<init>(DirectoryReader.java:1055)
> at
> org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:659)
> at
> org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302)
> at
> org.apache.solr.request.NumberedTermEnum.skipTo(UnInvertedField.java:1018)
> at
> org.apache.solr.request.UnInvertedField.getTermText(UnInvertedField.java:838)
> at
> org.apache.solr.request.UnInvertedField.getCounts(UnInvertedField.java:617)
> at
> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:279)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:312)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:174)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1354)
>
>
--
edgar