You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Burton-West, Tom" <tb...@umich.edu> on 2010/02/08 22:02:52 UTC
TermInfosReader.get ArrayIndexOutOfBoundsException
Hello all,
After optimizing rather large indexes on 10 shards (each index holds about 500,000 documents and is about 270-300 GB in size) we started getting intermittent TermInfosReader.get() ArrayIndexOutOfBounds exceptions. The exceptions sometimes seem to occur on all 10 shards at the same time and sometimes on one shard but not the others. We also sometimes get an "Internal Server Error" but that might be either a cause or an effect of the array index out of bounds. Here is the top part of the message:
java.lang.ArrayIndexOutOfBoundsException: -14127432
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
Any suggestions for troubleshooting would be appreciated.
Trace from tomcat logs appended below.
Tom Burton-West
-----------------------------------------------------------------------------------------------------------
Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.ArrayIndexOutOfBoundsException: -14127432
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:943)
at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:144)
at org.apache.lucene.search.Similarity.idf(Similarity.java:481)
at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:186)
at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:366)
at org.apache.lucene.search.Query.weight(Query.java:95)
at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
at org.apache.lucene.search.Searcher.search(Searcher.java:171)
at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:651)
at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:545)
at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:581)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:903)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:176)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Internal Server Error
Internal Server Error
request: http://solr-sdr-search-10:8081/serve-10/select
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:423)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242)
at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:422)
at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:394)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Michael McCandless <lu...@mikemccandless.com>.
Which version of Solr/Lucene are you using?
Can you run Lucene's CheckIndex tool (java -ea:org.apache.lucene
org.apache.lucene.index.CheckIndex /path/to/index) and then post the
output?
Have you altered any of IndexWriter's defaults (via solrconfig.xml)?
Eg the termIndexInterval?
Mike
On Mon, Feb 8, 2010 at 4:02 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Hello all,
>
> After optimizing rather large indexes on 10 shards (each index holds about 500,000 documents and is about 270-300 GB in size) we started getting intermittent TermInfosReader.get() ArrayIndexOutOfBounds exceptions. The exceptions sometimes seem to occur on all 10 shards at the same time and sometimes on one shard but not the others. We also sometimes get an "Internal Server Error" but that might be either a cause or an effect of the array index out of bounds. Here is the top part of the message:
>
>
> java.lang.ArrayIndexOutOfBoundsException: -14127432
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
>
> Any suggestions for troubleshooting would be appreciated.
>
> Trace from tomcat logs appended below.
>
> Tom Burton-West
>
> -----------------------------------------------------------------------------------------------------------
>
> Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.ArrayIndexOutOfBoundsException: -14127432
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:943)
> at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
> at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:144)
> at org.apache.lucene.search.Similarity.idf(Similarity.java:481)
> at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
> at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
> at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:186)
> at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:366)
> at org.apache.lucene.search.Query.weight(Query.java:95)
> at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:651)
> at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:545)
> at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:581)
> at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:903)
> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:176)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:619)
>
> Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Internal Server Error
>
> Internal Server Error
>
> request: http://solr-sdr-search-10:8081/serve-10/select
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:423)
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242)
> at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:422)
> at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:394)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
>
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Michael McCandless <lu...@mikemccandless.com>.
I opened a Lucene issue w/ patch to try:
https://issues.apache.org/jira/browse/LUCENE-2257
Tom let me know if you're able to test this... thanks!
Mike
On Tue, Feb 9, 2010 at 2:09 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Yes, the term count reported by CheckIndex is the total number of unique terms.
>
> It indeed looks like you are exceeding the unique term count limit --
> 16777214 * 128 (= the default term index interval) is 2147483392 which
> is mighty close to max/min 32 bit int value. This makes sense,
> because CheckIndex steps through the terms in order, one by one. So
> the first term just over the limit triggered the exception.
>
> Hmm -- can you try a patched Lucene in your area? I have one small
> change to try that may increase the limit to termIndexInterval
> (default 128) * 2.1 billion.
>
> Mike
>
> On Tue, Feb 9, 2010 at 12:23 PM, Tom Burton-West <tb...@gmail.com> wrote:
>>
>> Thanks Lance and Michael,
>>
>>
>> We are running Solr 1.3.0.2009.09.03.11.14.39 (Complete version info from
>> Solr admin panel appended below)
>>
>> I tried running CheckIndex (with the -ea: switch ) on one of the shards.
>> CheckIndex also produced an ArrayIndexOutOfBoundsException on the larger
>> segment containing 500K+ documents. (Complete CheckIndex output appended
>> below)
>>
>> Is it likely that all 10 shards are corrupted? Is it possible that we have
>> simply exceeded some lucene limit?
>>
>> I'm wondering if we could have exceeded the lucene limit of unique terms of
>> 2.1 billion as mentioned towards the end of the Lucene Index File Formats
>> document. If the small 731 document index has nine million unique terms as
>> reported by check index, then even though many terms are repeated, it is
>> concievable that the 500,000 document index could have more than 2.1 billion
>> terms.
>>
>> Do you know if the number of terms reported by CheckIndex is the number of
>> unique terms?
>>
>> On the other hand, we previously optimized a 1 million document index down
>> to 1 segment and had no problems. That was with an earlier version of Solr
>> and did not include CommonGrams which could conceivably increase the number
>> of terms in the index by 2 or 3 times.
>>
>>
>> Tom
>> -----------------------------------------------------------------------------------
>>
>> Solr Specification Version: 1.3.0.2009.09.03.11.14.39
>> Solr Implementation Version: 1.4-dev 793569 - root - 2009-09-03 11:14:39
>> Lucene Specification Version: 2.9-dev
>> Lucene Implementation Version: 2.9-dev 779312 - 2009-05-27 17:19:55
>>
>>
>> [tburtonw@slurm-4 ~]$ java -Xmx4096m -Xms4096m -cp
>> /l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib/lucene-core-2.9-dev.jar:/l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib
>> -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex
>> /l/solrs/1/.snapshot/serve-2010-02-07/data/index
>>
>> Opening index @ /l/solrs/1/.snapshot/serve-2010-02-07/data/index
>>
>> Segments file=segments_zo numSegments=2 version=FORMAT_DIAGNOSTICS [Lucene
>> 2.9]
>> 1 of 2: name=_29dn docCount=554799
>> compound=false
>> hasProx=true
>> numFiles=9
>> size (MB)=267,131.261
>> diagnostics = {optimize=true, mergeFactor=2,
>> os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
>> lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
>> os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
>> has deletions [delFileName=_29dn_7.del]
>> test: open reader.........OK [184 deleted docs]
>> test: fields, norms.......OK [6 fields]
>> test: terms, freq, prox...FAILED
>> WARNING: fixIndex() would remove reference to this segment; full
>> exception:
>> java.lang.ArrayIndexOutOfBoundsException: -16777214
>> at
>> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
>> at
>> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
>> at
>> org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57)
>> at
>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:474)
>> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:715)
>>
>> 2 of 2: name=_29im docCount=731
>> compound=false
>> hasProx=true
>> numFiles=8
>> size (MB)=421.261
>> diagnostics = {optimize=true, mergeFactor=3,
>> os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
>> lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
>> os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
>> no deletions
>> test: open reader.........OK
>> test: fields, norms.......OK [6 fields]
>> test: terms, freq, prox...OK [9504552 terms; 34864047 terms/docs pairs;
>> 144869629 tokens]
>> test: stored fields.......OK [3550 total field count; avg 4.856 fields
>> per doc]
>> test: term vectors........OK [0 total vector count; avg 0 term/freq
>> vector fields per doc]
>>
>> WARNING: 1 broken segments (containing 554615 documents) detected
>> WARNING: would write new segments file, and 554615 documents would be lost,
>> if -fix were specified
>>
>>
>> [tburtonw@slurm-4 ~]$
>>
>>
>> The index is corrupted. In some places ArrayIndex and NPE are not
>> wrapped as CorruptIndexException.
>>
>> Try running your code with the Lucene assertions on. Add this to the
>> JVM arguments: -ea:org.apache.lucene...
>>
>>
>> --
>> View this message in context: http://old.nabble.com/TermInfosReader.get-ArrayIndexOutOfBoundsException-tp27506243p27518800.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Feb 9, 2010 at 2:56 PM, Tom Burton-West <tb...@gmail.com> wrote:
> I'm not sure I understand. CheckIndex reported a negative number:
> -16777214.
Right, we are overflowing the positive ints, which wraps around to the
smallest int (-2.1 billion), and then dividing by 128 = ~ -1677214.
Lucene has an array of the indexed (every 128th) terms, keyed by int,
and it has an API to seek to any of those indexed terms. The problem
is, in setting the position (a long) in the term enum, it multiplies
128 by the index term, but fails to do this as a long multiply, so it
overflows.
I think your index isn't actually corrupt... it's just a limitation in
Lucene that hopefully the patch will fix.
Mike
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Michael McCandless <lu...@mikemccandless.com>.
I attached a patch to the issue that may fix it.
Maybe start by running CheckIndex first?
Mike
On Tue, Feb 9, 2010 at 2:56 PM, Tom Burton-West <tb...@gmail.com> wrote:
>
> Thanks Michael,
>
> I'm not sure I understand. CheckIndex reported a negative number:
> -16777214.
>
> But in any case we can certainly try running CheckIndex from a patched
> lucene We could also run a patched lucene on our dev server.
>
> Tom
>
>
>
> Yes, the term count reported by CheckIndex is the total number of unique
> terms.
>
> It indeed looks like you are exceeding the unique term count limit --
> 16777214 * 128 (= the default term index interval) is 2147483392 which
> is mighty close to max/min 32 bit int value. This makes sense,
> because CheckIndex steps through the terms in order, one by one. So
> the first term just over the limit triggered the exception.
>
> Hmm -- can you try a patched Lucene in your area? I have one small
> change to try that may increase the limit to termIndexInterval
> (default 128) * 2.1 billion.
>
> Mike
>
>
> --
> View this message in context: http://old.nabble.com/TermInfosReader.get-ArrayIndexOutOfBoundsException-tp27506243p27521111.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Tom Burton-West <tb...@gmail.com>.
Thanks Michael,
I'm not sure I understand. CheckIndex reported a negative number:
-16777214.
But in any case we can certainly try running CheckIndex from a patched
lucene We could also run a patched lucene on our dev server.
Tom
Yes, the term count reported by CheckIndex is the total number of unique
terms.
It indeed looks like you are exceeding the unique term count limit --
16777214 * 128 (= the default term index interval) is 2147483392 which
is mighty close to max/min 32 bit int value. This makes sense,
because CheckIndex steps through the terms in order, one by one. So
the first term just over the limit triggered the exception.
Hmm -- can you try a patched Lucene in your area? I have one small
change to try that may increase the limit to termIndexInterval
(default 128) * 2.1 billion.
Mike
--
View this message in context: http://old.nabble.com/TermInfosReader.get-ArrayIndexOutOfBoundsException-tp27506243p27521111.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Michael McCandless <lu...@mikemccandless.com>.
Yes, the term count reported by CheckIndex is the total number of unique terms.
It indeed looks like you are exceeding the unique term count limit --
16777214 * 128 (= the default term index interval) is 2147483392 which
is mighty close to max/min 32 bit int value. This makes sense,
because CheckIndex steps through the terms in order, one by one. So
the first term just over the limit triggered the exception.
Hmm -- can you try a patched Lucene in your area? I have one small
change to try that may increase the limit to termIndexInterval
(default 128) * 2.1 billion.
Mike
On Tue, Feb 9, 2010 at 12:23 PM, Tom Burton-West <tb...@gmail.com> wrote:
>
> Thanks Lance and Michael,
>
>
> We are running Solr 1.3.0.2009.09.03.11.14.39 (Complete version info from
> Solr admin panel appended below)
>
> I tried running CheckIndex (with the -ea: switch ) on one of the shards.
> CheckIndex also produced an ArrayIndexOutOfBoundsException on the larger
> segment containing 500K+ documents. (Complete CheckIndex output appended
> below)
>
> Is it likely that all 10 shards are corrupted? Is it possible that we have
> simply exceeded some lucene limit?
>
> I'm wondering if we could have exceeded the lucene limit of unique terms of
> 2.1 billion as mentioned towards the end of the Lucene Index File Formats
> document. If the small 731 document index has nine million unique terms as
> reported by check index, then even though many terms are repeated, it is
> concievable that the 500,000 document index could have more than 2.1 billion
> terms.
>
> Do you know if the number of terms reported by CheckIndex is the number of
> unique terms?
>
> On the other hand, we previously optimized a 1 million document index down
> to 1 segment and had no problems. That was with an earlier version of Solr
> and did not include CommonGrams which could conceivably increase the number
> of terms in the index by 2 or 3 times.
>
>
> Tom
> -----------------------------------------------------------------------------------
>
> Solr Specification Version: 1.3.0.2009.09.03.11.14.39
> Solr Implementation Version: 1.4-dev 793569 - root - 2009-09-03 11:14:39
> Lucene Specification Version: 2.9-dev
> Lucene Implementation Version: 2.9-dev 779312 - 2009-05-27 17:19:55
>
>
> [tburtonw@slurm-4 ~]$ java -Xmx4096m -Xms4096m -cp
> /l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib/lucene-core-2.9-dev.jar:/l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib
> -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex
> /l/solrs/1/.snapshot/serve-2010-02-07/data/index
>
> Opening index @ /l/solrs/1/.snapshot/serve-2010-02-07/data/index
>
> Segments file=segments_zo numSegments=2 version=FORMAT_DIAGNOSTICS [Lucene
> 2.9]
> 1 of 2: name=_29dn docCount=554799
> compound=false
> hasProx=true
> numFiles=9
> size (MB)=267,131.261
> diagnostics = {optimize=true, mergeFactor=2,
> os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
> lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
> os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
> has deletions [delFileName=_29dn_7.del]
> test: open reader.........OK [184 deleted docs]
> test: fields, norms.......OK [6 fields]
> test: terms, freq, prox...FAILED
> WARNING: fixIndex() would remove reference to this segment; full
> exception:
> java.lang.ArrayIndexOutOfBoundsException: -16777214
> at
> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
> at
> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
> at
> org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57)
> at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:474)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:715)
>
> 2 of 2: name=_29im docCount=731
> compound=false
> hasProx=true
> numFiles=8
> size (MB)=421.261
> diagnostics = {optimize=true, mergeFactor=3,
> os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
> lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
> os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
> no deletions
> test: open reader.........OK
> test: fields, norms.......OK [6 fields]
> test: terms, freq, prox...OK [9504552 terms; 34864047 terms/docs pairs;
> 144869629 tokens]
> test: stored fields.......OK [3550 total field count; avg 4.856 fields
> per doc]
> test: term vectors........OK [0 total vector count; avg 0 term/freq
> vector fields per doc]
>
> WARNING: 1 broken segments (containing 554615 documents) detected
> WARNING: would write new segments file, and 554615 documents would be lost,
> if -fix were specified
>
>
> [tburtonw@slurm-4 ~]$
>
>
> The index is corrupted. In some places ArrayIndex and NPE are not
> wrapped as CorruptIndexException.
>
> Try running your code with the Lucene assertions on. Add this to the
> JVM arguments: -ea:org.apache.lucene...
>
>
> --
> View this message in context: http://old.nabble.com/TermInfosReader.get-ArrayIndexOutOfBoundsException-tp27506243p27518800.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Tom Burton-West <tb...@gmail.com>.
Thanks Lance and Michael,
We are running Solr 1.3.0.2009.09.03.11.14.39 (Complete version info from
Solr admin panel appended below)
I tried running CheckIndex (with the -ea: switch ) on one of the shards.
CheckIndex also produced an ArrayIndexOutOfBoundsException on the larger
segment containing 500K+ documents. (Complete CheckIndex output appended
below)
Is it likely that all 10 shards are corrupted? Is it possible that we have
simply exceeded some lucene limit?
I'm wondering if we could have exceeded the lucene limit of unique terms of
2.1 billion as mentioned towards the end of the Lucene Index File Formats
document. If the small 731 document index has nine million unique terms as
reported by check index, then even though many terms are repeated, it is
concievable that the 500,000 document index could have more than 2.1 billion
terms.
Do you know if the number of terms reported by CheckIndex is the number of
unique terms?
On the other hand, we previously optimized a 1 million document index down
to 1 segment and had no problems. That was with an earlier version of Solr
and did not include CommonGrams which could conceivably increase the number
of terms in the index by 2 or 3 times.
Tom
-----------------------------------------------------------------------------------
Solr Specification Version: 1.3.0.2009.09.03.11.14.39
Solr Implementation Version: 1.4-dev 793569 - root - 2009-09-03 11:14:39
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 779312 - 2009-05-27 17:19:55
[tburtonw@slurm-4 ~]$ java -Xmx4096m -Xms4096m -cp
/l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib/lucene-core-2.9-dev.jar:/l/local/apache-tomcat-serve/webapps/solr-sdr-search/serve-10/WEB-INF/lib
-ea:org.apache.lucene... org.apache.lucene.index.CheckIndex
/l/solrs/1/.snapshot/serve-2010-02-07/data/index
Opening index @ /l/solrs/1/.snapshot/serve-2010-02-07/data/index
Segments file=segments_zo numSegments=2 version=FORMAT_DIAGNOSTICS [Lucene
2.9]
1 of 2: name=_29dn docCount=554799
compound=false
hasProx=true
numFiles=9
size (MB)=267,131.261
diagnostics = {optimize=true, mergeFactor=2,
os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_29dn_7.del]
test: open reader.........OK [184 deleted docs]
test: fields, norms.......OK [6 fields]
test: terms, freq, prox...FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.lang.ArrayIndexOutOfBoundsException: -16777214
at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
at
org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:474)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:715)
2 of 2: name=_29im docCount=731
compound=false
hasProx=true
numFiles=8
size (MB)=421.261
diagnostics = {optimize=true, mergeFactor=3,
os.version=2.6.18-164.6.1.el5, os=Linux, mergeDocStores=true,
lucene.version=2.9-dev 779312 - 2009-05-27 17:19:55, source=merge,
os.arch=amd64, java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
no deletions
test: open reader.........OK
test: fields, norms.......OK [6 fields]
test: terms, freq, prox...OK [9504552 terms; 34864047 terms/docs pairs;
144869629 tokens]
test: stored fields.......OK [3550 total field count; avg 4.856 fields
per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
WARNING: 1 broken segments (containing 554615 documents) detected
WARNING: would write new segments file, and 554615 documents would be lost,
if -fix were specified
[tburtonw@slurm-4 ~]$
The index is corrupted. In some places ArrayIndex and NPE are not
wrapped as CorruptIndexException.
Try running your code with the Lucene assertions on. Add this to the
JVM arguments: -ea:org.apache.lucene...
--
View this message in context: http://old.nabble.com/TermInfosReader.get-ArrayIndexOutOfBoundsException-tp27506243p27518800.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: TermInfosReader.get ArrayIndexOutOfBoundsException
Posted by Lance Norskog <go...@gmail.com>.
The index is corrupted. In some places ArrayIndex and NPE are not
wrapped as CorruptIndexException.
Try running your code with the Lucene assertions on. Add this to the
JVM arguments: -ea:org.apache.lucene...
On Mon, Feb 8, 2010 at 1:02 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Hello all,
>
> After optimizing rather large indexes on 10 shards (each index holds about 500,000 documents and is about 270-300 GB in size) we started getting intermittent TermInfosReader.get() ArrayIndexOutOfBounds exceptions. The exceptions sometimes seem to occur on all 10 shards at the same time and sometimes on one shard but not the others. We also sometimes get an "Internal Server Error" but that might be either a cause or an effect of the array index out of bounds. Here is the top part of the message:
>
>
> java.lang.ArrayIndexOutOfBoundsException: -14127432
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
>
> Any suggestions for troubleshooting would be appreciated.
>
> Trace from tomcat logs appended below.
>
> Tom Burton-West
>
> -----------------------------------------------------------------------------------------------------------
>
> Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.ArrayIndexOutOfBoundsException: -14127432
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:246)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:943)
> at org.apache.solr.search.SolrIndexReader.docFreq(SolrIndexReader.java:308)
> at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:144)
> at org.apache.lucene.search.Similarity.idf(Similarity.java:481)
> at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
> at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
> at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:186)
> at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:366)
> at org.apache.lucene.search.Query.weight(Query.java:95)
> at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
> at org.apache.lucene.search.Searcher.search(Searcher.java:171)
> at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:651)
> at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:545)
> at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:581)
> at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:903)
> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:176)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:619)
>
> Feb 5, 2010 8:09:02 AM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Internal Server Error
>
> Internal Server Error
>
> request: http://solr-sdr-search-10:8081/serve-10/select
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:423)
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242)
> at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:422)
> at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:394)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
>
--
Lance Norskog
goksron@gmail.com