You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by muraalee <mu...@gmail.com> on 2007/07/09 23:15:04 UTC
Lucene RAM Directory doesn't work for Index Size > 8 GB
Hi,
We are facing a strange problem with RAMDirectory for indices greater than 8
GB. We have indexed around 6.5 million lucene documents and the index size
is around 8 GB. Below is the contents of Index Directory.
2236964197 _1x.fdt
51811488 _1x.fdx
293 _1x.fnm
2234929832 _1x.frq
369156856 _1x.nrm
2746607717 _1x.prx
4979893 _1x.tii
389652682 _1x.tis
42 segments_3x
20 segments.gen
We could open this using FSDirectory and it seems to be working fine. But if
we switch to RAMDirectory, we get the following exception.
- We are using Lucene 2.2.0 currently
Caused by: java.lang.ArrayIndexOutOfBoundsException: -369
at
org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:64)
at org.apache.lucene.store.IndexInput.readVLong(IndexInput.java:77)
at
org.apache.lucene.index.MultiLevelSkipListReader.loadSkipLevels(MultiLevelSkipListReader.java:194)
at
org.apache.lucene.index.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:97)
at
org.apache.lucene.index.SegmentTermDocs.skipTo(SegmentTermDocs.java:164)
at
org.apache.lucene.search.PhrasePositions.skipTo(PhrasePositions.java:52)
at
org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:88)
at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81)
at
org.apache.lucene.search.DisjunctionSumScorer.initScorerDocQueue(DisjunctionSumScorer.java:105)
at
org.apache.lucene.search.DisjunctionSumScorer.next(DisjunctionSumScorer.java:144)
at
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:327)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.Searcher.search(Searcher.java:118)
at org.apache.lucene.search.Searcher.search(Searcher.java:97)
at
org.apache.lucene.search.QueryWrapperFilter.bits(QueryWrapperFilter.java:50)
Any help to address this problem is greatly appreciated.
thanks
Murali V
--
View this message in context: http://www.nabble.com/Lucene-RAM-Directory-doesn%27t-work-for-Index-Size-%3E-8-GB-tf4052298.html#a11510351
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene RAM Directory doesn't work for Index Size > 8 GB
Posted by Doron Cohen <DO...@il.ibm.com>.
hi Murali, I found a casting issue that can cause
this problem - see patch in
http://issues.apache.org/jira/browse/LUCENE-957
is the problem solved with this patch?
Doron
muraalee <mu...@gmail.com> wrote on 09/07/2007 14:15:04:
>
> Hi,
>
> We are facing a strange problem with RAMDirectory for indices
> greater than 8
> GB. We have indexed around 6.5 million lucene documents and theindex size
> is around 8 GB. Below is the contents of Index Directory.
>
> 2236964197 _1x.fdt
> 51811488 _1x.fdx
> 293 _1x.fnm
> 2234929832 _1x.frq
> 369156856 _1x.nrm
> 2746607717 _1x.prx
> 4979893 _1x.tii
> 389652682 _1x.tis
> 42 segments_3x
> 20 segments.gen
>
> We could open this using FSDirectory and it seems to be working
> fine. But if
> we switch to RAMDirectory, we get the following exception.
>
> - We are using Lucene 2.2.0 currently
>
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -369
> at
> org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:64)
> at org.apache.lucene.store.IndexInput.
> readVLong(IndexInput.java:77)
> at
> org.apache.lucene.index.MultiLevelSkipListReader.
> loadSkipLevels(MultiLevelSkipListReader.java:194)
> at
> org.apache.lucene.index.MultiLevelSkipListReader.
> skipTo(MultiLevelSkipListReader.java:97)
> at
> org.apache.lucene.index.SegmentTermDocs.skipTo(SegmentTermDocs.
> java:164)
> at
> org.apache.lucene.search.PhrasePositions.
> skipTo(PhrasePositions.java:52)
> at
> org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:88)
> at org.apache.lucene.search.PhraseScorer.
> next(PhraseScorer.java:81)
> at
> org.apache.lucene.search.DisjunctionSumScorer.
> initScorerDocQueue(DisjunctionSumScorer.java:105)
> at
> org.apache.lucene.search.DisjunctionSumScorer.
> next(DisjunctionSumScorer.java:144)
> at
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.
> java:327)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.
> java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.
> java:118)
> at org.apache.lucene.search.Searcher.search(Searcher.java:97)
> at
> org.apache.lucene.search.QueryWrapperFilter.
> bits(QueryWrapperFilter.java:50)
>
>
> Any help to address this problem is greatly appreciated.
>
>
> thanks
> Murali V
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org