Re: IOException: read past EOF when opening index built directly w/Lucene

[Michael McCandless <lu...@mikemccandless.com>]

Indeed, I can reproduce this: if I create an index on 2.3 and try to
read it on trunk w/ CheckIndex, I hit that same exception.  But: this
is [somewhat] expected, because trunk = 4.0, which can no longer read
indices created with Lucene <= 3.0.  However... instead of throwing a
weird exception, we ought to throw some sort of IndexTooOldException,
I think.  I'll open an issue.

But, when I try to check the 2.3 index with Lucene 2.9.x, 3.0.x, 3.x
(branch tips), it runs fine, which is counter to what you're seeing?
Ie Solr 1.4 uses Lucene 2.9.3, so I can't explain why you'd see this
exception on Solr 1.4...

Mike

On Thu, Jul 1, 2010 at 4:03 PM, Ken Krugler <kk...@transpac.com> wrote:
> I've got a version 2.3 index that appears to be valid - I can open it with
> Luke 1.0.1, and CheckIndex reports no problem.
>
> Just for grins, I crafted a matching schema, and tried to use the index with
> Solr 1.4 (and also Solr-trunk).
>
> In either case, I get this exception during startup:
>
> SEVERE: java.lang.RuntimeException: java.io.IOException: read past EOF
>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1067)
>        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:582)
>        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:431)
>        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:286)
>        at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:125)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86)
> ...
> Caused by: java.io.IOException: read past EOF
>        at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:154)
>        at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
>        at
> org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:40)
>        at org.apache.lucene.store.DataInput.readInt(DataInput.java:76)
>        at org.apache.lucene.index.SegmentInfo.<init>(SegmentInfo.java:171)
>        at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:230)
>        at
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:91)
>        at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:649)
>        at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:87)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:415)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:294)
>        at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1056)
>        ... 30 more
>
> and at the end of the startup logging, it says:
>
> Jul 1, 2010 12:51:25 PM org.apache.solr.core.SolrCore finalize
> SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@4513e9fd
> () has a reference count of 1
>
> Is what I'm trying to do something that's destined to fail? I would have
> expected schema/index miss-matches to show up later, not right when the
> index is being opened.
>
> I'd seen various posts about this type of error due to corrupt indexes, or
> having a buggy version of Java 1.6, or an obscure Lucene bug
> (https://issues.apache.org/jira/browse/SOLR-1778 and
> https://issues.apache.org/jira/browse/LUCENE-2270), but none of those seem
> to apply to my situation.
>
> Thanks,
>
> -- Ken
>
> PS - index dir looks like:
>
> 249K Jun 29 13:47 _0.fdt
>  12K Jun 29 13:47 _0.fdx
> 159B Jun 29 13:47 _0.fnm
> 3.6M Jun 29 13:47 _0.frq
>  23K Jun 29 13:47 _0.nrm
>  10M Jun 29 13:47 _0.prx
>  51K Jun 29 13:47 _0.tii
> 2.9M Jun 29 13:47 _0.tis
>  20B Jun 29 13:47 segments.gen
>  45B Jun 29 13:47 segments_2
>
> and Luke overview says:
>
>
>
>
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c   w e b   m i n i n g
>
>
>
>
>
>