Solr 4.4. creating an index that 4.3 can't read (but in LUCENE_43 mode)

[Daniel Collins <da...@gmail.com>]

I had been running a Solr 4.3.0 index, which I upgraded to 4.4.0 (but
hadn't changed LuceneVersion, so it was still using the LUCENE_43 codec).

I then had to back-out and return to a 4.3 system, and got an error when it
tried to read the index.

Now, it was only a dev system, so not a problem, and normally I would use
restore a backup anyway, but shouldn't this work?  If I haven't changed the
codec, then Solr 4.4 should be using the same code as 4.3, so the data
should be compatible, no?

I noticed its in DocValues, but I thought they were supposed to be
compatible using the default format which we do?

Caused by: org.apache.lucene.index.IndexFormatTooNewException: Format
version is not supported (resource:
NIOFSIndexInput(path="/bb/news/search/solr/main/data/index/_3bs_Lucene42_0.dvm")):
1 (needs to be between 0 and 0)
        at
org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:148)
        at
org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:130)
        at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.<init>(Lucene42DocValuesProducer.java:84)
        at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesFormat.fieldsProducer(Lucene42DocValuesFormat.java:133)
        at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:213)
        at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:282)
        at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:134)
        at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:56)
        at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
        at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
        at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
        at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
        at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
        at
org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
        ... 18 more


Cheers, Daniel

Re: Solr 4.4. creating an index that 4.3 can't read (but in LUCENE_43 mode)

[Daniel Collins <da...@gmail.com>]

It does seem that the Lucene42DocValuesProducer has changed its internal
version and that is what its complaining about.

Cheers Shawn, Ok my misunderstanding on the codec stuff then, as I said
probably not a common occurrence but good to know.


On 7 August 2013 17:32, Shawn Heisey <so...@elyograg.org> wrote:

> On 8/7/2013 3:33 AM, Daniel Collins wrote:
>
>> I had been running a Solr 4.3.0 index, which I upgraded to 4.4.0 (but
>> hadn't changed LuceneVersion, so it was still using the LUCENE_43 codec).
>>
>> I then had to back-out and return to a 4.3 system, and got an error when
>> it
>> tried to read the index.
>>
>> Now, it was only a dev system, so not a problem, and normally I would use
>> restore a backup anyway, but shouldn't this work?  If I haven't changed
>> the
>> codec, then Solr 4.4 should be using the same code as 4.3, so the data
>> should be compatible, no?
>>
>
> Using an index from a newer version is never guaranteed, and usually will
> NOT work.  The luceneMatchVersion setting doesn't typically affect index
> format, it usually affects how analysis and query parser components work -
> so you can tell Solr to use buggy behavior from an earlier release.
>
> Unless you actually change aspects of the codec (postings format,
> docvalues format, etc), Solr uses the Lucene codec defaults, which can (and
> usually does) change from release to release.
>
> Looking through the Lucene 4.4 CHANGES.txt file (not the Solr file),
> LUCENE-4936 looks like a change to the DocValues format.  I can't tell from
> the description whether LUCENE-5035 is a format change or a change in how
> Lucene handles sorting in memory.  The evidence I can find suggests that
> the format is still called Lucene42DocValuesFormat, but apparently it
> doesn't work the same.
>
> Thanks,
> Shawn
>
>

Re: Solr 4.4. creating an index that 4.3 can't read (but in LUCENE_43 mode)

[Shawn Heisey <so...@elyograg.org>]

On 8/7/2013 3:33 AM, Daniel Collins wrote:
> I had been running a Solr 4.3.0 index, which I upgraded to 4.4.0 (but
> hadn't changed LuceneVersion, so it was still using the LUCENE_43 codec).
>
> I then had to back-out and return to a 4.3 system, and got an error when it
> tried to read the index.
>
> Now, it was only a dev system, so not a problem, and normally I would use
> restore a backup anyway, but shouldn't this work?  If I haven't changed the
> codec, then Solr 4.4 should be using the same code as 4.3, so the data
> should be compatible, no?

Using an index from a newer version is never guaranteed, and usually 
will NOT work.  The luceneMatchVersion setting doesn't typically affect 
index format, it usually affects how analysis and query parser 
components work - so you can tell Solr to use buggy behavior from an 
earlier release.

Unless you actually change aspects of the codec (postings format, 
docvalues format, etc), Solr uses the Lucene codec defaults, which can 
(and usually does) change from release to release.

Looking through the Lucene 4.4 CHANGES.txt file (not the Solr file), 
LUCENE-4936 looks like a change to the DocValues format.  I can't tell 
from the description whether LUCENE-5035 is a format change or a change 
in how Lucene handles sorting in memory.  The evidence I can find 
suggests that the format is still called Lucene42DocValuesFormat, but 
apparently it doesn't work the same.

Thanks,
Shawn