How do I write in 3.x format to an upgradeded index using Lucene 4.10

[kiwi clive <ki...@yahoo.com.INVALID>]

Hi Guys
We have several hundred thousand indexes that have been written in Lucene 3.x format. I am trying to migrate to Lucene 4.10 without the need to reindex and  the process should be transparent to our customers. Reindexing all our legacy data is not an option.

The predominant analyzer we currently use is ClassicAnalyzer as we needed to backwards compatible with the old StandardAnalyzer from pre-Lucene 3.x days. 

Our latest application uses 4.10 lucene jars and we knobble it to use Lucene 3.x format. 
When we create IndexWiriters, we are doing this:IndexWriterConfig idxCfg = new IndexWriterConfig(Version.LUCENE_3_6, new ClassicAnalyzer());

New indexes could be written in Lucene 4.10 format and we aim to apply newer analyzers to these new indexes. So all new index reading and writing should be fine. We need to query lucene 4.10 indexes with lucene 4.10 analyzers and our architecture is such that we can query lucene 3.x indexes with lucene 3.x analyzers (using Lucene Versions etc).

However, there is a difference between how Lucene 3.x and Lucene 4.10 write indexes which breaks phrase queries. 

Lucene 3.x
Lucene 3.x seems to write tokens to the index adjacent to each other (and I assume the positions are stored elsewhere). This means if we index "Thanks for coming", it get indexed as:
"thanks", "coming" after stop-word removal.
If we use a phrase query and pass it to QueryParser as:
content:"Thanks for coming"
queryParser will (using lowercase and stopword removal via ClassicAnalyzer) apply the phrase query content:"thanks coming" and find the document correctly.
Lucene 4.10My understanding is that Lucene 4.10 keeps the position increments in the index as placeholders in the data. I believe this is due to a change in how StopFilter works. So if we index our "Thanks for coming" data in Lucene 4.10, it appears to be stored as:
"thanks", <pim>, "coming"
Where <pim> is some kind of position increment marker (excuse my ignorance, I don't know the low level details).
Now if we send the query through QueryParser and after lowercasing and stopword removal it is the same as before:content:"thanks coming".
This query fails because there is a <pim> between the two terms. 

If we turn on positionIncrements in queryParser, the document is returned. A phaseQuery with a slop of 1 also finds the document, but that is a different query to that used on lucene 3.x.
So, knowing which are 3.x indexes and which are 4.x indexes means we can toggle positionincrements in QueryParser and our customers should be unaware of any changes while we start our migration. Well, that was the plan!

Lucene IndexUpgrader

If we take our old 3.x index and apply IndexUpgrader to it, we end up with a 4.10 index. There are several lucene 4.x files created in the index directory and no errors are thrown. However, it appears that the index data is still in the 3.x format, namely it remains:
"thanks", "coming"
and not:
"thanks", <pim>, "coming"
This means that although the newly upgraded index is in theory a 4.10 index, we still have to use a 3.x QueryParser syntax  (positionincrement=false) for phrase queries. Not the end of the world, but if a new document is added "Ivan the Terrible", we end up with the index containing:
"thanks", "coming""ivan", <pim>, "terrible"
So now we have one record in 3.x format and one in 4.10 format and having a hybrid index means we cannot meaningfully use phrase queries on it.
The ProblemSo we need a way to write documents in 3.x format (no <pim>), to our upgraded indexes, new indexes can use native 4.10 format. 

I have tried turning off positions when writing indexes (DOCS_AND_FREQ only) but I can't see how phrase queries can work without positional information and Lucene complains about an illegal state when querying such an index.

So, my cry for help here is:How do I write documents in 3.x format to a 3.x index upgraded to a 4.10 index so that phrase queries work ?
Any other suggestions welcome!
Thank you for taking the time to work through this rather lengthy explanation and please let me know if I have not described the issue clearly.
If it's just "Duh, you need to just do this..", I'd be a happy man :-)
Many thanks,
Clive

Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10

[kiwi clive <ki...@yahoo.com.INVALID>]

Hi Tx,
This is just to close the loop. Thank you very much for your helpful suggestions.
This works fine and solves our problem.
Much appreciated.Clive


      From: kiwi clive <ki...@yahoo.com.INVALID>
 To: Trejkaz <tr...@trypticon.org>; Lucene Users Mailing List <ja...@lucene.apache.org> 
 Sent: Wednesday, February 1, 2017 7:37 PM
 Subject: Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10
   
Hi Tx
Thank you for the detailed response, that makes a lot of sense. I feel we may have to freeze some old analyzer code as we have indexes that were written with Lucene 2.3 analyzers and that is no longer supported. I'll need to do some experimentation to see how we go. Further reading has shown that StopFilter changed behaviour as of Lucene version 2.9.
Keeping old analyzer code forever is not great but as long as we can co-exist with newer indexes, we are in good shape as legacy indexes can be reindexed in slow time as necessary.
I'll do some digging !
May thanksClive
      From: Trejkaz <tr...@trypticon.org>
 To: Lucene Users Mailing List <ja...@lucene.apache.org>; kiwi clive <ki...@yahoo.com> 
 Sent: Wednesday, February 1, 2017 2:53 PM
 Subject: Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10
  
> If we take our old 3.x index and apply IndexUpgrader to it, we end up with a 4.10 index.
> There are several lucene 4.x files created in the index directory and no errors are thrown.
> However, it appears that the index data is still in the 3.x format, namely it remains:
> "thanks", "coming"
> and not:
> "thanks", <pim>, "coming"

Well, this is a different thing really. The index is in the 4.x
format, but the analysis which was performed remains the 3.x analysis,
because nothing was done to change the postings.

So this whole thing is really just a "make sure to use the same
analyser to query which you used to index" problem. So if you indexed
using a Lucene 3 analyser, then you should be using the same v3
analyser when you query against the index in Lucene 4.

So the usual rules apply:
  * Beware of Version.LATEST/LUCENE_CURRENT. Always use the exact
version, and keep using it.
  * If Lucene remove support for some Version you were using, don't
update the Version you're using. Instead, take a copy of the
Tokenizer/TokenFilter you were using from the older version and port
it to work on the new version. Maintain these frozen off analysis
components forever.

But that said, we didn't experience any problems like this from 3 to
4, but rather obscure problems where backwards compatibility was not
maintained in Lucene itself, e.g. places where despite passing in a
Version object, the older behaviour was not maintained. IIRC, the term
length limits being changed was one of these. And in these situations,
for the most part, freezing off a copy of the old behaviour works
fine.

That said, we don't use the "classic" query parser, but rather the
flexible one. And maybe if you're using the classic one, it might have
some misbehaviour around this which we didn't strike by using the
flexible one.

> So we need a way to write documents in 3.x format (no <pim>), to our upgraded indexes,
> new indexes can use native 4.10 format.

It sounds like you just need to use the same analyser you were
previously using, possibly forever...

TX


  

   

Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10

[kiwi clive <ki...@yahoo.com.INVALID>]

Hi Tx
Thank you for the detailed response, that makes a lot of sense. I feel we may have to freeze some old analyzer code as we have indexes that were written with Lucene 2.3 analyzers and that is no longer supported. I'll need to do some experimentation to see how we go. Further reading has shown that StopFilter changed behaviour as of Lucene version 2.9.
Keeping old analyzer code forever is not great but as long as we can co-exist with newer indexes, we are in good shape as legacy indexes can be reindexed in slow time as necessary.
I'll do some digging !
May thanksClive
      From: Trejkaz <tr...@trypticon.org>
 To: Lucene Users Mailing List <ja...@lucene.apache.org>; kiwi clive <ki...@yahoo.com> 
 Sent: Wednesday, February 1, 2017 2:53 PM
 Subject: Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10
   
> If we take our old 3.x index and apply IndexUpgrader to it, we end up with a 4.10 index.
> There are several lucene 4.x files created in the index directory and no errors are thrown.
> However, it appears that the index data is still in the 3.x format, namely it remains:
> "thanks", "coming"
> and not:
> "thanks", <pim>, "coming"

Well, this is a different thing really. The index is in the 4.x
format, but the analysis which was performed remains the 3.x analysis,
because nothing was done to change the postings.

So this whole thing is really just a "make sure to use the same
analyser to query which you used to index" problem. So if you indexed
using a Lucene 3 analyser, then you should be using the same v3
analyser when you query against the index in Lucene 4.

So the usual rules apply:
  * Beware of Version.LATEST/LUCENE_CURRENT. Always use the exact
version, and keep using it.
  * If Lucene remove support for some Version you were using, don't
update the Version you're using. Instead, take a copy of the
Tokenizer/TokenFilter you were using from the older version and port
it to work on the new version. Maintain these frozen off analysis
components forever.

But that said, we didn't experience any problems like this from 3 to
4, but rather obscure problems where backwards compatibility was not
maintained in Lucene itself, e.g. places where despite passing in a
Version object, the older behaviour was not maintained. IIRC, the term
length limits being changed was one of these. And in these situations,
for the most part, freezing off a copy of the old behaviour works
fine.

That said, we don't use the "classic" query parser, but rather the
flexible one. And maybe if you're using the classic one, it might have
some misbehaviour around this which we didn't strike by using the
flexible one.

> So we need a way to write documents in 3.x format (no <pim>), to our upgraded indexes,
> new indexes can use native 4.10 format.

It sounds like you just need to use the same analyser you were
previously using, possibly forever...

TX


   

Re: How do I write in 3.x format to an upgradeded index using Lucene 4.10

[Trejkaz <tr...@trypticon.org>]

> If we take our old 3.x index and apply IndexUpgrader to it, we end up with a 4.10 index.
> There are several lucene 4.x files created in the index directory and no errors are thrown.
> However, it appears that the index data is still in the 3.x format, namely it remains:
> "thanks", "coming"
> and not:
> "thanks", <pim>, "coming"

Well, this is a different thing really. The index is in the 4.x
format, but the analysis which was performed remains the 3.x analysis,
because nothing was done to change the postings.

So this whole thing is really just a "make sure to use the same
analyser to query which you used to index" problem. So if you indexed
using a Lucene 3 analyser, then you should be using the same v3
analyser when you query against the index in Lucene 4.

So the usual rules apply:
  * Beware of Version.LATEST/LUCENE_CURRENT. Always use the exact
version, and keep using it.
  * If Lucene remove support for some Version you were using, don't
update the Version you're using. Instead, take a copy of the
Tokenizer/TokenFilter you were using from the older version and port
it to work on the new version. Maintain these frozen off analysis
components forever.

But that said, we didn't experience any problems like this from 3 to
4, but rather obscure problems where backwards compatibility was not
maintained in Lucene itself, e.g. places where despite passing in a
Version object, the older behaviour was not maintained. IIRC, the term
length limits being changed was one of these. And in these situations,
for the most part, freezing off a copy of the old behaviour works
fine.

That said, we don't use the "classic" query parser, but rather the
flexible one. And maybe if you're using the classic one, it might have
some misbehaviour around this which we didn't strike by using the
flexible one.

> So we need a way to write documents in 3.x format (no <pim>), to our upgraded indexes,
> new indexes can use native 4.10 format.

It sounds like you just need to use the same analyser you were
previously using, possibly forever...

TX

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org