You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2017/11/09 22:07:31 UTC
Re: Lucene 6.1.0 index upgrade
Maybe try the addIndexes method that takes CodecReader[] instead? It will
be somewhat slower, since it fully reads and writes the index, but it
should work across major versions?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 3:59 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@cengage.com> wrote:
> Greetings.
>
>
> We are currently using Lucene 6.1.0, our typical index size is ~ 200GB and
> we are planning to move to Lucene 7.0.0.
>
>
> Our daily indexing workflow currently uses IndexWriter.addIndexes() to add
> an index(this will be on Lucene 7.0.0) to a copy of the of the 200GB index.
> Further it calls IndexWriter.forceMerge(1) to create a single segment. The
> addIndexes() now doesn't support different versions of index.
>
> We are facing an issue in upgrading to Lucene 7.0.0 due to this
> limitation. We couldn't afford to do full reindex as there about 50
> different indexes ~ 200GB each.
>
>
> We tried using the LuceneUpgrade tool to upgrade our index to latest
> version, however the IndexCreatedVersionMajor is still LUCENE_6_1_0.major,
> This is preventing us calling IndexWriter.addIndexes() post upgrade.
>
>
> I am wondering what are our options to upgrade without doing a full
> reindex and to continue to use IndexWriter.addIndexes() post upgrade.
>
> The code snippet below shows the different api calls we make.
>
>
>
> @Test
> public void addIndexWith7SegmentToDirectoryOnUpgradedSegment() throws
> IOException {
> IndexUpgrader upgrader = new IndexUpgrader(lucene610Dir); // lucene 6.1.0
> directory 200 GB
> upgrader.upgrade();
>
> IndexWriter w = new IndexWriter(lucene610Dir, newIndexWriterConfig());
> exception.expect(IllegalArgumentException.class);
> exception.expectMessage(
> CoreMatchers.containsString("Cannot use addIndexes(Directory) with
> indexes that have been created by a different Lucene version"));
> w.addIndexes(lucene700Dir); // lucene 7.0.0 daily index
> w.forceMerge(1);
> w.close();
> }
>
> Let me know if you need more details.
>
>
> Many thanks
>
> Kannan.
>
>
Re: Lucene 6.1.0 index upgrade
Posted by Trejkaz <tr...@trypticon.org>.
On Sat, Nov 11, 2017 at 7:09 AM, Krishnamurthy, Kannan
<Ka...@cengage.com> wrote:
> Never mind my previous question, understood what you meant about the impact to norms after
> looking at the uses of CreatedMajorVersion in various Similarity classes. It almost looks
> re-indexing is the only option here. This will be quite a big change to embrace for the users
> of addIndexes().
Reindexing isn't even an option for some of us, because we don't have
the original content which was indexed.
So personally I'm hoping that IndexUpgrader continues to work. :)
TX
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 6.1.0 index upgrade
Posted by "Krishnamurthy, Kannan" <Ka...@cengage.com>.
Never mind my previous question, understood what you meant about the impact to norms after looking at the uses of CreatedMajorVersion in various Similarity classes. It almost looks re-indexing is the only option here. This will be quite a big change to embrace for the users of addIndexes(). (
Many thanks Mike.
On 11/10/17, 1:06 PM, "Krishnamurthy, Kannan" <Ka...@cengage.com> wrote:
Will the norms still be a concern if the 6.1.0 index was upgraded using the IndexUpgrader? If not would we upgrade first then fake the index creation version to be 7.0.0 ?
Thanks again.
Kannan
From: Michael McCandless <lu...@mikemccandless.com>
Date: Friday, November 10, 2017 at 6:41 AM
To: Lucene Users <ja...@lucene.apache.org>, "Krishnamurthy, Kannan" <Ka...@cengage.com>
Subject: Re: Lucene 6.1.0 index upgrade
OK I'm sorry but I think you'll have to re-index; e.g. how norms are computed and stored in the index has changed, which means even if you could somehow copy or merge the index over to the new version, the norms would be incorrect.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 6:46 PM, Krishnamurthy, Kannan <Ka...@cengage.com>> wrote:
Hi Mike,
It doesn’t work either, it throws “Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7" message. Let me try it after an upgrade as well. Same code below,
@Test
public void mergeUsingCodecReader() throws IOException {
DirectoryReader ir = DirectoryReader.open(lucene610Dir);
SegmentReader air = (SegmentReader)ir.leaves().get(0).reader();
IndexWriterConfig config = newIndexWriterConfig();
IndexWriter w = new IndexWriter(newEmptyDirectory("target"), config);
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7");
w.addIndexes(air);
w.forceMerge(1);
w.close();
}
Thanks,
Kannan
On 11/9/17, 5:07 PM, "Michael McCandless" <lu...@mikemccandless.com>> wrote:
Maybe try the addIndexes method that takes CodecReader[] instead? It will
be somewhat slower, since it fully reads and writes the index, but it
should work across major versions?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 3:59 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@cengage.com<ma...@cengage.com>> wrote:
> Greetings.
>
>
> We are currently using Lucene 6.1.0, our typical index size is ~ 200GB and
> we are planning to move to Lucene 7.0.0.
>
>
> Our daily indexing workflow currently uses IndexWriter.addIndexes() to add
> an index(this will be on Lucene 7.0.0) to a copy of the of the 200GB index.
> Further it calls IndexWriter.forceMerge(1) to create a single segment. The
> addIndexes() now doesn't support different versions of index.
>
> We are facing an issue in upgrading to Lucene 7.0.0 due to this
> limitation. We couldn't afford to do full reindex as there about 50
> different indexes ~ 200GB each.
>
>
> We tried using the LuceneUpgrade tool to upgrade our index to latest
> version, however the IndexCreatedVersionMajor is still LUCENE_6_1_0.major,
> This is preventing us calling IndexWriter.addIndexes() post upgrade.
>
>
> I am wondering what are our options to upgrade without doing a full
> reindex and to continue to use IndexWriter.addIndexes() post upgrade.
>
> The code snippet below shows the different api calls we make.
>
>
>
> @Test
> public void addIndexWith7SegmentToDirectoryOnUpgradedSegment() throws
> IOException {
> IndexUpgrader upgrader = new IndexUpgrader(lucene610Dir); // lucene 6.1.0
> directory 200 GB
> upgrader.upgrade();
>
> IndexWriter w = new IndexWriter(lucene610Dir, newIndexWriterConfig());
> exception.expect(IllegalArgumentException.class);
> exception.expectMessage(
> CoreMatchers.containsString("Cannot use addIndexes(Directory) with
> indexes that have been created by a different Lucene version"));
> w.addIndexes(lucene700Dir); // lucene 7.0.0 daily index
> w.forceMerge(1);
> w.close();
> }
>
> Let me know if you need more details.
>
>
> Many thanks
>
> Kannan.
>
>
Re: Lucene 6.1.0 index upgrade
Posted by "Krishnamurthy, Kannan" <Ka...@cengage.com>.
Will the norms still be a concern if the 6.1.0 index was upgraded using the IndexUpgrader? If not would we upgrade first then fake the index creation version to be 7.0.0 ?
Thanks again.
Kannan
From: Michael McCandless <lu...@mikemccandless.com>
Date: Friday, November 10, 2017 at 6:41 AM
To: Lucene Users <ja...@lucene.apache.org>, "Krishnamurthy, Kannan" <Ka...@cengage.com>
Subject: Re: Lucene 6.1.0 index upgrade
OK I'm sorry but I think you'll have to re-index; e.g. how norms are computed and stored in the index has changed, which means even if you could somehow copy or merge the index over to the new version, the norms would be incorrect.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 6:46 PM, Krishnamurthy, Kannan <Ka...@cengage.com>> wrote:
Hi Mike,
It doesn’t work either, it throws “Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7" message. Let me try it after an upgrade as well. Same code below,
@Test
public void mergeUsingCodecReader() throws IOException {
DirectoryReader ir = DirectoryReader.open(lucene610Dir);
SegmentReader air = (SegmentReader)ir.leaves().get(0).reader();
IndexWriterConfig config = newIndexWriterConfig();
IndexWriter w = new IndexWriter(newEmptyDirectory("target"), config);
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7");
w.addIndexes(air);
w.forceMerge(1);
w.close();
}
Thanks,
Kannan
On 11/9/17, 5:07 PM, "Michael McCandless" <lu...@mikemccandless.com>> wrote:
Maybe try the addIndexes method that takes CodecReader[] instead? It will
be somewhat slower, since it fully reads and writes the index, but it
should work across major versions?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 3:59 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@cengage.com<ma...@cengage.com>> wrote:
> Greetings.
>
>
> We are currently using Lucene 6.1.0, our typical index size is ~ 200GB and
> we are planning to move to Lucene 7.0.0.
>
>
> Our daily indexing workflow currently uses IndexWriter.addIndexes() to add
> an index(this will be on Lucene 7.0.0) to a copy of the of the 200GB index.
> Further it calls IndexWriter.forceMerge(1) to create a single segment. The
> addIndexes() now doesn't support different versions of index.
>
> We are facing an issue in upgrading to Lucene 7.0.0 due to this
> limitation. We couldn't afford to do full reindex as there about 50
> different indexes ~ 200GB each.
>
>
> We tried using the LuceneUpgrade tool to upgrade our index to latest
> version, however the IndexCreatedVersionMajor is still LUCENE_6_1_0.major,
> This is preventing us calling IndexWriter.addIndexes() post upgrade.
>
>
> I am wondering what are our options to upgrade without doing a full
> reindex and to continue to use IndexWriter.addIndexes() post upgrade.
>
> The code snippet below shows the different api calls we make.
>
>
>
> @Test
> public void addIndexWith7SegmentToDirectoryOnUpgradedSegment() throws
> IOException {
> IndexUpgrader upgrader = new IndexUpgrader(lucene610Dir); // lucene 6.1.0
> directory 200 GB
> upgrader.upgrade();
>
> IndexWriter w = new IndexWriter(lucene610Dir, newIndexWriterConfig());
> exception.expect(IllegalArgumentException.class);
> exception.expectMessage(
> CoreMatchers.containsString("Cannot use addIndexes(Directory) with
> indexes that have been created by a different Lucene version"));
> w.addIndexes(lucene700Dir); // lucene 7.0.0 daily index
> w.forceMerge(1);
> w.close();
> }
>
> Let me know if you need more details.
>
>
> Many thanks
>
> Kannan.
>
>
Re: Lucene 6.1.0 index upgrade
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK I'm sorry but I think you'll have to re-index; e.g. how norms are
computed and stored in the index has changed, which means even if you could
somehow copy or merge the index over to the new version, the norms would be
incorrect.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 6:46 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@cengage.com> wrote:
> Hi Mike,
>
> It doesn’t work either, it throws “Cannot merge a segment that has been
> created with major version 6 into this index which has been created by
> major version 7" message. Let me try it after an upgrade as well. Same code
> below,
>
> @Test
> public void mergeUsingCodecReader() throws IOException {
> DirectoryReader ir = DirectoryReader.open(lucene610Dir);
> SegmentReader air = (SegmentReader)ir.leaves().get
> (0).reader();
>
> IndexWriterConfig config = newIndexWriterConfig();
> IndexWriter w = new IndexWriter(newEmptyDirectory("target"),
> config);
> exception.expect(IllegalArgumentException.class);
> exception.expectMessage("Cannot merge a segment that has
> been created with major version 6 into this index which has been created by
> major version 7");
>
> w.addIndexes(air);
> w.forceMerge(1);
> w.close();
> }
> Thanks,
> Kannan
>
> On 11/9/17, 5:07 PM, "Michael McCandless" <lu...@mikemccandless.com>
> wrote:
>
> Maybe try the addIndexes method that takes CodecReader[] instead? It
> will
> be somewhat slower, since it fully reads and writes the index, but it
> should work across major versions?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Nov 9, 2017 at 3:59 PM, Krishnamurthy, Kannan <
> Kannan.Krishnamurthy@cengage.com> wrote:
>
> > Greetings.
> >
> >
> > We are currently using Lucene 6.1.0, our typical index size is ~
> 200GB and
> > we are planning to move to Lucene 7.0.0.
> >
> >
> > Our daily indexing workflow currently uses IndexWriter.addIndexes()
> to add
> > an index(this will be on Lucene 7.0.0) to a copy of the of the 200GB
> index.
> > Further it calls IndexWriter.forceMerge(1) to create a single
> segment. The
> > addIndexes() now doesn't support different versions of index.
> >
> > We are facing an issue in upgrading to Lucene 7.0.0 due to this
> > limitation. We couldn't afford to do full reindex as there about 50
> > different indexes ~ 200GB each.
> >
> >
> > We tried using the LuceneUpgrade tool to upgrade our index to latest
> > version, however the IndexCreatedVersionMajor is still
> LUCENE_6_1_0.major,
> > This is preventing us calling IndexWriter.addIndexes() post upgrade.
> >
> >
> > I am wondering what are our options to upgrade without doing a full
> > reindex and to continue to use IndexWriter.addIndexes() post upgrade.
> >
> > The code snippet below shows the different api calls we make.
> >
> >
> >
> > @Test
> > public void addIndexWith7SegmentToDirectoryOnUpgradedSegment()
> throws
> > IOException {
> > IndexUpgrader upgrader = new IndexUpgrader(lucene610Dir); // lucene
> 6.1.0
> > directory 200 GB
> > upgrader.upgrade();
> >
> > IndexWriter w = new IndexWriter(lucene610Dir,
> newIndexWriterConfig());
> > exception.expect(IllegalArgumentException.class);
> > exception.expectMessage(
> > CoreMatchers.containsString("Cannot use addIndexes(Directory) with
> > indexes that have been created by a different Lucene version"));
> > w.addIndexes(lucene700Dir); // lucene 7.0.0 daily index
> > w.forceMerge(1);
> > w.close();
> > }
> >
> > Let me know if you need more details.
> >
> >
> > Many thanks
> >
> > Kannan.
> >
> >
>
>
>
Re: Lucene 6.1.0 index upgrade
Posted by "Krishnamurthy, Kannan" <Ka...@cengage.com>.
Hi Mike,
It doesn’t work either, it throws “Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7" message. Let me try it after an upgrade as well. Same code below,
@Test
public void mergeUsingCodecReader() throws IOException {
DirectoryReader ir = DirectoryReader.open(lucene610Dir);
SegmentReader air = (SegmentReader)ir.leaves().get(0).reader();
IndexWriterConfig config = newIndexWriterConfig();
IndexWriter w = new IndexWriter(newEmptyDirectory("target"), config);
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7");
w.addIndexes(air);
w.forceMerge(1);
w.close();
}
Thanks,
Kannan
On 11/9/17, 5:07 PM, "Michael McCandless" <lu...@mikemccandless.com> wrote:
Maybe try the addIndexes method that takes CodecReader[] instead? It will
be somewhat slower, since it fully reads and writes the index, but it
should work across major versions?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Nov 9, 2017 at 3:59 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@cengage.com> wrote:
> Greetings.
>
>
> We are currently using Lucene 6.1.0, our typical index size is ~ 200GB and
> we are planning to move to Lucene 7.0.0.
>
>
> Our daily indexing workflow currently uses IndexWriter.addIndexes() to add
> an index(this will be on Lucene 7.0.0) to a copy of the of the 200GB index.
> Further it calls IndexWriter.forceMerge(1) to create a single segment. The
> addIndexes() now doesn't support different versions of index.
>
> We are facing an issue in upgrading to Lucene 7.0.0 due to this
> limitation. We couldn't afford to do full reindex as there about 50
> different indexes ~ 200GB each.
>
>
> We tried using the LuceneUpgrade tool to upgrade our index to latest
> version, however the IndexCreatedVersionMajor is still LUCENE_6_1_0.major,
> This is preventing us calling IndexWriter.addIndexes() post upgrade.
>
>
> I am wondering what are our options to upgrade without doing a full
> reindex and to continue to use IndexWriter.addIndexes() post upgrade.
>
> The code snippet below shows the different api calls we make.
>
>
>
> @Test
> public void addIndexWith7SegmentToDirectoryOnUpgradedSegment() throws
> IOException {
> IndexUpgrader upgrader = new IndexUpgrader(lucene610Dir); // lucene 6.1.0
> directory 200 GB
> upgrader.upgrade();
>
> IndexWriter w = new IndexWriter(lucene610Dir, newIndexWriterConfig());
> exception.expect(IllegalArgumentException.class);
> exception.expectMessage(
> CoreMatchers.containsString("Cannot use addIndexes(Directory) with
> indexes that have been created by a different Lucene version"));
> w.addIndexes(lucene700Dir); // lucene 7.0.0 daily index
> w.forceMerge(1);
> w.close();
> }
>
> Let me know if you need more details.
>
>
> Many thanks
>
> Kannan.
>
>