You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by Jyothsna Bavisetti <jy...@oracle.com> on 2019/10/01 21:19:12 UTC

RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti 
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jörn Franke <jo...@gmail.com>.

I think it highly depends on how you want to do it. You could continue to run 4.6 and have another instance(a) with 8.2+. Then you just Index as you used to do it until now on the new one. Once you have all the data indexed you switch you lt application to 8.2. Of course you need to test it in dev, test etc environments.
Maybe in order to reduce time for the future and it you don’t use a database import. Convert your “rich” documents (eg pdf, wird, excel) in a suitable text format (or html) and start indexing from there. This will save you a lot of time on converting those documents in case of reindexing.

Then depending on the use case it could be also worthy to think about not indexing anymore old data (=> datensparsamkeit).

> Am 01.10.2019 um 23:19 schrieb Jyothsna Bavisetti <jy...@oracle.com>:
> 
> Hi Shawn,
> 
> Any doc or links for re indexing process. We are using Lucene core 8.0.0.
> 
> 
> Thanks,
> Jyothsna
> 
> -----Original Message-----
> From: Jyothsna Bavisetti 
> Sent: Thursday, September 26, 2019 11:51 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue
> 
> Hi Shawn,
> 
> Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?
> 
> Thanks,
> Jyothsna
> -----Original Message-----
> From: Shawn Heisey <el...@elyograg.org>
> Sent: Thursday, September 26, 2019 11:35 PM
> To: dev@lucene.apache.org
> Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue
> 
>> On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
>> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
>> to upgrade tool using:
>> 
>> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>> 
>> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \
> 
> <snip>
> 
>> Please let me know any option other than reindexing.
> 
> If you're upgrading more than one major version, you must reindex. 
> Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.
> 
> When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.
> 
> Thanks,
> Shawn
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jyothsna Bavisetti <jy...@oracle.com>.

In latest version form query is included in TermsQuery . So no difference in TermsQuery. 
Still unable to figure it out the reason why filter is not applicable when form field data is in different indexing folder.
Please suggest any ideas to debug.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti <jy...@oracle.com> 
Sent: Wednesday, October 30, 2019 11:53 AM
To: 'dev@lucene.apache.org' <de...@lucene.apache.org>
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi All,

1. After Upgrading Lucene from 4.6 to 8, facing issue in search process.
2.We are creating 5 different folder for indexing with different index id ( 5 folders from 5 different tables). During search process we will join all these data to display.  After upgrading we are facing issue in search data.
3. When we are applying filter for different fields from different index_id. It is searching for only one field. 
4. I am seeing difference in Join Query. 
JoinUtil.createJoinQuery(fromField, false, toField, Query, srch, ScoreMode.None); Query formation with different versions:
Lucene 4.6.0:
TermsQuery{field=case.id}
fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter) 

Lucene 8.0.0:
TermsQuery{field=case.id fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000}

fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter is also visible.)

Actual Filter Condition:
[Filter:
  condition: (&(case.caseType in (Scase))(case.sourceName in (SDEDUPLICATE))(&(history.attribute in (SExtendedAttribute1))(history.modifiedBy eq 3ff0000000000000))) ]

Please Suggest me , as new to the Lucene unable to predict it. 

Thank you.

Thanks,
Jyothsna
-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jyothsna Bavisetti <jy...@oracle.com>.

Hi All,

1. After Upgrading Lucene from 4.6 to 8, facing issue in search process.
2.We are creating 5 different folder for indexing with different index id ( 5 folders from 5 different tables). During search process we will join all these data to display.  After upgrading we are facing issue in search data.
3. When we are applying filter for different fields from different index_id. It is searching for only one field. 
4. I am seeing difference in Join Query. 
JoinUtil.createJoinQuery(fromField, false, toField, Query, srch, ScoreMode.None); 
Query formation with different versions:
Lucene 4.6.0:
TermsQuery{field=case.id}
fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter)
Lucene 8.0.0:
TermsQuery{field=case.idfromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000}

fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter is also visible.)

Please Suggest me , as new to the Lucene unable to predict it. 

Thank you.

Thanks,
Jyothsna
-----Original Message-----
From: Jyothsna Bavisetti 
Sent: Wednesday, October 2, 2019 2:49 AM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jyothsna Bavisetti <jy...@oracle.com>.

Hi All,

1. After Upgrading Lucene from 4.6 to 8, facing issue in search process.
2.We are creating 5 different folder for indexing with different index id ( 5 folders from 5 different tables). During search process we will join all these data to display.  After upgrading we are facing issue in search data.
3. When we are applying filter for different fields from different index_id. It is searching for only one field. 
4. I am seeing difference in Join Query. 
JoinUtil.createJoinQuery(fromField, false, toField, Query, srch, ScoreMode.None); 
Query formation with different versions:
Lucene 4.6.0:
TermsQuery{field=case.id}
fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter)
Lucene 8.0.0:
TermsQuery{field=case.idfromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000}

fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter is also visible.)

Please Suggest me , as new to the Lucene unable to predict it. 

-----Original Message-----
From: Jyothsna Bavisetti 
Sent: Wednesday, October 2, 2019 2:49 AM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Thank you,

Thanks,
Jyothsna

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jyothsna Bavisetti <jy...@oracle.com>.

Hi All,

Could please share some points when to go with longBitset vs FixedBitSet.

I read below Points:

BitSet of fixed length (numBits), backed by accessible (getBits()) long[], accessed with a long index. Use it only if you intend to store more than 2.1B bits, otherwise you should use FixedBitSet.
NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

Thanks,
Jyothsna
-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: Lucene index upgrade from 4.6 to 8 facing issue

Posted by Jyothsna Bavisetti <jy...@oracle.com>.

Hi All,

Please help me with below error:
java.lang.IllegalStateException: unexpected docvalues type NONE for field 'case.id' (expected one of [BINARY, SORTED]). Re-index with correct docvalues type.
	at org.apache.lucene.index.DocValues.checkField(DocValues.java:317)
	at org.apache.lucene.index.DocValues.getBinary(DocValues.java:352)
	at org.apache.lucene.search.join.DocValuesTermsCollector.lambda$binaryDocValues$0(DocValuesTermsCollector.java:48)
	at org.apache.lucene.search.join.DocValuesTermsCollector.doSetNextReader(DocValuesTermsCollector.java:44)
	at org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:33)
	at org.apache.lucene.search.join.GenericTermsCollector$2.getLeafCollector(GenericTermsCollector.java:128)


any pointers for this issue please let me know.

Thank you!

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti 
Sent: Wednesday, October 2, 2019 2:49 AM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: dev@lucene.apache.org
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <el...@elyograg.org>
Sent: Thursday, September 26, 2019 11:35 PM
To: dev@lucene.apache.org
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying 
> to upgrade tool using:
> 
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
> 
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex. 
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org