You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/04/27 03:10:42 UTC
[GitHub] [lucene] dnhatn opened a new pull request, #842: LUCENE-10518: Relax field consistency check for old indices
dnhatn opened a new pull request, #842:
URL: https://github.com/apache/lucene/pull/842
This change relaxes the field consistency check for old indices as we didn't enforce that in the previous versions. This commit also disables the optimization that relies on the field consistency for old indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mayya-sharipova commented on a diff in pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on code in PR #842:
URL: https://github.com/apache/lucene/pull/842#discussion_r860262352
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -178,7 +179,15 @@ public static FieldInfos getMergedFieldInfos(IndexReader reader) {
.filter(Objects::nonNull)
.findAny()
.orElse(null);
- final Builder builder = new Builder(new FieldNumbers(softDeletesField));
+ final int indexCreatedVersionMajor =
+ leaves.stream()
+ .map(l -> l.reader().getMetaData())
+ .filter(Objects::nonNull)
+ .mapToInt(r -> r.getCreatedVersionMajor())
+ .min()
+ .orElse(Version.LATEST.major);
Review Comment:
I've checked and even if we force merge an older index, its new segment will still have metadata `getCreatedVersionMajor()` of older version, so we are good here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dnhatn commented on pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
dnhatn commented on PR #842:
URL: https://github.com/apache/lucene/pull/842#issuecomment-1112420113
@mayya-sharipova @jpountz Thanks for reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dnhatn merged pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
dnhatn merged PR #842:
URL: https://github.com/apache/lucene/pull/842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mayya-sharipova commented on a diff in pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on code in PR #842:
URL: https://github.com/apache/lucene/pull/842#discussion_r861981910
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -178,7 +179,15 @@ public static FieldInfos getMergedFieldInfos(IndexReader reader) {
.filter(Objects::nonNull)
.findAny()
.orElse(null);
- final Builder builder = new Builder(new FieldNumbers(softDeletesField));
+ final int indexCreatedVersionMajor =
+ leaves.stream()
+ .map(l -> l.reader().getMetaData())
+ .filter(Objects::nonNull)
+ .mapToInt(r -> r.getCreatedVersionMajor())
+ .min()
+ .orElse(Version.LATEST.major);
Review Comment:
I am interested when it is possible for segments to have different `getCreatedVersionMajor()`? Looking at the [code](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1125) of `IndexWriter` it looks like if we open an `IndexWriter` for append, new segments will always have the same `indexCreatedVersionMajor` as previous segments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dnhatn commented on a diff in pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
dnhatn commented on code in PR #842:
URL: https://github.com/apache/lucene/pull/842#discussion_r861098734
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -350,6 +360,11 @@ static final class FieldNumbers {
this.omitNorms = new HashMap<>();
this.storeTermVectors = new HashMap<>();
this.softDeletesFieldName = softDeletesFieldName;
+ this.strictlyConsistent = indexCreatedVersionMajor >= 9;
+ }
+
+ FieldNumbers(String softDeletesFieldName) {
Review Comment:
+1, fixed in https://github.com/apache/lucene/pull/842/commits/10c110500e4d0f78fcdf0b167e4061025d6b6927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
jpountz commented on code in PR #842:
URL: https://github.com/apache/lucene/pull/842#discussion_r860789641
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -350,6 +360,11 @@ static final class FieldNumbers {
this.omitNorms = new HashMap<>();
this.storeTermVectors = new HashMap<>();
this.softDeletesFieldName = softDeletesFieldName;
+ this.strictlyConsistent = indexCreatedVersionMajor >= 9;
+ }
+
+ FieldNumbers(String softDeletesFieldName) {
Review Comment:
does this ctor have many call sites, could we always use the 2-args ctor?
##########
lucene/CHANGES.txt:
##########
@@ -94,8 +94,10 @@ Bug Fixes
no documents instead of throwing an NPE. (Greg Miller)
* LUCENE-10470: Check if polygon has been successfully tessellated before we fail (we are failing some valid
- tessellations) and allow filtering edges that fold on top of the previous one. (Ignacio Vera)
-
+ tessellations) and allow filtering edges that fold on top of the previous one. (Ignacio Vera)
+
Review Comment:
remove the space on this empty line?
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -178,7 +179,15 @@ public static FieldInfos getMergedFieldInfos(IndexReader reader) {
.filter(Objects::nonNull)
.findAny()
.orElse(null);
- final Builder builder = new Builder(new FieldNumbers(softDeletesField));
+ final int indexCreatedVersionMajor =
+ leaves.stream()
+ .map(l -> l.reader().getMetaData())
+ .filter(Objects::nonNull)
+ .mapToInt(r -> r.getCreatedVersionMajor())
+ .min()
+ .orElse(Version.LATEST.major);
Review Comment:
I'd like it a bit better if we moved to a for-each loop and threw an error if two segments happen to have different versions, since we'd be in trouble in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mayya-sharipova commented on a diff in pull request #842: LUCENE-10518: Relax field consistency check for old indices
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on code in PR #842:
URL: https://github.com/apache/lucene/pull/842#discussion_r861981910
##########
lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:
##########
@@ -178,7 +179,15 @@ public static FieldInfos getMergedFieldInfos(IndexReader reader) {
.filter(Objects::nonNull)
.findAny()
.orElse(null);
- final Builder builder = new Builder(new FieldNumbers(softDeletesField));
+ final int indexCreatedVersionMajor =
+ leaves.stream()
+ .map(l -> l.reader().getMetaData())
+ .filter(Objects::nonNull)
+ .mapToInt(r -> r.getCreatedVersionMajor())
+ .min()
+ .orElse(Version.LATEST.major);
Review Comment:
I am interested when it is possible for segments to have different `getCreatedVersionMajor()`? Looking at the [code](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1125) of `IndexWriter` it looks like if we open an `IndexWriter` for append, new segments will always have the same `indexCreatedVersion` as previous segments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org