You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "NazerkeBS (via GitHub)" <gi...@apache.org> on 2024/01/04 10:51:34 UTC

[PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

NazerkeBS opened a new pull request, #2176:
URL: https://github.com/apache/solr/pull/2176

   
   
   *https://issues.apache.org/jira/browse/SOLR-17097
   
   
   # Description
   
   Upgrade Solr to use Lucene 9.9.1
   
   # Solution
   
   Please provide a short description of the approach taken to implement your solution.
   
   # Tests
   
   Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
   - [ ] I have created a Jira issue and added the issue ID to my pull request title.
   - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
   - [ ] I have developed this patch against the `main` branch.
   - [ ] I have run `./gradlew check`.
   - [ ] I have added tests for my changes.
   - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.2 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1471203055


##########
solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java:
##########
@@ -109,9 +109,10 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
           minVersion = leafVersion;
         }
       }
-      int createdVersionMajor =
-          reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor();
-      metaData = new LeafMetaData(createdVersionMajor, minVersion, null);
+      LeafMetaData leafMetaData = reader.leaves().get(0).reader().getMetaData();
+      metaData =
+          new LeafMetaData(
+              leafMetaData.getCreatedVersionMajor(), minVersion, null, leafMetaData.hasBlocks());

Review Comment:
   > Out of curiosity, why don't we also specify the `sort` parameter here (3rd parameter)? It could be retrieved from existing leaf metadata (this is not new with this change).
   
   I wonder if this might be because we don't yet have 'full' index sorting directly configurable in Solr -- https://issues.apache.org/jira/browse/SOLR-13681 exists for that -- i.e. commonly the `sort` would indeed be `null` except when a `SortingMergePolicy` is configured in which case we'd have 'partial' index sorting i.e. some segments could be sorted and some could be unsorted and thus it would be tricky to confidently set the `sort` value to anything other than the `null` value.
   
   > ... (this is not new with this change).
   
   My preference would be to not change the parameter setting as part of the Lucene upgrade but to perhaps leave a note on SOLR-13681 or have a separate issue for that change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1877156966

   Looks like two of the tests are failing?   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1893752822

   cross-referencing w.r.t. the `TestScoreJoinQPNoScore#testRandomJoin` failures:
   * https://lists.apache.org/thread/ms65gkmm86cpp5ntdo555ob8mrgg64o5 thread on the Lucene Dev mailing list - thanks @NazerkeBS for creating it.
   * https://github.com/apache/lucene/issues/13018 issue opened based on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.2 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1916961121

   > ... I'm going to test this out locally ...
   
   Ran tests locally too, they passed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1446372088


##########
solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java:
##########
@@ -97,7 +97,7 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
     in = reader;
     in.registerParentReader(this);
     if (reader.leaves().isEmpty()) {
-      metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null);
+      metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null, false);

Review Comment:
   ref: https://github.com/apache/lucene/commit/26043819a5b16858683737d3bdc708e05f33d49f



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1446410422


##########
solr/core/src/test/org/apache/solr/search/TestSolrQueryParser.java:
##########
@@ -247,9 +247,16 @@ public void testSyntax() throws Exception {
       SchemaField foo_dt = h.getCore().getLatestSchema().getField("foo_dt");
       String expected = "foo_dt:2013-09-11T00:00:00Z";
       if (foo_dt.getType().isPointField()) {
-        expected = "(foo_dt:[1378857600000 TO 1378857600000])";
+        expected = "foo_dt:[1378857600000 TO 1378857600000]";
         if (foo_dt.hasDocValues() && foo_dt.indexed()) {
-          expected = "IndexOrDocValuesQuery" + expected;
+          expected =
+              "IndexOrDocValuesQuery(IndexOrDocValuesQuery(indexQuery="
+                  + expected
+                  + ", dvQuery="
+                  + expected
+                  + "))";
+        } else {
+          expected = "(" + expected + ")";

Review Comment:
   ref: https://github.com/apache/lucene/commit/690462ceb433a4394ca3836c190e0311ce15cd63



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.2 [solr]

Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija merged PR #2176:
URL: https://github.com/apache/solr/pull/2176


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "psalagnac (via GitHub)" <gi...@apache.org>.
psalagnac commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1447433615


##########
solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java:
##########
@@ -109,9 +109,10 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
           minVersion = leafVersion;
         }
       }
-      int createdVersionMajor =
-          reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor();
-      metaData = new LeafMetaData(createdVersionMajor, minVersion, null);
+      LeafMetaData leafMetaData = reader.leaves().get(0).reader().getMetaData();
+      metaData =
+          new LeafMetaData(
+              leafMetaData.getCreatedVersionMajor(), minVersion, null, leafMetaData.hasBlocks());

Review Comment:
   Out of curiosity, why don't we also specify the `sort` parameter here (3rd parameter)? It could be retrieved from existing leaf metadata (this is not new with this change).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1446374641


##########
solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java:
##########
@@ -111,7 +111,12 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
       }
       int createdVersionMajor =
           reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor();
-      metaData = new LeafMetaData(createdVersionMajor, minVersion, null);
+      metaData =
+          new LeafMetaData(
+              createdVersionMajor,
+              minVersion,
+              null,
+              reader.leaves().get(0).reader().getMetaData().hasBlocks());

Review Comment:
   minor/subjective: `reader.leaves().get(0).reader().getMetaData()` assignment to a local variable so that both the `createdVersionMajor` compute above and the `hasBlock` determination here can use it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "NazerkeBS (via GitHub)" <gi...@apache.org>.
NazerkeBS commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1448955485


##########
solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java:
##########
@@ -109,9 +109,10 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
           minVersion = leafVersion;
         }
       }
-      int createdVersionMajor =
-          reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor();
-      metaData = new LeafMetaData(createdVersionMajor, minVersion, null);
+      LeafMetaData leafMetaData = reader.leaves().get(0).reader().getMetaData();
+      metaData =
+          new LeafMetaData(
+              leafMetaData.getCreatedVersionMajor(), minVersion, null, leafMetaData.hasBlocks());

Review Comment:
   good point, we could specify `leafMetaData.getSort() ` if docs are in no particular order, it would be null. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.2 [solr]

Posted by "NazerkeBS (via GitHub)" <gi...@apache.org>.
NazerkeBS commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1914814723

   all tests passed locally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.2 [solr]

Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1916755993

   This looks awesome, thanks to all!  I'm going to test this out locally and then plan to merge later today!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "NazerkeBS (via GitHub)" <gi...@apache.org>.
NazerkeBS commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1877411210

   > It's great that we are working to stay up to date with Lucene! Looks like two of the tests are failing?
   
   `TestSolrQueryParser#testSyntax `- re-run it a few times locally , surprisingly it passed then failed, passed, failed; somewhat flaky.     
   
   `TestScoreJoinQPNoScore#testRandomJoin` - took a look into it;   it seems the problem relates to ScoreMode `ScoreJoinQParserPlugin`. Locally in my machine`org.apache.lucene.search.ScoreMode` class  is not found and this surprised me. Looked into jars and seeing `ScoreMode.class` instead of `ScoreMode`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1910399161

   FYI I've tried out the Solr-with-local-Lucene documentation steps -- #2223 tweaks are a side effect -- to confirm and have confirmed that https://github.com/apache/lucene/pull/13014 solves the `TestScoreJoinQPNoScore#testRandomJoin` failures.
   
   ```
   ./gradlew test --tests TestScoreJoinQPNoScore.testRandomJoin -Plucene.dev.version=9.9.2-SNAPSHOT
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "NazerkeBS (via GitHub)" <gi...@apache.org>.
NazerkeBS commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1910469447

   > FYI I've tried out the Solr-with-local-Lucene documentation steps -- #2223 tweaks are a side effect -- to confirm and have confirmed that [apache/lucene#13014](https://github.com/apache/lucene/pull/13014) solves the `TestScoreJoinQPNoScore#testRandomJoin` failures.
   > 
   > ```
   > ./gradlew test --tests TestScoreJoinQPNoScore.testRandomJoin -Plucene.dev.version=9.9.2-SNAPSHOT
   > ```
   
   Thanks Christine for quickly testing it.  I will wait for the v.9.9.2 official release and update this PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on code in PR #2176:
URL: https://github.com/apache/solr/pull/2176#discussion_r1446361971


##########
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java:
##########
@@ -497,7 +498,8 @@ protected int compare(int i, int j) {
           // :TODO: would be simpler to always serialize every position of SortField[]
           if (type == SortField.Type.SCORE || type == SortField.Type.DOC) continue;
 
-          FieldComparator<?> comparator = sortField.getComparator(1, true);
+          FieldComparator<?> comparator =
+              sortField.getComparator(1, Pruning.GREATER_THAN_OR_EQUAL_TO);

Review Comment:
   ref: https://github.com/apache/lucene/commit/d9109907bcafd9e3ac6a23bff9e5b1b2605bb2f8



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17097: Upgrade Solr to use Lucene 9.9.1 [solr]

Posted by "cpoerschke (via GitHub)" <gi...@apache.org>.
cpoerschke commented on PR #2176:
URL: https://github.com/apache/solr/pull/2176#issuecomment-1883502943

   > `TestSolrQueryParser#testSyntax `- re-run it a few times locally , surprisingly it passed then failed, passed, failed; somewhat flaky.
   
   Was able to reproduce the failure locally with `./gradlew test --tests TestSolrQueryParser.testSyntax -Dtests.seed=9EEE6CC2A6BE4FEA` from the CI run. Added a commit to account for the https://github.com/apache/lucene/commit/690462ceb433a4394ca3836c190e0311ce15cd63 change and now it passes locally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org