You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "gf2121 (via GitHub)" <gi...@apache.org> on 2023/12/11 12:44:45 UTC

[PR] Add BWC test to reveal #12895 [lucene]

gf2121 opened a new pull request, #12912:
URL: https://github.com/apache/lucene/pull/12912

   This patch get a 40000-terms subset (~200KB) of `wikibigall` terms dict, which can reproduce #12895. I have verified #12900 can fix the expection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12912:
URL: https://github.com/apache/lucene/pull/12912#discussion_r1422446225


##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   wwwwuuuuuuaaaaaaah. This must go away.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12912:
URL: https://github.com/apache/lucene/pull/12912#issuecomment-1855739201

   Solr used that, as solr is no longer part of our tree we could add sleep to fobiddenaps (maybe globally for both tests and main code). Maybe Lucene does not sleep in tests, so it could be an easy PR. Let's investigate. 🫣


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on code in PR #12912:
URL: https://github.com/apache/lucene/pull/12912#discussion_r1422443348


##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   Thanks for review @uschindler @benwtrent !
   
   I closed this since I say @mikemccand [did](https://github.com/apache/lucene/issues/12901#issuecomment-1849981059) similar things and his dictionary is much smaller than mine :)
   
   > this must go away, please add nocommit.
   
   I'm very new to this part. I copied this pattern from another [existing test ](https://github.com/apache/lucene/blob/069c04877011a7e305dfd9e23e194fe4fd5b59e6/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java#L228). I can open another PR to clean this up if we are sure it makes no sense.



##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   Thanks for review @uschindler @benwtrent !
   
   I closed this since I saw @mikemccand [did](https://github.com/apache/lucene/issues/12901#issuecomment-1849981059) similar things and his dictionary is much smaller than mine :)
   
   > this must go away, please add nocommit.
   
   I'm very new to this part. I copied this pattern from another [existing test ](https://github.com/apache/lucene/blob/069c04877011a7e305dfd9e23e194fe4fd5b59e6/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java#L228). I can open another PR to clean this up if we are sure it makes no sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #12912:
URL: https://github.com/apache/lucene/pull/12912#issuecomment-1852755681

   Oh thanks for working on this @gf2121 -- sorry that we both did it at the same time!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12912:
URL: https://github.com/apache/lucene/pull/12912#discussion_r1422428244


##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   this must go away, please add `nocommit`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 closed pull request #12912: Add BWC test to reveal #12895
URL: https://github.com/apache/lucene/pull/12912


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #12912:
URL: https://github.com/apache/lucene/pull/12912#discussion_r1422433156


##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   I thought `-Ptests.bwcdir=` made this sleep hack unnecessary? Seems like we shouldn't have any of these data generative tests using `Thread.sleep`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on code in PR #12912:
URL: https://github.com/apache/lucene/pull/12912#discussion_r1422514730


##########
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBackwardsCompatibility.java:
##########
@@ -2265,4 +2268,47 @@ public void testReadNMinusTwoSegmentInfos() throws IOException {
       }
     }
   }
+
+  public static final String[] oldWikiTermsNames = {"wikiterms.9.8.0.zip"};
+
+  public void testCreateWikiTermsIndex() throws Exception {
+    Path indexDir = getIndexDir().resolve("wikiterms");
+    Files.deleteIfExists(indexDir);
+    Directory dir = newFSDirectory(indexDir);
+
+    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig());
+    BytesRefIterator termsEnum = TermsTestUtil.load(TermsTestUtil.WIKI_40000_TERMS_FILE);
+    BytesRef term;
+    while ((term = termsEnum.next()) != null) {
+      Document document = new Document();
+      document.add(new StringField("body", term, Field.Store.NO));
+      writer.addDocument(document);
+    }
+
+    writer.flush();
+    writer.commit();
+    writer.forceMerge(1);
+    writer.close();
+    dir.close();
+
+    // Gives you time to copy the index out!: (there is also
+    // a test option to not remove temp dir...):
+    Thread.sleep(100000);

Review Comment:
   I raised https://github.com/apache/lucene/pull/12914



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #12912:
URL: https://github.com/apache/lucene/pull/12912#issuecomment-1855716194

   ... but at least it lead to discovering a horrifying `Thread.sleep` in our test code!
   
   Can we ban `Thread.sleep` throughout our code?  Or are there actually useful places for it... I do like sleeping myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add BWC test to reveal #12895 [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #12912:
URL: https://github.com/apache/lucene/pull/12912#issuecomment-1856040733

   > Solr used that, as solr is no longer part of our tree we could add sleep to fobiddenaps (maybe globally for both tests and main code). Maybe Lucene does not sleep in tests, so it could be an easy PR. Let's investigate. 🫣
   
   Awesome, I opened #12946 for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org