You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/03/03 22:16:43 UTC
[GitHub] [lucene-solr] msfroh opened a new pull request #1313: LUCENE-8962:
Split test case
msfroh opened a new pull request #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313
The testMergeOnCommit test case was trying to verify too many things
at once: basic semantics of merge on commit and proper behavior when
a bunch of indexing threads are writing and committing all at once.
Splitting the test into two should make the tests more robust - one
will verify basic behavior, with strict assertions on invariants, while
the other just verifies that everything gets indexed and we don't throw
an exception when multiple threads are indexing and merging on commit.
Also, the part of the test that is now testMultithreadedMergeOnCommit
can take several seconds to run, so moving it to the @Nightly set.
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Lucene or Solr:
* https://issues.apache.org/jira/projects/LUCENE
* https://issues.apache.org/jira/projects/SOLR
You will need to create an account in Jira in order to create an issue.
The title of the PR should reference the Jira issue number in the form:
* LUCENE-####: <short description of problem or changes>
* SOLR-####: <short description of problem or changes>
LUCENE and SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Fixing an intermittent test failure on testMergeOnCommit.
# Solution
Split the logic from testMergeOnCommit into two test cases. The basic test has consistently passed, and actually verifies the merge on commit invariants. The more complicated, more potentially-brittle multithreaded test doesn't necessarily satisfy clear invariants (as we may be merging on commit from multiple threads, which could result in multiple segments in the end), but it should never throw an exception or lose any updates.
# Tests
Split existing test case into two test cases. Ran tests multiple times.
# Checklist
Please review the following and check all that apply:
- [X] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [X] I have created a Jira issue and added the issue ID to my pull request title.
- [X] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [X] I have developed this patch against the `master` branch.
- [X] I have run `ant precommit` and the appropriate test suite.
- [X] I have added tests for my changes.
- [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on a change in pull request #1313:
LUCENE-8962: Split test case
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#discussion_r388264716
##########
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriterMergePolicy.java
##########
@@ -298,63 +320,44 @@ public void testMergeOnCommit() throws IOException, InterruptedException {
DirectoryReader firstReader = DirectoryReader.open(firstWriter);
assertEquals(5, firstReader.leaves().size());
firstReader.close();
- firstWriter.close();
-
- MergePolicy mergeOnCommitPolicy = new LogDocMergePolicy() {
- @Override
- public MergeSpecification findFullFlushMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergeContext mergeContext) {
- // Optimize down to a single segment on commit
- if (mergeTrigger == MergeTrigger.COMMIT && segmentInfos.size() > 1) {
- List<SegmentCommitInfo> nonMergingSegments = new ArrayList<>();
- for (SegmentCommitInfo sci : segmentInfos) {
- if (mergeContext.getMergingSegments().contains(sci) == false) {
- nonMergingSegments.add(sci);
- }
- }
- if (nonMergingSegments.size() > 1) {
- MergeSpecification mergeSpecification = new MergeSpecification();
- mergeSpecification.add(new OneMerge(nonMergingSegments));
- return mergeSpecification;
- }
- }
- return null;
- }
- };
+ firstWriter.close(); // When this writer closes, it does not merge on commit.
- AtomicInteger abandonedMerges = new AtomicInteger(0);
IndexWriterConfig iwc = newIndexWriterConfig(new MockAnalyzer(random()))
- .setMergePolicy(mergeOnCommitPolicy)
- .setIndexWriterEvents(new IndexWriterEvents() {
- @Override
- public void beginMergeOnCommit() {
-
- }
-
- @Override
- public void finishMergeOnCommit() {
+ .setMergePolicy(MERGE_ON_COMMIT_POLICY);
- }
-
- @Override
- public void abandonedMergesOnCommit(int abandonedCount) {
- abandonedMerges.incrementAndGet();
- }
- });
IndexWriter writerWithMergePolicy = new IndexWriter(dir, iwc);
-
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // No changes. Commit doesn't trigger a merge.
DirectoryReader unmergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(5, unmergedReader.leaves().size()); // Don't merge unless there's a change
+ assertEquals(5, unmergedReader.leaves().size());
unmergedReader.close();
TestIndexWriter.addDoc(writerWithMergePolicy);
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // Doc added, do merge on commit.
+ assertEquals(1, writerWithMergePolicy.getSegmentCount()); //
DirectoryReader mergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(1, mergedReader.leaves().size()); // Now we merge on commit
+ assertEquals(1, mergedReader.leaves().size());
mergedReader.close();
+ try (IndexReader reader = writerWithMergePolicy.getReader()) {
+ IndexSearcher searcher = new IndexSearcher(reader);
+ assertEquals(6, reader.numDocs());
+ assertEquals(6, searcher.count(new MatchAllDocsQuery()));
+ }
+
+ writerWithMergePolicy.close();
+ dir.close();
+ }
+
+ // Test that when we have multiple indexing threads merging on commit, we never throw an exception.
+ @Nightly
Review comment:
Yes, I think given it does not assert anything -- just makes sure no exceptions occur -- we should already be well-covered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov merged pull request #1313: LUCENE-8962:
Split test case
Posted by GitBox <gi...@apache.org>.
msokolov merged pull request #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on a change in pull request #1313:
LUCENE-8962: Split test case
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#discussion_r387682345
##########
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriterMergePolicy.java
##########
@@ -298,63 +320,44 @@ public void testMergeOnCommit() throws IOException, InterruptedException {
DirectoryReader firstReader = DirectoryReader.open(firstWriter);
assertEquals(5, firstReader.leaves().size());
firstReader.close();
- firstWriter.close();
-
- MergePolicy mergeOnCommitPolicy = new LogDocMergePolicy() {
- @Override
- public MergeSpecification findFullFlushMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergeContext mergeContext) {
- // Optimize down to a single segment on commit
- if (mergeTrigger == MergeTrigger.COMMIT && segmentInfos.size() > 1) {
- List<SegmentCommitInfo> nonMergingSegments = new ArrayList<>();
- for (SegmentCommitInfo sci : segmentInfos) {
- if (mergeContext.getMergingSegments().contains(sci) == false) {
- nonMergingSegments.add(sci);
- }
- }
- if (nonMergingSegments.size() > 1) {
- MergeSpecification mergeSpecification = new MergeSpecification();
- mergeSpecification.add(new OneMerge(nonMergingSegments));
- return mergeSpecification;
- }
- }
- return null;
- }
- };
+ firstWriter.close(); // When this writer closes, it does not merge on commit.
- AtomicInteger abandonedMerges = new AtomicInteger(0);
IndexWriterConfig iwc = newIndexWriterConfig(new MockAnalyzer(random()))
- .setMergePolicy(mergeOnCommitPolicy)
- .setIndexWriterEvents(new IndexWriterEvents() {
- @Override
- public void beginMergeOnCommit() {
-
- }
-
- @Override
- public void finishMergeOnCommit() {
+ .setMergePolicy(MERGE_ON_COMMIT_POLICY);
- }
-
- @Override
- public void abandonedMergesOnCommit(int abandonedCount) {
- abandonedMerges.incrementAndGet();
- }
- });
IndexWriter writerWithMergePolicy = new IndexWriter(dir, iwc);
-
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // No changes. Commit doesn't trigger a merge.
DirectoryReader unmergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(5, unmergedReader.leaves().size()); // Don't merge unless there's a change
+ assertEquals(5, unmergedReader.leaves().size());
unmergedReader.close();
TestIndexWriter.addDoc(writerWithMergePolicy);
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // Doc added, do merge on commit.
+ assertEquals(1, writerWithMergePolicy.getSegmentCount()); //
DirectoryReader mergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(1, mergedReader.leaves().size()); // Now we merge on commit
+ assertEquals(1, mergedReader.leaves().size());
mergedReader.close();
+ try (IndexReader reader = writerWithMergePolicy.getReader()) {
+ IndexSearcher searcher = new IndexSearcher(reader);
+ assertEquals(6, reader.numDocs());
+ assertEquals(6, searcher.count(new MatchAllDocsQuery()));
+ }
+
+ writerWithMergePolicy.close();
+ dir.close();
+ }
+
+ // Test that when we have multiple indexing threads merging on commit, we never throw an exception.
+ @Nightly
Review comment:
I ran this test a few times, and noticed it takes 2-3 minutes to complete. Was it this slow before? I don't remember it taking so long in the previous version. Perhaps it's because it's Nightly now so the constants are made larger?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on issue #1313: LUCENE-8962: Split
test case
Posted by GitBox <gi...@apache.org>.
msokolov commented on issue #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#issuecomment-595436417
I verified this fixes the `TestIndexWriterExceptions2.testBasics` reported by @jpountz and also beasted that test 1000x just in case. I think we need to get ahead of this given all the fail emails from these tests, and the upcoming 8.5 release, so I'll push today
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msfroh commented on a change in pull request #1313:
LUCENE-8962: Split test case
Posted by GitBox <gi...@apache.org>.
msfroh commented on a change in pull request #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#discussion_r387894237
##########
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriterMergePolicy.java
##########
@@ -298,63 +320,44 @@ public void testMergeOnCommit() throws IOException, InterruptedException {
DirectoryReader firstReader = DirectoryReader.open(firstWriter);
assertEquals(5, firstReader.leaves().size());
firstReader.close();
- firstWriter.close();
-
- MergePolicy mergeOnCommitPolicy = new LogDocMergePolicy() {
- @Override
- public MergeSpecification findFullFlushMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergeContext mergeContext) {
- // Optimize down to a single segment on commit
- if (mergeTrigger == MergeTrigger.COMMIT && segmentInfos.size() > 1) {
- List<SegmentCommitInfo> nonMergingSegments = new ArrayList<>();
- for (SegmentCommitInfo sci : segmentInfos) {
- if (mergeContext.getMergingSegments().contains(sci) == false) {
- nonMergingSegments.add(sci);
- }
- }
- if (nonMergingSegments.size() > 1) {
- MergeSpecification mergeSpecification = new MergeSpecification();
- mergeSpecification.add(new OneMerge(nonMergingSegments));
- return mergeSpecification;
- }
- }
- return null;
- }
- };
+ firstWriter.close(); // When this writer closes, it does not merge on commit.
- AtomicInteger abandonedMerges = new AtomicInteger(0);
IndexWriterConfig iwc = newIndexWriterConfig(new MockAnalyzer(random()))
- .setMergePolicy(mergeOnCommitPolicy)
- .setIndexWriterEvents(new IndexWriterEvents() {
- @Override
- public void beginMergeOnCommit() {
-
- }
-
- @Override
- public void finishMergeOnCommit() {
+ .setMergePolicy(MERGE_ON_COMMIT_POLICY);
- }
-
- @Override
- public void abandonedMergesOnCommit(int abandonedCount) {
- abandonedMerges.incrementAndGet();
- }
- });
IndexWriter writerWithMergePolicy = new IndexWriter(dir, iwc);
-
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // No changes. Commit doesn't trigger a merge.
DirectoryReader unmergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(5, unmergedReader.leaves().size()); // Don't merge unless there's a change
+ assertEquals(5, unmergedReader.leaves().size());
unmergedReader.close();
TestIndexWriter.addDoc(writerWithMergePolicy);
- writerWithMergePolicy.commit();
+ writerWithMergePolicy.commit(); // Doc added, do merge on commit.
+ assertEquals(1, writerWithMergePolicy.getSegmentCount()); //
DirectoryReader mergedReader = DirectoryReader.open(writerWithMergePolicy);
- assertEquals(1, mergedReader.leaves().size()); // Now we merge on commit
+ assertEquals(1, mergedReader.leaves().size());
mergedReader.close();
+ try (IndexReader reader = writerWithMergePolicy.getReader()) {
+ IndexSearcher searcher = new IndexSearcher(reader);
+ assertEquals(6, reader.numDocs());
+ assertEquals(6, searcher.count(new MatchAllDocsQuery()));
+ }
+
+ writerWithMergePolicy.close();
+ dir.close();
+ }
+
+ // Test that when we have multiple indexing threads merging on commit, we never throw an exception.
+ @Nightly
Review comment:
Oh, that's probably the case.
Given that there are other tests that flex indexing across multiple threads and `MockRandomMergePolicy` will occasionally merge on commit, maybe this test doesn't add any value?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on issue #1313: LUCENE-8962: Split
test case
Posted by GitBox <gi...@apache.org>.
dnhatn commented on issue #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#issuecomment-595503029
@msfroh @msokolov Thank you for working on the fix. Unfortunately, this is still an issue. Many Elasticsearch tests are [failing](https://github.com/elastic/elasticsearch/issues/53195) even with this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on issue #1313: LUCENE-8962: Split
test case
Posted by GitBox <gi...@apache.org>.
dnhatn commented on issue #1313: LUCENE-8962: Split test case
URL: https://github.com/apache/lucene-solr/pull/1313#issuecomment-595597021
I've left some comments in https://github.com/apache/lucene-solr/pull/1155.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org