You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2023/09/11 13:31:27 UTC
[GitHub] [lucene] jpountz opened a new pull request, #12549: Run merge-on-full-flush even though no changes got flushed.
jpountz opened a new pull request, #12549:
URL: https://github.com/apache/lucene/pull/12549
Currently, merge-on-full-flush only checks if merges need to run if changes have been flushed to disk. This prevents from having different merging logic for refreshes and commits, since the merge policy would not be checked upon commit if no new documents got indexed since the previous refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12549: Run merge-on-full-flush even though no changes got flushed.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12549:
URL: https://github.com/apache/lucene/pull/12549#discussion_r1322592471
##########
lucene/core/src/test/org/apache/lucene/index/TestIndexWriter.java:
##########
@@ -518,11 +518,10 @@ public void testFlushWithNoMerging() throws IOException {
doc.add(newField("field", "aaa", customType));
for (int i = 0; i < 19; i++) writer.addDocument(doc);
writer.flush(false, true);
- writer.close();
- SegmentInfos sis = SegmentInfos.readLatestCommit(dir);
// Since we flushed w/o allowing merging we should now
// have 10 segments
- assertEquals(10, sis.size());
+ assertEquals(10, writer.getSegmentCount());
+ writer.close();
Review Comment:
Note to reviewers: this test started failing because `writer.close()` now triggers merges because of the change. So I changed the test to check the number of segments after the `flush` call but before the `close` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Run merge-on-full-flush even though no changes got flushed. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #12549:
URL: https://github.com/apache/lucene/pull/12549#issuecomment-1777530886
I reverted as this causes a deadlock in TestStressIndexing:
```
"TEST-TestStressIndexing.testStressIndexAndSearching-seed#[D4B60FA81EB58FF3]" #42 [282516] prio=5 os_prio=0 cpu=314.69ms elapsed=8.44s tid=0x00007f9144078860 nid=282516 in Object.wait() [0x00007f9195122000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait0(java.base@21/Native Method)
- waiting on <0x00000006207504f0> (a org.apache.lucene.index.IndexWriter)
at java.lang.Object.wait(java.base@21/Object.java:366)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:5419)
- locked <0x00000006207504f0> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2723)
- locked <0x00000006207504f0> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2471)
- locked <0x00000006207504f0> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2451)
- locked <0x0000000620750a88> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1336)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1364)
at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:973)
- locked <0x000000062702c9f8> (a org.apache.lucene.tests.store.MockDirectoryWrapper)
at org.apache.lucene.index.TestStressIndexing.testStressIndexAndSearching(TestStressIndexing.java:171)
at java.lang.invoke.LambdaForm$DMH/0x00007f919810d000.invokeVirtual(java.base@21/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f919810d400.invoke(java.base@21/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21/Invokers$Holder)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21/DirectMethodHandleAccessor.java:153)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21/DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke(java.base@21/Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda/0x00007f919808ed60.run(Unknown Source)
at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
at java.lang.Thread.run(java.base@21/Thread.java:1583)
"Lucene Merge Thread #0" #53 [282533] daemon prio=5 os_prio=0 cpu=1.59ms elapsed=7.81s tid=0x00007f910815ee70 nid=282533 waiting for monitor entry [0x00007f9195022000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lucene.tests.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:692)
- waiting to lock <0x000000062702c9f8> (a org.apache.lucene.tests.store.MockDirectoryWrapper)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.index.ConcurrentMergeScheduler$2.createOutput(ConcurrentMergeScheduler.java:290)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:130)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:233)
at org.apache.lucene.index.SegmentMerger$$Lambda/0x00007f91981801f8.merge(Unknown Source)
at org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:273)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:110)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5172)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4705)
at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6460)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:639)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:700)
```
This seems to be more a problem with `MockDirectoryWrapper` than with the change to me, though I'm unsure how to fix MDW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Run merge-on-full-flush even though no changes got flushed. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #12549:
URL: https://github.com/apache/lucene/pull/12549#issuecomment-1779155292
Thanks! I was thinking of something along the lines of the diff you shared, I had not thought of the SerialMergeScheduler approach. I'll check it works and push this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Run merge-on-full-flush even though no changes got flushed. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz merged PR #12549:
URL: https://github.com/apache/lucene/pull/12549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Run merge-on-full-flush even though no changes got flushed. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #12549:
URL: https://github.com/apache/lucene/pull/12549#issuecomment-1779816530
I ended up implementing your other suggestion. MDW generally expects that this IndexWriter instantiation will not do merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12549: Run merge-on-full-flush even though no changes got flushed.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12549:
URL: https://github.com/apache/lucene/pull/12549#discussion_r1322599113
##########
lucene/core/src/test/org/apache/lucene/index/TestIndexWriterDelete.java:
##########
@@ -1315,7 +1315,8 @@ public void testTryDeleteDocument() throws Exception {
w.addDocument(doc);
w.close();
- iwc = new IndexWriterConfig(new MockAnalyzer(random()));
+ iwc = new IndexWriterConfig(new MockAnalyzer(random()))
+ .setMergePolicy(NoMergePolicy.INSTANCE);
Review Comment:
Note to reviewers: this test now triggers a merge because of deletes. Since the test is only about checking how deletes get applied, I simply disabled merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Run merge-on-full-flush even though no changes got flushed. [lucene]
Posted by "s1monw (via GitHub)" <gi...@apache.org>.
s1monw commented on PR #12549:
URL: https://github.com/apache/lucene/pull/12549#issuecomment-1778953836
I think this is only triggered because of your change but the problem was already there. We hold the lock in MDW#close() such that we can not run a concurrent merge. We could either prevent the merge with something like this:
```diff
index bc1f95e0b58..f9acdd55793 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/tests/store/MockDirectoryWrapper.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/tests/store/MockDirectoryWrapper.java
@@ -970,7 +970,10 @@ public class MockDirectoryWrapper extends BaseDirectoryWrapper {
DirectoryReader ir1 = DirectoryReader.open(this);
int numDocs1 = ir1.numDocs();
ir1.close();
- new IndexWriter(this, new IndexWriterConfig(null)).close();
+ new IndexWriter(this, new IndexWriterConfig(null)
+ .setCommitOnClose(false)
+ .setMaxFullFlushMergeWaitMillis(0))
+ .close();
DirectoryReader ir2 = DirectoryReader.open(this);
int numDocs2 = ir2.numDocs();
ir2.close();
```
or use a `SerialMergeScheduler` to run the merge in the caller thread. I would opt for the MergeScheduler. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org