You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2010/08/24 14:33:16 UTC
[jira] Created: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Intermittent failure in 3.x's backwards TestThreadedOptimize
------------------------------------------------------------
Key: LUCENE-2618
URL: https://issues.apache.org/jira/browse/LUCENE-2618
Project: Lucene - Java
Issue Type: Bug
Components: Index
Reporter: Michael McCandless
Fix For: 3.1, 4.0
Failure looks like this:
{noformat}
[junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
[junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError: null
[junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
[junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
{noformat}
I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2618.
----------------------------------------
Resolution: Fixed
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924150#action_12924150 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
OK Mike .I understood the sequence of operations that led to this exception before. What didn't add up is why is it thrown during optimize, and not say up front when IW is opened, or when the Directory was added through addIndexes.
We should fix the code to throw the exception immediately. Is there a way to check a Directory if it's old or not? If not, such exception could really throw you off your chair, when you hit it at a point in time not remotely related to when it was added to the index.
I don't mind if you continue w/ the fix to the test as you did, but IMO it just hides the real problem. I.e., allowing all merges caused by optimize() to finish is a correct fix. But catching that exception upon IW.close() is a bad one IMO - people who read the code learn how to use Lucene, and catching that exception on close() makes absolutely no sense, at least to me. Could you plz add a TODO there to get rid of that code when we fix IW to detect old indexes up front? That way, if someone reads the code, he'll at least understand that this is a temporary solution.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923761#action_12923761 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
Ok - I agree maybeMerge is probably less frequently called than optimize. And perhaps we can look at it that way: when you call optimize, you know exactly what to expect. You control the # of final segments. When you call maybeMerge lucene does not guarantee the final result. Unless you know exactly the state of all the segments in the index (which except than from unit tests I think it's very unlikely) and exactly what your MP is doing, you cannot expect any guaranteed outcome from calling maybeMerge, except for it "doing the best effort".
What bothered me is that even if maybeMerge and optimize may go through several levels of merging following one call to them, one is guaranteed to complete and the other isn't. But since optimize is more common in apps than the other, I'm willing to make that exception. Perhaps then add to maybeMerge docs that if you want to guarantee merges finish when close is called, you should wait for merges? Or should we add it to close?
I'm fine now with this fix. +1 to commit.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923989#action_12923989 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
Does this mean I'll need to catch that exception every time I close an IW, or at least prepare to catch it? If so, shouldn't we document it? Is it only relevant to the test?
Somehow this change / fix starts to get complicated. Can IW swallow those exceptions internally, and relieve the application from all this? When I close(false), I should be prepared to hit MergeAbortedException, it's kinda part of the API contract. But when I close(true), why do I need to be prepared to handle any exception, except for real IO ones?
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923579#action_12923579 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
I don't personally mind either way. Just want to point out that calling maybeMerge is as explicit as calling optimize. You can argue for both that if an app wants to wait for merges it can call waitForMerges. In fact, an app calling close() already stated it wants to wait for merges - it's as if it called waitForMerges followed by close.
I think you're trying to distinguish merges that started because the MP decided they should run following a certain commit to those triggered by explicit call to optimize. So IMO maybeMerge and optimize are the same as both were explicitly initiated by the application.
This test fails because it assumes optimize will run to completion. What if the test assumed maybeMerge runs to completion? Isn't that a valid expectation from an application calling close()? We're also distinguishing the first round of merges from subsequent rounds, only when maybeMerge is called, but not optimize...
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923564#action_12923564 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
We do allow all running merges to run to completion.
But, we don't allow new merges to start, unless it's part of an ongoing optimize (as of this patch).
I think this distinction makes sense? Since optimize was an explicit call, it should run until completion. But merging can simply pick up the next time the index is opened?
If an app really wants to allow all merges to run before closing (even new ones starting) it can call waitForMerges and then close.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902550#action_12902550 ]
Mark Miller commented on LUCENE-2618:
-------------------------------------
I'm catching something similar on current tests I think:
{code}
[junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
[junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
[junit] expected:<248> but was:<256>
[junit] junit.framework.AssertionFailedError: expected:<248> but was:<256>
[junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:119)
[junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:142)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:380)
[junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:372)
[junit]
[junit]
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.733 sec
{code}
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2618:
---------------------------------------
Attachment: LUCENE-2618.patch
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924008#action_12924008 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
I think there's a separate issue open (Uwe?) to have IW immediately throw this exc on open, instead of during optimize/close.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924298#action_12924298 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
Ugh -- last night's 3.x build just failed again! So this was not the [only] cause. Hmm. I'll leave this reopened....
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923430#action_12923430 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
Ok Mike, that makes sense. You want to allow optimize() to finish all possible merges. Why then not let regular merges finish all the way through, even if we're closing? I mean, the application wants to wait for all running merges, so why is optimize() different than maybeMerge()?
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924167#action_12924167 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
Thanks Mike.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923995#action_12923995 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
bq. Does this mean I'll need to catch that exception every time I close an IW, or at least prepare to catch it?
Well, IndexFormatTooOldExc subclasses IOE... but, yes, if there's a risk you'll open a too-old index, you should try to handle this.
IW.close does alot... flush the last segment, let MS run any pending merges, do commit, delete now-not-need files, etc. So there's plenty of chances for interested excs.
bq. Is it only relevant to the test?
Well, this test opens an IW on a too-old index... so if your app may do that....
bq. Can IW swallow those exceptions internally, and relieve the application from all this?
Whoa no way!
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902018#action_12902018 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
I'm guessing that this line fails (which is 126 in my most recent checkout):
{code}
assertTrue(reader.isOptimized());
{code}
Is this the one that's pointed by your code Mike? If so, I suggest we include a message to the assertion, something like "index should be optimized". It's annoying that JUnit does not print "should be true but was false", or something like that, and instead prints 'null', which is more intimidating :).
Perhaps we should also add some more info to the print, like the number of segments in and index and whether there are deletions, so we'd have a better clue why the test failed?
I've tried to run the test a couple of times, but it passed ...
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924164#action_12924164 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
OK I opened LUCENE-2720.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924006#action_12924006 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
bq. If my app knowingly opened a too old index, would it get this exception always, if it will call optimize followed by close? Or is it a special scenario hit by the test?
Not always. It's only if the MP registered more than 1 merge for the optimize, and, you're using SMS.
But, really if your app has risk of opening a too-old index, it should be prepared for this exc...
bq. namely, what's the connection between optimize + close and an old index?
MP enrolled 2 merges for the optimize... the first one hits exc... then test calls close... and close lets MS run... and MS is SMS... and it runs the 2nd merge, which this the exc.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923428#action_12923428 ]
Robert Muir commented on LUCENE-2618:
-------------------------------------
thanks for tracking this down...!
I think if we fix this one, then we are really into the long tail of random test fails (at least for now)
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923866#action_12923866 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
OK thanks Shai... I'll commit shortly.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923946#action_12923946 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
Hmm.... indeed you can repro with:
{noformat}ant test-core -Dtestcase=TestBackwardsCompatibility -Dtestmethod=testUnsupportedOldIndexes -Dtests.seed=-7202471693621265890:9015568443891620555{noformat}
I'll revert until I can figure this out... sorry!
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923426#action_12923426 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
{quote}
If close(false) is called after optimize() was called, it means the app would like to abort merges ASAP. If so, why would we consult the MP if we're instructed to abort?
Are you talking about a different use case?
{quote}
Sorry, different use case.
This use case is you call .optimize(doWait=false) then you call a normal .close() (ie, wait for merges). In this case we wait for all running merges to finish, but don't start any new ones. My patch would still allow new ones to start if the merges are due to a running optimize.
Your use case, where .close(false) is called, will in fact abort all running merges and close quickly. Ie we will not start new merges, even for optimize, if you pass false to close, with this pattch.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reopened LUCENE-2618:
-----------------------------------
This commit sometimes fails TestBackwards because IndexWriter.close() now also throws IndexFormatTooOldException if the previous call to optimize() have thrown it already.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902023#action_12902023 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
Sorry, I've missed the part about this happening in backwards tests. The line numbers match for me, and I see the assertion messages. I do think though that additional info such as the number of segments and deleted docs would be useful, since reader.isOptimize() will return false if either of these two is wrong.
And we can add the same message to the regular test as well ...
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923395#action_12923395 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
For education purposes - why should we consult the MP if it's an optimize, even while closing? If close(false) is called after optimize() was called, it means the app would like to abort merges ASAP. If so, why would we consult the MP if we're instructed to abort?
Are you talking about a different use case?
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924002#action_12924002 ]
Shai Erera commented on LUCENE-2618:
------------------------------------
I see. It's just that you describe this exception as being thrown because close is called while optimize was running over an old index - but I don't understand why it has to be thrown in this case - namely, what's the connection between optimize + close and an old index? If my app knowingly opened a too old index, would it get this exception always, if it will call optimize followed by close? Or is it a special scenario hit by the test?
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2618:
---------------------------------------
Attachment: LUCENE-2618.patch
I think I found this!
After a merge completes, IW then checks w/ the merge policy to see if followon merges are now necessary.
But this check is skipped if IW.close is pending (ie has been called and is waiting for merges to complete).
However, if that merge is an optimize, then we should in fact consult the merge policy even when a close is pending, which we are not doing today.
Tiny patch (attached) should fix it.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924163#action_12924163 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
bq. We should fix the code to throw the exception immediately. Is there a way to check a Directory if it's old or not?
I agree -- IW.open should fail immediately if any of the segments are too old.
Unfortunately, I don't see a simple way to do this. We can't just look at the version of the segments_N file, for example, because one segment could be from 2.9, and [say] 3.1 had last opened the index and written the 3.x file format for segments_N. See, IW does not go and open all SegmentReaders on open. It's only on merge, applying deletes, or opening an NRT reader, that we go and open segments for reading.
I think to do this correctly we should modify segments_N format to record the oldest segment in the index? Then IW can check this easily on open.
bq. I don't mind if you continue w/ the fix to the test as you did, but IMO it just hides the real problem. I.e., allowing all merges caused by optimize() to finish is a correct fix.
I agree.
There is already a pre-existing TODO in the test stating that we should fix IW to throw this exc on open. I'll also add a TODO to IW's ctor and go open an issue...
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923987#action_12923987 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
OK so I think we should fix this test to also accept an IndexTooOldExc during close.
The .optimize() call for only the 29.nocfs case (for some reason) enrolls 2 pending merges to IW.
The 1st merge hits an exception, throwing up through the .optimize() to the test. But the 2nd merge remains queued, and in IW.close() we give MS a chance to run any merges it needs to, and that 2nd merge then also hits an exc.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's
backwards TestThreadedOptimize
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923665#action_12923665 ]
Michael McCandless commented on LUCENE-2618:
--------------------------------------------
bq. Just want to point out that calling maybeMerge is as explicit as calling optimize.
But: apps don't normally call maybeMerge? This is typically called within IW, eg on segment flush.
I mean, it is public so apps can call it, but I expect very few do (vs optimize which apps use alot). It's the exception not the rule...
I guess I feel that close should try to close quickly -- an app would not expect close to randomly take a long time (it's already bad enough since a large merge could be in process...). So, allowing other merges to start up, which could easily be large merges since they are follow-on ones, would make that worse.
Alternatively, we could define the semantics of close as being allowed to prevent a running optimize from actually completing? Then we'd have to change this test, eg to call .waitForMerges before close.
> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
> Key: LUCENE-2618
> URL: https://issues.apache.org/jira/browse/LUCENE-2618
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
> [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
> [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
> [junit] null
> [junit] junit.framework.AssertionFailedError: null
> [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
> [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
> [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org