You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Busch <bu...@gmail.com> on 2007/10/02 08:33:13 UTC
Exceptions in TestConcurrentMergeScheduler
Hi,
I noticed the following exception in TestConcurrentMergeScheduler:
[junit] Testsuite: org.apache.lucene.index.TestConcurrentMergeScheduler
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 17.765 sec
[junit] ------------- Standard Error -----------------
[junit] Exception in thread "Thread-113"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.NullPointerException
[junit] at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:263)
[junit] Caused by: java.lang.NullPointerException
[junit] at
org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:2744)
[junit] at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:238)
[junit] ------------- ---------------- ---------------
I ran the test again a couple of times, but didn't see the exception
again. Also, these exceptions don't let the testcase fail, so even if
the exceptions occur in the nightly build we probably won't notice it.
- Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Ning Li" <ni...@gmail.com> wrote:
> The cause is that in MergeThread.run(), merge in the try block is a
> local variable, while merge in the catch block is the class variable.
> Merge in the try block could be one different from the original merge,
> but the catch block always checks the abort flag of the original
> merge.
Aha! You are right :) Good catch.
I will fix. Thanks Ning!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Ning Li <ni...@gmail.com>.
The cause is that in MergeThread.run(), merge in the try block is a
local variable, while merge in the catch block is the class variable.
Merge in the try block could be one different from the original merge,
but the catch block always checks the abort flag of the original
merge.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael Busch <bu...@gmail.com>.
> I can't get this to happen. Which OS/hardware are you seeing this on?
>
I seeing it on my Laptop: 2.0 GHz Centrino, 2 GB RAM, Windows XP, Sun
JRE 1.5.0_06.
I tried the testcase a couple more times and it's approximately failing
in 1 to 2 runs out of 10.
> But it'd be nice to do this across the board, ie, for any junit test
> if one of CMS's threads (or, threads launched elsewhere) hits an
> unhandled exception, fail the testcase that's currently running.
> I'll dig and see if there's some central way to do this with junit...
>
Yeah, that'd be great!
- Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael McCandless <lu...@mikemccandless.com>.
Indeed this will work! I was unsure whether you could fail() a test
inside tearDown() but it seems to work fine -- I just tested it.
Only downside is a mass replacement of all "extends TestCase" with
"extends LuceneTestCase" (but that's a one time cost, now), and making
sure tearDown() always calls super.tearDown().
OK I like this solution better; I plan to go with that. I'll open a
separate issue for it. Thanks Erik :)
Mike
"Erik Hatcher" <er...@ehatchersolutions.com> wrote:
> Mike,
>
> Would it work to have a common LuceneTestCase base class that could
> do that check and fail() in tearDown?
>
> Erik
>
>
> On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
>
> >
> > OK, I think I found one possibility here. With ant's junit task, you
> > can define a custom formatter implementing this interface:
> >
> > org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
> >
> > That interface has a method endTestSuite that is invoked once at the
> > end of all the test cases. So I can define a customer formatter, and
> > in this method, I can check with ConcurrentMergeScheduler and if any
> > unhandled exceptions has occurred, I can throw an exception and the
> > suite/testcase is marked as failed. It seems to work.
> >
> > This is a nice solution in that we don't have to modify every unit
> > test to do its own checking. However, it's not really a "normal" use
> > case because formatters are supposed to just "format" the test result
> > output. It also adds a dependence from Lucene's unit test sources to
> > ant. But at least it does work ("progress not perfection").
> >
> > And objections to this approach? Is there a better approach?
> >
> > Mike
> >
> > "Michael McCandless" <lu...@mikemccandless.com> wrote:
> >>
> >> "Chris Hostetter" <ho...@fucit.org> wrote:
> >>>
> >>> : But it'd be nice to do this across the board, ie, for any junit
> >>> test
> >>> : if one of CMS's threads (or, threads launched elsewhere) hits an
> >>> : unhandled exception, fail the testcase that's currently running.
> >>> : I'll dig and see if there's some central way to do this with
> >>> junit...
> >>>
> >>> FYI: i did some casual investigation of this and the only thing that
> >>> jumped out at me is the static
> >>> Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
> >>> 1.4
> >>> there doesn't seem to be a generic way to notice an uncaught
> >>> exception
> >>> from any thread.
> >>
> >> Thanks Hoss.
> >>
> >> Catching the exception is actually not the hard part because I "own"
> >> all the threads spawned by ConcurrentMergeScheduler. What's
> >> tricky is
> >> finding a way to force the currently running JUnit testcase or suite
> >> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> >> to see if there's some hook somewhere where we could insert a check,
> >> just before the suite finishes, to assert that no exceptions were
> >> hit.
> >> Or, if I can somehow "look up" the current Test that's running, I
> >> could add an error to it.
> >>
> >> If there are any JUnit and ant experts out there (I'm not!) please
> >> chime in!
> >>
> >> Mike
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Mike,
Would it work to have a common LuceneTestCase base class that could
do that check and fail() in tearDown?
Erik
On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
>
> OK, I think I found one possibility here. With ant's junit task, you
> can define a custom formatter implementing this interface:
>
> org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
>
> That interface has a method endTestSuite that is invoked once at the
> end of all the test cases. So I can define a customer formatter, and
> in this method, I can check with ConcurrentMergeScheduler and if any
> unhandled exceptions has occurred, I can throw an exception and the
> suite/testcase is marked as failed. It seems to work.
>
> This is a nice solution in that we don't have to modify every unit
> test to do its own checking. However, it's not really a "normal" use
> case because formatters are supposed to just "format" the test result
> output. It also adds a dependence from Lucene's unit test sources to
> ant. But at least it does work ("progress not perfection").
>
> And objections to this approach? Is there a better approach?
>
> Mike
>
> "Michael McCandless" <lu...@mikemccandless.com> wrote:
>>
>> "Chris Hostetter" <ho...@fucit.org> wrote:
>>>
>>> : But it'd be nice to do this across the board, ie, for any junit
>>> test
>>> : if one of CMS's threads (or, threads launched elsewhere) hits an
>>> : unhandled exception, fail the testcase that's currently running.
>>> : I'll dig and see if there's some central way to do this with
>>> junit...
>>>
>>> FYI: i did some casual investigation of this and the only thing that
>>> jumped out at me is the static
>>> Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
>>> 1.4
>>> there doesn't seem to be a generic way to notice an uncaught
>>> exception
>>> from any thread.
>>
>> Thanks Hoss.
>>
>> Catching the exception is actually not the hard part because I "own"
>> all the threads spawned by ConcurrentMergeScheduler. What's
>> tricky is
>> finding a way to force the currently running JUnit testcase or suite
>> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
>> to see if there's some hook somewhere where we could insert a check,
>> just before the suite finishes, to assert that no exceptions were
>> hit.
>> Or, if I can somehow "look up" the current Test that's running, I
>> could add an error to it.
>>
>> If there are any JUnit and ant experts out there (I'm not!) please
>> chime in!
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK, I think I found one possibility here. With ant's junit task, you
can define a custom formatter implementing this interface:
org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
That interface has a method endTestSuite that is invoked once at the
end of all the test cases. So I can define a customer formatter, and
in this method, I can check with ConcurrentMergeScheduler and if any
unhandled exceptions has occurred, I can throw an exception and the
suite/testcase is marked as failed. It seems to work.
This is a nice solution in that we don't have to modify every unit
test to do its own checking. However, it's not really a "normal" use
case because formatters are supposed to just "format" the test result
output. It also adds a dependence from Lucene's unit test sources to
ant. But at least it does work ("progress not perfection").
And objections to this approach? Is there a better approach?
Mike
"Michael McCandless" <lu...@mikemccandless.com> wrote:
>
> "Chris Hostetter" <ho...@fucit.org> wrote:
> >
> > : But it'd be nice to do this across the board, ie, for any junit test
> > : if one of CMS's threads (or, threads launched elsewhere) hits an
> > : unhandled exception, fail the testcase that's currently running.
> > : I'll dig and see if there's some central way to do this with junit...
> >
> > FYI: i did some casual investigation of this and the only thing that
> > jumped out at me is the static
> > Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4
> > there doesn't seem to be a generic way to notice an uncaught exception
> > from any thread.
>
> Thanks Hoss.
>
> Catching the exception is actually not the hard part because I "own"
> all the threads spawned by ConcurrentMergeScheduler. What's tricky is
> finding a way to force the currently running JUnit testcase or suite
> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> to see if there's some hook somewhere where we could insert a check,
> just before the suite finishes, to assert that no exceptions were hit.
> Or, if I can somehow "look up" the current Test that's running, I
> could add an error to it.
>
> If there are any JUnit and ant experts out there (I'm not!) please
> chime in!
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Chris Hostetter" <ho...@fucit.org> wrote:
>
> : But it'd be nice to do this across the board, ie, for any junit test
> : if one of CMS's threads (or, threads launched elsewhere) hits an
> : unhandled exception, fail the testcase that's currently running.
> : I'll dig and see if there's some central way to do this with junit...
>
> FYI: i did some casual investigation of this and the only thing that
> jumped out at me is the static
> Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4
> there doesn't seem to be a generic way to notice an uncaught exception
> from any thread.
Thanks Hoss.
Catching the exception is actually not the hard part because I "own"
all the threads spawned by ConcurrentMergeScheduler. What's tricky is
finding a way to force the currently running JUnit testcase or suite
to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
to see if there's some hook somewhere where we could insert a check,
just before the suite finishes, to assert that no exceptions were hit.
Or, if I can somehow "look up" the current Test that's running, I
could add an error to it.
If there are any JUnit and ant experts out there (I'm not!) please
chime in!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Chris Hostetter <ho...@fucit.org>.
: But it'd be nice to do this across the board, ie, for any junit test
: if one of CMS's threads (or, threads launched elsewhere) hits an
: unhandled exception, fail the testcase that's currently running.
: I'll dig and see if there's some central way to do this with junit...
FYI: i did some casual investigation of this and the only thing that
jumped out at me is the static
Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4
there doesn't seem to be a generic way to notice an uncaught exception
from any thread.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Exceptions in TestConcurrentMergeScheduler
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Michael Busch" <bu...@gmail.com> wrote:
> Hi,
>
> I noticed the following exception in TestConcurrentMergeScheduler:
>
> [junit] Testsuite: org.apache.lucene.index.TestConcurrentMergeScheduler
> [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 17.765
> sec
> [junit] ------------- Standard Error -----------------
> [junit] Exception in thread "Thread-113"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.NullPointerException
> [junit] at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:263)
> [junit] Caused by: java.lang.NullPointerException
> [junit] at
> org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:2744)
> [junit] at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:238)
> [junit] ------------- ---------------- ---------------
I can't get this to happen. Which OS/hardware are you seeing this on?
It looks like there is a concurrent merge that tried to initialize
itself after its writer had closed. This is likely during the
"testNoWaitClose" testcase which does iterations where it closes
writer without waiting (forcing an abort of all running merges). I
think somehow a merge is getting created but not properly aborted...
I will try to find it.
> I ran the test again a couple of times, but didn't see the exception
> again. Also, these exceptions don't let the testcase fail, so even if
> the exceptions occur in the nightly build we probably won't notice it.
Good point. I can definitely add checking that no exceptions occurred
in these test cases.
But it'd be nice to do this across the board, ie, for any junit test
if one of CMS's threads (or, threads launched elsewhere) hits an
unhandled exception, fail the testcase that's currently running.
I'll dig and see if there's some central way to do this with junit...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org