You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Daniel John Debrunner <dj...@apache.org> on 2006/10/17 18:39:22 UTC
Some JUnit tests now split out from old harness and do not run as
part of derbyall
As discussed and recorded in DERBY-1952 I have split some of the JUnit
tests out so that they are *no longer* run under the old test harness
and hence are *not* part of derbyall. (on the trunk).
A top-level JUnit test (suite) exists that runs all the tests that were
removed from derbyall, the test's name is:
org.apache.derbyTesting.functionTests.suites.All
JUnit tests will continue to be added so that they run under this All
test and thus be removed from the old harness.
Instructions on how to run this Junit test are in
http://wiki.apache.org/db-derby/DerbyJUnitTesting
with notes on the configuration setup and how to write suite() methods
in http://wiki.apache.org/db-derby/DerbyJunitTestConfiguration
I've been updating the wiki so that I think it is up to date with the
latest changes.
I would encourage those that regularly run nightly or other testing to
add this JUnit test into their framework. I'm sure there will be some
issues getting it running on all the platforms but they should be easy
to work through.
There is more work to be done for some of the tests in All so that they
also run in the client server by themselves, and I'm working on it for
the tests in tests.jdbcapi.
I'm also planning on laying out some form of a plan or work items that
would move us completely over to JUnit.
Dan.
PS. The current runtime for this suite is a few minutes.
Re: Some JUnit tests now split out from old harness and do not run as
part of derbyall
Posted by Dy...@Sun.COM.
Daniel John Debrunner <dj...@apache.org> writes:
> As discussed and recorded in DERBY-1952 I have split some of the JUnit
> tests out so that they are *no longer* run under the old test harness
> and hence are *not* part of derbyall. (on the trunk).
>
> A top-level JUnit test (suite) exists that runs all the tests that
> were removed from derbyall, the test's name is:
>
> org.apache.derbyTesting.functionTests.suites.All
[snip]
When running this suite with swingui.TestRunner -noloading all tests passed,
but the TestRunner throws an exception when it is shut down (because
it is not allowed to save the history?):
Exception in thread "AWT-EventQueue-0" java.security.AccessControlException: access denied (java.io.FilePermission /home/dt136804/.junitsession write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.SecurityManager.checkWrite(SecurityManager.java:962)
at java.io.FileOutputStream.<init>(FileOutputStream.java:169)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at java.io.FileWriter.<init>(FileWriter.java:73)
at junit.swingui.TestRunner.saveHistory(TestRunner.java:648)
at junit.swingui.TestRunner.terminate(TestRunner.java:745)
at junit.swingui.TestRunner$8.actionPerformed(TestRunner.java:270)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6038)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3260)
at java.awt.Component.processEvent(Component.java:5803)
at java.awt.Container.processEvent(Container.java:2058)
at java.awt.Component.dispatchEventImpl(Component.java:4410)
at java.awt.Container.dispatchEventImpl(Container.java:2116)
at java.awt.Component.dispatchEvent(Component.java:4240)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4322)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3986)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3916)
at java.awt.Container.dispatchEventImpl(Container.java:2102)
at java.awt.Window.dispatchEventImpl(Window.java:2429)
at java.awt.Component.dispatchEvent(Component.java:4240)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:273)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:183)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:173)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:168)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:160)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:121)
--
dt
Re: Some JUnit tests now split out from old harness and do not run as
part of derbyall
Posted by Dy...@Sun.COM.
Vemund Ostgaard <Ve...@Sun.COM> writes:
> Daniel John Debrunner wrote:
>
>>
>> I would encourage those that regularly run nightly or other testing
>> to add this JUnit test into their framework. I'm sure there will be
>> some issues getting it running on all the platforms but they should
>> be easy to work through.
>
> I'm extending the nightly testing with this top-level junit suite,
> executed with junit.textui.TestRunner, from tonight. If it turns out
> to run ok on all platforms we will start reporting from this suite as
> well in the mails from Henri.
>
>>
>> There is more work to be done for some of the tests in All so that
>> they also run in the client server by themselves, and I'm working on
>> it for the tests in tests.jdbcapi.
>>
>> I'm also planning on laying out some form of a plan or work items
>> that would move us completely over to JUnit.
>
> When running the top-level suite now, the only output I got was a lot
> of dots, the time it took to run the suite and the number of OK tests
> run.
>
> Have anyone considered a framework or interface for logging events and
> information in the junit tests? I think it will be very difficult in
> the future to analyse test failures only based on the text from an
> assertion failure or exception. Especially when we start porting
> multhreaded stress tests and other more complicated parts of the old
> suite.
>
> It would also give a warm fussy feeling to be able to see that the
> tests actually did something (in addition to all the dots written by
> the TestRunner).
>
> I've been used to using the java logging API (java.util.logging.*) for
> similar needs, and think that would cover what is needed for this.
>
> Other suggestions or views on this?
Have you looked at generating html reports via ant? See
http://junit.sourceforge.net/doc/faq/faq.htm#running_6
I haven't tried this though...
--
dt
Re: JUnit disagnostice (Was Re: Some JUnit tests now split out from
old harness and do not run as part of derbyall)
Posted by Daniel John Debrunner <dj...@apache.org>.
Daniel John Debrunner wrote:
> Øystein Grøvlen wrote:
>> Vemund Ostgaard wrote:
>>
>>> When running the top-level suite now, the only output I got was a lot
>>> of dots, the time it took to run the suite and the number of OK tests
>>> run.
The Eclipse JUnit test runner is similar to the swing one but better.
http://open.ncsu.edu/se/tutorials/junit/
It correctly shows which tests have run (the swing test runner seems to
have bugs in it) and dynamically indicates which test is currently
running. It also shows the decorators and allows one to go to the source
file from an entry (e.g. a text fixture method) in the test hierarchy.
Dan.
Re: JUnit disagnostice (Was Re: Some JUnit tests now split out from
old harness and do not run as part of derbyall)
Posted by Vemund Ostgaard <Ve...@Sun.COM>.
Daniel John Debrunner wrote:
>> Vemund Ostgaard wrote:
>>
>>> When running the top-level suite now, the only output I got was a
>>> lot of dots, the time it took to run the suite and the number of OK
>>> tests run.
>>>
>>> Have anyone considered a framework or interface for logging events
>>> and information in the junit tests? I think it will be very
>>> difficult in the future to analyse test failures only based on the
>>> text from an assertion failure or exception. Especially when we
>>> start porting multhreaded stress tests and other more complicated
>>> parts of the old suite.
>>
<snip>
> The Swing test runner shows more information about which tests have
> run and passed. Remember the textui test runner is just a simple test
> runner.
>
> I would encourage investigation of what others have done in this area,
> see if there are better test runners, rather than invent a new
> mechanism. I would like that the Derby JUnit tests continue to run as
> other Junit tests so others can run & integrate them up easily. If the
> default running of Derby's JUnit tests produce a lot of output that
> will confuse others who are used to the model where a test produces no
> output if it succeeds.
The default behaviour could be to write nothing if that is what we want,
while still making it possible to get log from the tests for those who
want that. With for instance the logging API that I suggest the
logstatements will be in the code but wether the log actually gets
printed anywhere (console or file) can be controlled dynamically with
properties. Different runners can give more detailed information on the
exact tests that were run, but no runners will give us insight into what
each test actually did.
I very much agree that the tests should run as other JUnit tests and not
be confusing when others want to integrate them in junit-compatible
environments.
I think the type of testsuite you have and the breadth of test execution
you do influences the need for logging in the tests. If you have a
testuite of "unit like" tests that just do a single method call into the
product (as JUnit was intended for), the exception thrown from a test
will probably give you the information you need to be able to analyze
what happened. If you have a testsuite of many
complex/long/multithreaded tests run within a variety of configurations
and decorators, errors can be difficult to analyze and you will more
often have to put som debug in the test and rerun to get what you need.
Tests are run nightly on several platforms and jvms. When the junit
suite grows to a size comparable with the old testsuite and beyond,
there will be problems to analyze every night. If these problems are
accompanied with a sensible amount of log it will ease the work of the
analyzer, who often will not be the original author of the test. The
tests will not be perfect, there will be situations that should lead to
an exception being thrown that doesn't, but instead create difficulties
for following tests. These issues will also be very costly to track down
without a better view into what each test actually did.
A last issue is that when you make changes to a test, it is good to get
some verification that the changes actually had the desired effect.
Being able to take a look at the log from the nightly run of tests and
verify that the output from your fix got printed is a nice way of doing
just that. We all develop on different systems so even when it worked on
yours it might not have the same effect on the one its failing on in the
nightlies. You may not even be able to reproduce the problem on the
machines you have access to and depend on getting more output when it
happens, perhaps intermittently) in others testing.
If you think about the old harness, how often have you made use of
information from the testlogs that you wouldn't have gotten from just an
exception thrown?
Vemund
Re: JUnit disagnostice (Was Re: Some JUnit tests now split out from
old harness and do not run as part of derbyall)
Posted by Daniel John Debrunner <dj...@apache.org>.
Øystein Grøvlen wrote:
> Vemund Ostgaard wrote:
>
>> When running the top-level suite now, the only output I got was a lot
>> of dots, the time it took to run the suite and the number of OK tests
>> run.
>>
>> Have anyone considered a framework or interface for logging events and
>> information in the junit tests? I think it will be very difficult in
>> the future to analyse test failures only based on the text from an
>> assertion failure or exception. Especially when we start porting
>> multhreaded stress tests and other more complicated parts of the old
>> suite.
>
> I totally agree with you. I had an experience when porting
> LobLengthTest to junit that illustrate this. When I had the test to
> jdbcapi/_Suite, OOM errors occurred and the test hanged when I ran it
> with the DerbyNetClient. There was no practical way to find out which
> test was hanging. I could have counted dots and tried to determine
> which test case that represented, but that would have been both error
> prone and very time consuming. (The problem went away when Fernanda
> checked in the clean-up to the SUR-test.)
>
>>
>> It would also give a warm fussy feeling to be able to see that the
>> tests actually did something (in addition to all the dots written by
>> the TestRunner).
>
> I agree on this too. The way it is today, if I by mistake disable some
> test, this is very difficult to detect.
The Swing test runner shows more information about which tests have run
and passed. Remember the textui test runner is just a simple test runner.
I would encourage investigation of what others have done in this area,
see if there are better test runners, rather than invent a new
mechanism. I would like that the Derby JUnit tests continue to run as
other Junit tests so others can run & integrate them up easily. If the
default running of Derby's JUnit tests produce a lot of output that will
confuse others who are used to the model where a test produces no output
if it succeeds.
Dan.
JUnit disagnostice (Was Re: Some JUnit tests now split out from old
harness and do not run as part of derbyall)
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Vemund Ostgaard wrote:
> When running the top-level suite now, the only output I got was a lot of
> dots, the time it took to run the suite and the number of OK tests run.
>
> Have anyone considered a framework or interface for logging events and
> information in the junit tests? I think it will be very difficult in the
> future to analyse test failures only based on the text from an assertion
> failure or exception. Especially when we start porting multhreaded
> stress tests and other more complicated parts of the old suite.
I totally agree with you. I had an experience when porting
LobLengthTest to junit that illustrate this. When I had the test to
jdbcapi/_Suite, OOM errors occurred and the test hanged when I ran it
with the DerbyNetClient. There was no practical way to find out which
test was hanging. I could have counted dots and tried to determine
which test case that represented, but that would have been both error
prone and very time consuming. (The problem went away when Fernanda
checked in the clean-up to the SUR-test.)
>
> It would also give a warm fussy feeling to be able to see that the tests
> actually did something (in addition to all the dots written by the
> TestRunner).
I agree on this too. The way it is today, if I by mistake disable some
test, this is very difficult to detect.
--
Øystein
Re: Some JUnit tests now split out from old harness and do not run as
part of derbyall
Posted by Vemund Ostgaard <Ve...@Sun.COM>.
Daniel John Debrunner wrote:
>
> I would encourage those that regularly run nightly or other testing to
> add this JUnit test into their framework. I'm sure there will be some
> issues getting it running on all the platforms but they should be easy
> to work through.
I'm extending the nightly testing with this top-level junit suite,
executed with junit.textui.TestRunner, from tonight. If it turns out to
run ok on all platforms we will start reporting from this suite as well
in the mails from Henri.
>
> There is more work to be done for some of the tests in All so that
> they also run in the client server by themselves, and I'm working on
> it for the tests in tests.jdbcapi.
>
> I'm also planning on laying out some form of a plan or work items that
> would move us completely over to JUnit.
When running the top-level suite now, the only output I got was a lot of
dots, the time it took to run the suite and the number of OK tests run.
Have anyone considered a framework or interface for logging events and
information in the junit tests? I think it will be very difficult in the
future to analyse test failures only based on the text from an assertion
failure or exception. Especially when we start porting multhreaded
stress tests and other more complicated parts of the old suite.
It would also give a warm fussy feeling to be able to see that the tests
actually did something (in addition to all the dots written by the
TestRunner).
I've been used to using the java logging API (java.util.logging.*) for
similar needs, and think that would cover what is needed for this.
Other suggestions or views on this?
Vemund