You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Apache Hudson Server <hu...@hudson.apache.org> on 2010/11/03 10:31:29 UTC
Build failed in Hudson: River-trunk-QA #50
See <https://hudson.apache.org/hudson/job/River-trunk-QA/50/changes>
Changes:
[sijskes] added license
[sijskes] fixed exit value, add soulfile as property as well
[sijskes] added qa-build target
[sijskes] added permission
[sijskes] disabled heart for now
------------------------------------------
[...truncated 15683 lines...]
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeIfExistsTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeIfExistsWaitTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeNO_WAITTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeReadTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeWaitTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteLeaseANYTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteLeaseFOREVERTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteNegativeLeaseTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeIfExistsNotifyTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeIfExistsTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeNotifyTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteLeaseANYTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteLeaseFOREVERTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteNegativeLeaseTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/AdminIFShutdownTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/AdminIFTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/LeaseExpireCancelTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/LeaseExpireRenewTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/LeaseMapTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/LeaseTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/MahaloCreateShutdownTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/MahaloIFTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/MahaloImplReadyStateTest.td
[java] Test Skipped: verifiers are: com.sun.jini.test.impl.mercury.ActivatableMercuryVerifier com.sun.jini.qa.harness.SkipConfigTestVerifier
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/NestableServerTransactionCreatedToStringTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/NestableTransactionCreatedToStringTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/PrepareAndCommitExceptionTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/PrepareAndCommitExceptionTest2.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/PrepareAndCommitExceptionTest3.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/PrepareAndCommitExceptionTest4.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/PrepareAndCommitExceptionTest5.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/RandomStressTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/ServerTransactionEqualityTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/ServerTransactionToStringTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TransactionCreatedToStringTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TransactionManagerCreatedToStringTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TxnMgrImplNullActivationConfigEntries.td
[java] Test Skipped: verifiers are: com.sun.jini.test.impl.mahalo.ActivatableMahaloVerifier
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TxnMgrImplNullConfigEntries.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TxnMgrImplNullRecoveredLocators.td
[java] Test Skipped: verifiers are: com.sun.jini.test.impl.mahalo.ActivatableMahaloVerifier
[java] -----------------------------------------
[java] com/sun/jini/test/impl/mahalo/TxnMgrProxyEqualityTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/AsynchAbortOnCommitTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/AsynchAbortOnPrepareTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/CommitExpiredTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/CommitTimeoutTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/GetStateTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/JoinIdempotentTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/JoinWhileActiveTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/ManyParticipantsTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/PrepareTimeoutTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/RollBackErrorTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/RollForwardErrorTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java] com/sun/jini/test/spec/txnmanager/TwoPhaseTest.td
[java] Test Passed: OK
[java]
[java] -----------------------------------------
[java]
[java] # of tests started = 1409
[java] # of tests completed = 1409
[java] # of tests skipped = 47
[java] # of tests passed = 1404
[java] # of tests failed = 5
[java]
[java] -----------------------------------------
[java]
[java] Date finished:
[java] Wed Nov 03 09:29:27 UTC 2010
[java] Time elapsed:
[java] 76319 seconds
[java]
[java] Java Result: 1
collect-result:
[copy] Copying 1 file to <https://hudson.apache.org/hudson/job/River-trunk-QA/ws/jtsk/trunk/qa/result>
[copy] Copying 1 file to <https://hudson.apache.org/hudson/job/River-trunk-QA/ws/jtsk/trunk/qa/result>
[zip] Building zip: <https://hudson.apache.org/hudson/job/River-trunk-QA/50/artifact/jtsk/trunk/qa/result/qaresults-i386-Linux-1.6.0_20.zip>
BUILD FAILED
<https://hudson.apache.org/hudson/job/River-trunk-QA/ws/jtsk/trunk/build.xml>:2105: The following error occurred while executing this line:
<https://hudson.apache.org/hudson/job/River-trunk-QA/ws/jtsk/trunk/qa/build.xml>:337: The following error occurred while executing this line:
<https://hudson.apache.org/hudson/job/River-trunk-QA/ws/jtsk/trunk/qa/build.xml>:311: condition satisfied
Total time: 1277 minutes 12 seconds
Publishing Javadoc
Archiving artifacts
Re: Build failed in Hudson: River-trunk-QA #50
Posted by Sim IJskes - QCG <si...@qcg.nl>.
On 03-11-10 10:31, Apache Hudson Server wrote:
> See<https://hudson.apache.org/hudson/job/River-trunk-QA/50/changes>
> [java] # of tests failed = 5
2 of these were because of the policy file syntax error.
Gr. Sim
--
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Jonathan Costers <jo...@googlemail.com>.
>
> Some of them may duplicate existing QA or JUnit tests, so there are three
> end points for a given jtreg test: drop it, convert it to QA, or convert it
> to JUnit. In some cases "convert" may mean write a new test that fits in one
> of the preferred frameworks and tests the same issues as the jtreg test.
>
>
This is what I was aiming at, yes. This effort could also be done for many
of the current QA tests, that dont really need the QA infrastructure. Those
could be moved to JUnit (tests in the "id" and "config" categories seem good
candidates).
> Any thoughts on how to organize this effort? A hundred separate JIRA issues
> seems excessive. Maybe one JIRA and use comments to say which tests we are
> working on?
>
One JIRA issue seems appropriate. But I believe there are some other JIRA
issues open regarding jtreg already.
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Patricia Shanahan <pa...@acm.org>.
Jonathan Costers wrote:
...
> Maybe we can migrate the valuable tests in that jtreg suite to either JUnit
> or QA tests? I am aware there are caveats (test isolation level for
> instance), but they seem to be manageable. I'm talking about a gradual
> process here, converting test after test over a long period of time. We are
> talking about around 100 jtreg tests, with varying complexity and isolation
> levels.
...
+1 on the general concept, subject to some experimentation.
Some of them may duplicate existing QA or JUnit tests, so there are
three end points for a given jtreg test: drop it, convert it to QA, or
convert it to JUnit. In some cases "convert" may mean write a new test
that fits in one of the preferred frameworks and tests the same issues
as the jtreg test.
Any thoughts on how to organize this effort? A hundred separate JIRA
issues seems excessive. Maybe one JIRA and use comments to say which
tests we are working on?
Also, perhaps we could set up a Wiki page or similar to share any ideas
and insights on questions such as how to decide whether to convert to QA
or JUnit, and how to go about the conversion?
Patricia
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Peter Firmstone <ji...@zeus.net.au>.
Sim IJskes - QCG wrote:
> On 11/06/2010 11:29 AM, Patricia Shanahan wrote:
>> If I were doing this sort of test migration, I would test the test by
>> modifying the code under test to introduce the bug it is testing for,
>> and consider the operation complete only when the test detects and
>> reports it.
>
> It might well be the only way to do it. Unless only the framework is
> changed. We could go for a reporting backend that reports in junit
> style? I've never worked with jtreg, is it cumbersome to use?
>
> Gr. Sim
>
>
Actually it's quite ok, just different, it has some good features, like
multi jvm, policy's and detailed html reports. It has more similarities
to the qa suite than junit. Junit's focus is for testing a single
object or class. The good thing about jtreg is that it's maintained by
someone else, as opposed to our home grown qa suite, unique to Jini.
The issue we have is the multitude of different test methods, however
that might be a blessing also, it gives wider test coverage. If we have
everything running from ant (provided it works on all platforms,
including Windows), you don't have to understand the test suite to run
the tests, only to write them.
Cheers,
Peter.
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Sim IJskes - QCG <si...@qcg.nl>.
On 11/06/2010 11:29 AM, Patricia Shanahan wrote:
> If I were doing this sort of test migration, I would test the test by
> modifying the code under test to introduce the bug it is testing for,
> and consider the operation complete only when the test detects and
> reports it.
It might well be the only way to do it. Unless only the framework is
changed. We could go for a reporting backend that reports in junit
style? I've never worked with jtreg, is it cumbersome to use?
Gr. Sim
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Peter Firmstone <ji...@zeus.net.au>.
Patricia Shanahan wrote:
> On 11/5/2010 10:59 PM, Peter Firmstone wrote:
>> Jonathan Costers wrote:
> ...
>>> Maybe we can migrate the valuable tests in that jtreg suite to either
>>> JUnit
>>> or QA tests? I am aware there are caveats (test isolation level for
>>> instance), but they seem to be manageable. I'm talking about a gradual
>>> process here, converting test after test over a long period of time.
>>> We are
>>> talking about around 100 jtreg tests, with varying complexity and
>>> isolation
>>> levels.
>>
>> Well I don't think Junit would be suitable, since multiple jvm's are
>> employed. I've also had to modify some jtreg tests that made some
>> assumptions about ClassLoader visibility (jre/lib/ext related) and
>> failed later when we made some changes. The qa suite might be suitable,
>> but I don't think the effort's worthwhile, we can't guarantee that we're
>> simulating the failure conditions properly, even if only a few of us run
>> these tests, it's better than everyone running them if they're not
>> genuinely simulating failure conditions.
> ...
>
> Yes, needing multiple JVMs seems like a good reason for using the QA
> framework.
>
> If I were doing this sort of test migration, I would test the test by
> modifying the code under test to introduce the bug it is testing for,
> and consider the operation complete only when the test detects and
> reports it.
>
> Patricia
>
Hmm, yes that would be the only way to do it.
Cheers,
Peter.
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Patricia Shanahan <pa...@acm.org>.
On 11/5/2010 10:59 PM, Peter Firmstone wrote:
> Jonathan Costers wrote:
...
>> Maybe we can migrate the valuable tests in that jtreg suite to either
>> JUnit
>> or QA tests? I am aware there are caveats (test isolation level for
>> instance), but they seem to be manageable. I'm talking about a gradual
>> process here, converting test after test over a long period of time.
>> We are
>> talking about around 100 jtreg tests, with varying complexity and
>> isolation
>> levels.
>
> Well I don't think Junit would be suitable, since multiple jvm's are
> employed. I've also had to modify some jtreg tests that made some
> assumptions about ClassLoader visibility (jre/lib/ext related) and
> failed later when we made some changes. The qa suite might be suitable,
> but I don't think the effort's worthwhile, we can't guarantee that we're
> simulating the failure conditions properly, even if only a few of us run
> these tests, it's better than everyone running them if they're not
> genuinely simulating failure conditions.
...
Yes, needing multiple JVMs seems like a good reason for using the QA
framework.
If I were doing this sort of test migration, I would test the test by
modifying the code under test to introduce the bug it is testing for,
and consider the operation complete only when the test detects and
reports it.
Patricia
Re: test framework migration
Posted by Zsolt Kúti <la...@gmail.com>.
On Mon, 15 Nov 2010 12:38:06 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> While useful, I recommend against that feature in general.
> Philosophically, it's bad practice to connect the unit tests -- they
> should be as simple as possible and stand alone. Technologically,
> when the first test fails, then the second one fails with a really
> obscure error. I can't remember the specifics, but it was bad enough
> to convince me to rewrite all of my "dependsOnMethod" tests.
I agree in general :-)
In some cases test setup can take long or a resource cannot be reused
quickly enough and (test) class level setup is in order. For the former
doing an exhaustive testing of database functionality can be an
example. I also had a case involving extended XML trees for which
initializations would have been cumbersome and would have made more
hardly followable code. (Admittedly it is still a trade-off considering
the brittleness you mention.) For the latter I have only a gut feeling
that some use of the resources by River might fall into that category.
Zsolt
Ps.: It should be noted that TestNG runs test cases parallely by
default.
RE: test framework migration
Posted by Christopher Dolan <ch...@avid.com>.
While useful, I recommend against that feature in general. Philosophically, it's bad practice to connect the unit tests -- they should be as simple as possible and stand alone. Technologically, when the first test fails, then the second one fails with a really obscure error. I can't remember the specifics, but it was bad enough to convince me to rewrite all of my "dependsOnMethod" tests.
Chris
-----Original Message-----
From: Zsolt Kúti [mailto:la.tinca@gmail.com]
Sent: Saturday, November 13, 2010 7:29 AM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration
On Fri, 12 Nov 2010 13:16:58 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> One more thing: the annotations are named similarly, but they are NOT
> shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
> similarity makes it easier to change the source code via
> s/junit/testng/g but it's not zero-effort.
> Chris
A further thing I found useful: test methods can be
ordered/made dependent.
@Test
public void m1() {...}
@Test(dependsOnMethods="m1")
public void m2() {...}
Zsolt
Re: test framework migration
Posted by Zsolt Kúti <la...@gmail.com>.
On Fri, 12 Nov 2010 13:16:58 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> One more thing: the annotations are named similarly, but they are NOT
> shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
> similarity makes it easier to change the source code via
> s/junit/testng/g but it's not zero-effort.
> Chris
A further thing I found useful: test methods can be
ordered/made dependent.
@Test
public void m1() {...}
@Test(dependsOnMethods="m1")
public void m2() {...}
Zsolt
RE: test framework migration - was: Re: Hudson build is back to normal
Posted by Christopher Dolan <ch...@avid.com>.
One more thing: the annotations are named similarly, but they are NOT
shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
similarity makes it easier to change the source code via
s/junit/testng/g but it's not zero-effort.
Chris
-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]
Sent: Thursday, November 11, 2010 7:04 PM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration - was: Re: Hudson build is back to
normal
[...snip...]
Has anyone else written any junit tests other than myself? If TestNG is
justifiably better than Junit, I'd be prepared to convert my tests, I
believe many of the annotations are common?
[...snip...]
RE: test framework migration - was: Re: Hudson build is back to normal
Posted by Christopher Dolan <ch...@avid.com>.
Last year my project switched all of our tests from Junit 3 to TestNG. My big win with TestNG vs. Junit 4 was the ability to classify the tests and decide at runtime which to run. Most of my tests are annotated with @Test(groups="build") which are the tests executed by my continuous integration process. Only a subset of my code is Mac-specific, so those tests are marked as @Test(groups={"build","macos"}). My really slow tests are marked as @Test(groups="all") and are only executed on specific machines.
>From Ant, I choose which groups to run via <testng groups="build" ... /> or <testng groups="macos" ... /> etc.
TestNG also has an @DataProvider notation that dramatically simplifies groups of tests that have the exact same code but different initial conditions. Here's a too-simple example just to show the technique:
@DataProvider(name="testcases")
public Object[][] getTestcases() {
return new Object[][] {
{ 1, "1" },
{ -0, "0" },
};
}
@Test(groups="build", dataProvider="testcases")
public void testNumification(int num, String expected) {
Assert.assertEquals("" + num, expected);
}
Another useful annotation regards exceptions, which says to fail the test unless an exception class (or subclass) is thrown
@Test(groups="build", expectedException=IllegalArgumentException.class)
public void testParseIntFailure() {
Integer.parseInt("not a number");
}
TestNG also lets you throw org.testng.SkipException at runtime to neither pass nor fail the test. For example, if I want to run a test on Win7 but skip it on WinXP.
But, I agree with Peter that it has to be all TestNG or all JUnit. You can make them play nice, but it's not worth the hassle. The conversion was a little tedious (do NOT trust the auto-convert tools! Convert by hand instead!) but bearable. TestNG has their own org.testng.Assert that has a different API from org.junit.Assert, but both throw AssertionError in the end so it doesn't actually matter which you use. I like TestNG's API a little better (methods arguments are ordered "got, expected, message" vs. Junit's "message, expected, got").
Chris
-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]
Sent: Friday, November 12, 2010 4:06 AM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration - was: Re: Hudson build is back to normal
Thanks Tom, I agree, if someone has a good reason...
Cheers,
Peter.
Tom Hobbs wrote:
> I've written a couple of JUnit tests.
>
> I've not used TestNG for many years so can't really comment on it's
> comparison with JUnit. I don't object to converting my tests, however
> I'd rather convert them because "TestNG does X which we need to do Y"
> rather than "We should swap to TestNG because it's better."
>
> I agree, this should be an either/or decision. We should definitely
> *not* be using both.
>
> On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
>
>> Patricia Shanahan wrote:
>>
>>> Zsolt Kúti wrote:
>>>
>>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>>
>>>> ...
>>>>
>>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>>> JUnit. The others would be QA candidates.
>>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>>> requires knowledge of how the jtreg framework operates, as well as
>>>>> the proposed target framework (JUnit, QA).
>>>>>
>>>>>
>>>>>
>>>>>> JUnit's good when we're only testing a single object
>>>>>> implementation, we can document and expect people to utilse the qa
>>>>>> suite for more complex tests.
>>>>>>
>>>>> Agreed.
>>>>>
>>>> Hello hard workers,
>>>>
>>>> It would be worth considering the use of TestNG instead of JUnit.
>>>> I have no experience in their comparison, so relied on other
>>>> sources when I was to decid what framework to use (like this:
>>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>>> TestNG features that are missing from JUnit can be useful in a complex
>>>> test environment like that of River.
>>>>
>>> If we were starting cold, with no existing tests, I might be open to this
>>> suggestion. As it is, we already have a QA framework that can do all the
>>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>>
>>> I think the objective in converting jtreg tests would be to reduce the
>>> number of frameworks, and the amount of software we need installed, in order
>>> to run a full test. Switching them to TestNG, or anything else other than
>>> JUnit or the River QA framework, would not achieve that.
>>>
>>> Patricia
>>>
>>>
>> Has anyone else written any junit tests other than myself? If TestNG is
>> justifiably better than Junit, I'd be prepared to convert my tests, I
>> believe many of the annotations are common?
>>
>> But it would have to be Junit OR TestNG, not both.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Peter Firmstone <ji...@zeus.net.au>.
Thanks Tom, I agree, if someone has a good reason...
Cheers,
Peter.
Tom Hobbs wrote:
> I've written a couple of JUnit tests.
>
> I've not used TestNG for many years so can't really comment on it's
> comparison with JUnit. I don't object to converting my tests, however
> I'd rather convert them because "TestNG does X which we need to do Y"
> rather than "We should swap to TestNG because it's better."
>
> I agree, this should be an either/or decision. We should definitely
> *not* be using both.
>
> On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
>
>> Patricia Shanahan wrote:
>>
>>> Zsolt Kúti wrote:
>>>
>>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>>
>>>> ...
>>>>
>>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>>> JUnit. The others would be QA candidates.
>>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>>> requires knowledge of how the jtreg framework operates, as well as
>>>>> the proposed target framework (JUnit, QA).
>>>>>
>>>>>
>>>>>
>>>>>> JUnit's good when we're only testing a single object
>>>>>> implementation, we can document and expect people to utilse the qa
>>>>>> suite for more complex tests.
>>>>>>
>>>>> Agreed.
>>>>>
>>>> Hello hard workers,
>>>>
>>>> It would be worth considering the use of TestNG instead of JUnit.
>>>> I have no experience in their comparison, so relied on other
>>>> sources when I was to decid what framework to use (like this:
>>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>>> TestNG features that are missing from JUnit can be useful in a complex
>>>> test environment like that of River.
>>>>
>>> If we were starting cold, with no existing tests, I might be open to this
>>> suggestion. As it is, we already have a QA framework that can do all the
>>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>>
>>> I think the objective in converting jtreg tests would be to reduce the
>>> number of frameworks, and the amount of software we need installed, in order
>>> to run a full test. Switching them to TestNG, or anything else other than
>>> JUnit or the River QA framework, would not achieve that.
>>>
>>> Patricia
>>>
>>>
>> Has anyone else written any junit tests other than myself? If TestNG is
>> justifiably better than Junit, I'd be prepared to convert my tests, I
>> believe many of the annotations are common?
>>
>> But it would have to be Junit OR TestNG, not both.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Tom Hobbs <tv...@googlemail.com>.
I've written a couple of JUnit tests.
I've not used TestNG for many years so can't really comment on it's
comparison with JUnit. I don't object to converting my tests, however
I'd rather convert them because "TestNG does X which we need to do Y"
rather than "We should swap to TestNG because it's better."
I agree, this should be an either/or decision. We should definitely
*not* be using both.
On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
> Patricia Shanahan wrote:
>>
>> Zsolt Kúti wrote:
>>>
>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>
>>> ...
>>>>
>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>> JUnit. The others would be QA candidates.
>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>> requires knowledge of how the jtreg framework operates, as well as
>>>> the proposed target framework (JUnit, QA).
>>>>
>>>>
>>>>> JUnit's good when we're only testing a single object
>>>>> implementation, we can document and expect people to utilse the qa
>>>>> suite for more complex tests.
>>>>
>>>> Agreed.
>>>
>>> Hello hard workers,
>>>
>>> It would be worth considering the use of TestNG instead of JUnit.
>>> I have no experience in their comparison, so relied on other
>>> sources when I was to decid what framework to use (like this:
>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>> TestNG features that are missing from JUnit can be useful in a complex
>>> test environment like that of River.
>>
>> If we were starting cold, with no existing tests, I might be open to this
>> suggestion. As it is, we already have a QA framework that can do all the
>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>
>> I think the objective in converting jtreg tests would be to reduce the
>> number of frameworks, and the amount of software we need installed, in order
>> to run a full test. Switching them to TestNG, or anything else other than
>> JUnit or the River QA framework, would not achieve that.
>>
>> Patricia
>>
> Has anyone else written any junit tests other than myself? If TestNG is
> justifiably better than Junit, I'd be prepared to convert my tests, I
> believe many of the annotations are common?
>
> But it would have to be Junit OR TestNG, not both.
>
> Cheers,
>
> Peter.
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Peter Firmstone <ji...@zeus.net.au>.
Patricia Shanahan wrote:
> Zsolt Kúti wrote:
>> On Sat, 6 Nov 2010 17:40:19 +0100
>> Jonathan Costers <jo...@googlemail.com> wrote:
>>
>> ...
>>> Not all of them use/need a multi VM setup. Those are candidates for
>>> JUnit. The others would be QA candidates.
>>> I'm not saying it is easy to migrate any of these though, doing so
>>> requires knowledge of how the jtreg framework operates, as well as
>>> the proposed target framework (JUnit, QA).
>>>
>>>
>>>> JUnit's good when we're only testing a single object
>>>> implementation, we can document and expect people to utilse the qa
>>>> suite for more complex tests.
>>> Agreed.
>>
>> Hello hard workers,
>>
>> It would be worth considering the use of TestNG instead of JUnit.
>> I have no experience in their comparison, so relied on other
>> sources when I was to decid what framework to use (like this:
>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>> TestNG features that are missing from JUnit can be useful in a complex
>> test environment like that of River.
>
> If we were starting cold, with no existing tests, I might be open to
> this suggestion. As it is, we already have a QA framework that can do
> all the complex, multi-JVM tests, and we have over 1000 existing tests
> using it.
>
> I think the objective in converting jtreg tests would be to reduce the
> number of frameworks, and the amount of software we need installed, in
> order to run a full test. Switching them to TestNG, or anything else
> other than JUnit or the River QA framework, would not achieve that.
>
> Patricia
>
Has anyone else written any junit tests other than myself? If TestNG is
justifiably better than Junit, I'd be prepared to convert my tests, I
believe many of the annotations are common?
But it would have to be Junit OR TestNG, not both.
Cheers,
Peter.
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Patricia Shanahan <pa...@acm.org>.
Zsolt Kúti wrote:
> On Sat, 6 Nov 2010 17:40:19 +0100
> Jonathan Costers <jo...@googlemail.com> wrote:
>
> ...
>> Not all of them use/need a multi VM setup. Those are candidates for
>> JUnit. The others would be QA candidates.
>> I'm not saying it is easy to migrate any of these though, doing so
>> requires knowledge of how the jtreg framework operates, as well as
>> the proposed target framework (JUnit, QA).
>>
>>
>>> JUnit's good when we're only testing a single object
>>> implementation, we can document and expect people to utilse the qa
>>> suite for more complex tests.
>> Agreed.
>
> Hello hard workers,
>
> It would be worth considering the use of TestNG instead of JUnit.
> I have no experience in their comparison, so relied on other
> sources when I was to decid what framework to use (like this:
> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
> TestNG features that are missing from JUnit can be useful in a complex
> test environment like that of River.
If we were starting cold, with no existing tests, I might be open to
this suggestion. As it is, we already have a QA framework that can do
all the complex, multi-JVM tests, and we have over 1000 existing tests
using it.
I think the objective in converting jtreg tests would be to reduce the
number of frameworks, and the amount of software we need installed, in
order to run a full test. Switching them to TestNG, or anything else
other than JUnit or the River QA framework, would not achieve that.
Patricia
test framework migration - was: Re: Hudson build is back to normal
Posted by Zsolt Kúti <la...@gmail.com>.
On Sat, 6 Nov 2010 17:40:19 +0100
Jonathan Costers <jo...@googlemail.com> wrote:
...
> Not all of them use/need a multi VM setup. Those are candidates for
> JUnit. The others would be QA candidates.
> I'm not saying it is easy to migrate any of these though, doing so
> requires knowledge of how the jtreg framework operates, as well as
> the proposed target framework (JUnit, QA).
>
>
> >
> > JUnit's good when we're only testing a single object
> > implementation, we can document and expect people to utilse the qa
> > suite for more complex tests.
>
> Agreed.
Hello hard workers,
It would be worth considering the use of TestNG instead of JUnit.
I have no experience in their comparison, so relied on other
sources when I was to decid what framework to use (like this:
http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
TestNG features that are missing from JUnit can be useful in a complex
test environment like that of River.
Zsolt
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Jonathan Costers <jo...@googlemail.com>.
2010/11/6 Peter Firmstone <ji...@zeus.net.au>
>
> Infra provided a solaris zone:
>
> Sun Microsystems Inc. SunOS 5.10 Generic January 2005
> -bash-3.00$ uname -a
> SunOS river 5.10 Generic_137112-04 i86pc i386 i86pc
> -bash-3.00$ prtconf -D
> System Configuration: Sun Microsystems i86pc
> Memory size: 8192 Megabytes
> System Peripherals (Software Nodes):
>
> prtconf: devinfo facility not available
>
> The KDC and squid just need to be set up. We could establish a registrar
> there also I suppose.
>
> OK great. There is a JIRA issue open already for this I believe.
>
> Well I don't think Junit would be suitable, since multiple jvm's are
> employed. I've also had to modify some jtreg tests that made some
> assumptions about ClassLoader visibility (jre/lib/ext related) and failed
> later when we made some changes. The qa suite might be suitable, but I
> don't think the effort's worthwhile, we can't guarantee that we're
> simulating the failure conditions properly, even if only a few of us run
> these tests, it's better than everyone running them if they're not genuinely
> simulating failure conditions.
>
Not all of them use/need a multi VM setup. Those are candidates for JUnit.
The others would be QA candidates.
I'm not saying it is easy to migrate any of these though, doing so requires
knowledge of how the jtreg framework operates, as well as the proposed
target framework (JUnit, QA).
>
> JUnit's good when we're only testing a single object implementation, we can
> document and expect people to utilse the qa suite for more complex tests.
>
Agreed.
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Peter Firmstone <ji...@zeus.net.au>.
Jonathan Costers wrote:
> 2010/11/5 Peter Firmstone <ji...@zeus.net.au>
>
>
>> That's, quite an achievement for all involved, is that all the tests?
>>
>>
>
> We have enabled most known QA test categories now. There are some left out
> there, but mostly empty ones.
>
> Some of the QA tests were deliberately set to be ignored, like the Kerberos
> tests (also bc we need a KDC for that).
>
>
>
>> For jtreg it's about time I got the KDC server set up, not sure what to do
>> for a squid proxy server though, do you think it would be safe to run squid
>> on the same server with the KDC, seeing as they're there mostly for tests?
>>
>>
>
> I guess we need to pose the question to INFRA: if we need external
> infrastructure in place for our tests, like a KDC or a proxy server, what
> would be the best way to set that up, if at all possible? I believe an
> initial discussion took place at one time, but not sure what conclusion was
> drawn from that (if any).
>
Infra provided a solaris zone:
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
-bash-3.00$ uname -a
SunOS river 5.10 Generic_137112-04 i86pc i386 i86pc
-bash-3.00$ prtconf -D
System Configuration: Sun Microsystems i86pc
Memory size: 8192 Megabytes
System Peripherals (Software Nodes):
prtconf: devinfo facility not available
The KDC and squid just need to be set up. We could establish a
registrar there also I suppose.
> Setting up a separate zone just for a KDC seems like overkill to me, but I'm
> not a system admin. Also considering the fact that any client (the Hudson
> builders) would need Kerberos client software installed anyway.
> On my machine, I installed and configured a KDC (there is a JIRA ticket
> explaining what I did for that, on Ubuntu), and I am running the tests from
> the same machine (pointing QA config to my KDC). That seems to work fine.
>
> An additional hurdle for the jtreg suite is the jtreg software that would
> need to be installed on the Hudson builders. Jtreg was originally meant to
> regression test the JDK, and because JERI & co was originally intended to be
> included in the standard Java spec (see JSR-76 and JSR-78), these jtreg
> tests came to exist ... Things changed dramatically after both JSRs were
> rejected, though.
> Maybe we can migrate the valuable tests in that jtreg suite to either JUnit
> or QA tests? I am aware there are caveats (test isolation level for
> instance), but they seem to be manageable. I'm talking about a gradual
> process here, converting test after test over a long period of time. We are
> talking about around 100 jtreg tests, with varying complexity and isolation
> levels.
>
Well I don't think Junit would be suitable, since multiple jvm's are
employed. I've also had to modify some jtreg tests that made some
assumptions about ClassLoader visibility (jre/lib/ext related) and
failed later when we made some changes. The qa suite might be suitable,
but I don't think the effort's worthwhile, we can't guarantee that we're
simulating the failure conditions properly, even if only a few of us run
these tests, it's better than everyone running them if they're not
genuinely simulating failure conditions.
> Reducing the number of ways to test things is probably also good for general
> understanding :-)
>
JUnit's good when we're only testing a single object implementation, we
can document and expect people to utilse the qa suite for more complex
tests.
An interesting developments for Jtreg is newer versions have had some
speed improvements recently.
Cheers,
Peter.
>
>
>
>> Regards,
>>
>> Peter.
>>
>>
>> Jonathan Costers wrote:
>>
>>
>>> Hooray!
>>>
>>> All 1409 tests passed on ubuntu, taking 17hrs to run:
>>>
>>>
>>> [java]
>>> [java] # of tests started = 1409
>>> [java] # of tests completed = 1409
>>> [java] # of tests skipped = 47
>>> [java] # of tests passed = 1409
>>> [java] # of tests failed = 0
>>> [java]
>>> [java] -----------------------------------------
>>> [java]
>>> [java] Date finished:
>>> [java] Thu Nov 04 03:00:44 UTC 2010
>>> [java] Time elapsed:
>>> [java] 62200 seconds
>>>
>>>
>>> Thanks to all for getting RIVER-301 out of the way!
>>>
>>> At this point, I believe we are sufficiently armed to validate some of the
>>> interesting proposals, improvements and experiments that have been posted
>>> to
>>> this list the last couple of months.
>>>
>>> Closing RIVER-301 (which had been open/in progress for a very long time)
>>> is
>>> also another step towards graduation from the incubator, since one of the
>>> main goals was to setup a testing framework.
>>>
>>> 2010/11/4 Apache Hudson Server <hu...@hudson.apache.org>
>>>
>>>
>>>
>>>
>>>> See <https://hudson.apache.org/hudson/job/River-trunk-QA/51/changes>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Jonathan Costers <jo...@googlemail.com>.
2010/11/5 Peter Firmstone <ji...@zeus.net.au>
> That's, quite an achievement for all involved, is that all the tests?
>
We have enabled most known QA test categories now. There are some left out
there, but mostly empty ones.
Some of the QA tests were deliberately set to be ignored, like the Kerberos
tests (also bc we need a KDC for that).
> For jtreg it's about time I got the KDC server set up, not sure what to do
> for a squid proxy server though, do you think it would be safe to run squid
> on the same server with the KDC, seeing as they're there mostly for tests?
>
I guess we need to pose the question to INFRA: if we need external
infrastructure in place for our tests, like a KDC or a proxy server, what
would be the best way to set that up, if at all possible? I believe an
initial discussion took place at one time, but not sure what conclusion was
drawn from that (if any).
Setting up a separate zone just for a KDC seems like overkill to me, but I'm
not a system admin. Also considering the fact that any client (the Hudson
builders) would need Kerberos client software installed anyway.
On my machine, I installed and configured a KDC (there is a JIRA ticket
explaining what I did for that, on Ubuntu), and I am running the tests from
the same machine (pointing QA config to my KDC). That seems to work fine.
An additional hurdle for the jtreg suite is the jtreg software that would
need to be installed on the Hudson builders. Jtreg was originally meant to
regression test the JDK, and because JERI & co was originally intended to be
included in the standard Java spec (see JSR-76 and JSR-78), these jtreg
tests came to exist ... Things changed dramatically after both JSRs were
rejected, though.
Maybe we can migrate the valuable tests in that jtreg suite to either JUnit
or QA tests? I am aware there are caveats (test isolation level for
instance), but they seem to be manageable. I'm talking about a gradual
process here, converting test after test over a long period of time. We are
talking about around 100 jtreg tests, with varying complexity and isolation
levels.
Reducing the number of ways to test things is probably also good for general
understanding :-)
> Regards,
>
> Peter.
>
>
> Jonathan Costers wrote:
>
>> Hooray!
>>
>> All 1409 tests passed on ubuntu, taking 17hrs to run:
>>
>>
>> [java]
>> [java] # of tests started = 1409
>> [java] # of tests completed = 1409
>> [java] # of tests skipped = 47
>> [java] # of tests passed = 1409
>> [java] # of tests failed = 0
>> [java]
>> [java] -----------------------------------------
>> [java]
>> [java] Date finished:
>> [java] Thu Nov 04 03:00:44 UTC 2010
>> [java] Time elapsed:
>> [java] 62200 seconds
>>
>>
>> Thanks to all for getting RIVER-301 out of the way!
>>
>> At this point, I believe we are sufficiently armed to validate some of the
>> interesting proposals, improvements and experiments that have been posted
>> to
>> this list the last couple of months.
>>
>> Closing RIVER-301 (which had been open/in progress for a very long time)
>> is
>> also another step towards graduation from the incubator, since one of the
>> main goals was to setup a testing framework.
>>
>> 2010/11/4 Apache Hudson Server <hu...@hudson.apache.org>
>>
>>
>>
>>> See <https://hudson.apache.org/hudson/job/River-trunk-QA/51/changes>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Peter Firmstone <ji...@zeus.net.au>.
That's, quite an achievement for all involved, is that all the tests?
For jtreg it's about time I got the KDC server set up, not sure what to
do for a squid proxy server though, do you think it would be safe to run
squid on the same server with the KDC, seeing as they're there mostly
for tests?
Regards,
Peter.
Jonathan Costers wrote:
> Hooray!
>
> All 1409 tests passed on ubuntu, taking 17hrs to run:
>
>
> [java]
> [java] # of tests started = 1409
> [java] # of tests completed = 1409
> [java] # of tests skipped = 47
> [java] # of tests passed = 1409
> [java] # of tests failed = 0
> [java]
> [java] -----------------------------------------
> [java]
> [java] Date finished:
> [java] Thu Nov 04 03:00:44 UTC 2010
> [java] Time elapsed:
> [java] 62200 seconds
>
>
> Thanks to all for getting RIVER-301 out of the way!
>
> At this point, I believe we are sufficiently armed to validate some of the
> interesting proposals, improvements and experiments that have been posted to
> this list the last couple of months.
>
> Closing RIVER-301 (which had been open/in progress for a very long time) is
> also another step towards graduation from the incubator, since one of the
> main goals was to setup a testing framework.
>
> 2010/11/4 Apache Hudson Server <hu...@hudson.apache.org>
>
>
>> See <https://hudson.apache.org/hudson/job/River-trunk-QA/51/changes>
>>
>>
>>
>>
>
>
Re: Hudson build is back to normal : River-trunk-QA #51
Posted by Jonathan Costers <jo...@googlemail.com>.
Hooray!
All 1409 tests passed on ubuntu, taking 17hrs to run:
[java]
[java] # of tests started = 1409
[java] # of tests completed = 1409
[java] # of tests skipped = 47
[java] # of tests passed = 1409
[java] # of tests failed = 0
[java]
[java] -----------------------------------------
[java]
[java] Date finished:
[java] Thu Nov 04 03:00:44 UTC 2010
[java] Time elapsed:
[java] 62200 seconds
Thanks to all for getting RIVER-301 out of the way!
At this point, I believe we are sufficiently armed to validate some of the
interesting proposals, improvements and experiments that have been posted to
this list the last couple of months.
Closing RIVER-301 (which had been open/in progress for a very long time) is
also another step towards graduation from the incubator, since one of the
main goals was to setup a testing framework.
2010/11/4 Apache Hudson Server <hu...@hudson.apache.org>
> See <https://hudson.apache.org/hudson/job/River-trunk-QA/51/changes>
>
>
>
Hudson build is back to normal : River-trunk-QA #51
Posted by Apache Hudson Server <hu...@hudson.apache.org>.
See <https://hudson.apache.org/hudson/job/River-trunk-QA/51/changes>