You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@jakarta.apache.org by Sc...@lotus.com on 2001/02/10 22:06:38 UTC
Test Infrastructure Project Proposal
Ross Burton <ro...@mail.com> wrote:
> Having spent all day upto my arm pits in bits of cocoon, xerces and
> xalan, I've come to the conclusion that tracking down bugs in bits of
> cocoon can be nearly impossible at the moment. How would you guys feel
> about us starting to use JUnit to run unit tests over cocoon?
First, sorry for the large cross-posting. But I wanted to make sure this
note is received by a large audience. Replies should be sent only to
general@xml.apache.org (I'm not subscribed to general@jakarta.apache.org).
Xalan being a project that 1) requires a *lot* of testing, and 2) is
sandwiched inbetween Cocoon and Xerces, 3) is dependent on other
technologies like BSF, and 4) is used in several other pipeline scenarios,
we (i.e. the folks at Lotus who are involved in the Xalan project) have
been doing a lot of thinking about this subject.
What is happening in the XML/Web world is the integration of a lot of
smaller components, plugged together via (hopefully) standard interfaces.
This increases the need for unit testing, and integration testing in a big
way. In systems such as we are building, robustness is everything, and
fragility is becoming an increasing problem. I feel this is probably the
most critical issue xml.apache.org and the Jakarta projects are facing,
even above performance issues. The days have ended when Xerces can release
without testing with Xalan, and Xalan can release without testing with
Cocoon, etc.
Also, I feel what we are all practising is pretty close to "Extreme
Programming" (http://www.extremeprogramming.org/), by our very motto of
"release early and often". Extreme programming is very reliant on having a
*lot* of tests that are constantly run (see
http://www.extremeprogramming.org/rules/unittests.html and
http://www.extremeprogramming.org/rules/functionaltests.html). This means
that tests must be very fast to create, easy to plug in, easy to have them
become part of the perminant acceptence tests, running the tests must be
extremely convenient, and diagnosing problems from the reports must also be
easy. And when a bug is found by a user, a test should almost always be
added to the acceptence tests
(http://www.extremeprogramming.org/rules/bugs.html).
We have analyzed JUnit and feel it doesn't address our needs, nor the
integration testing needs, though we like it's Ant base.
I propose a project for testing infrastructure, that covers the needs of
unit testing, stress testing, performance testing, negative testing (i.e.
testing of error conditions), integration testing, and error logging and
reporting. I don't know or care if this is a Jakarta project or an
xml.apache.org project. I believe the Jakarta project has been thinking
somewhat along these lines?
The Xalan project already has a fair amount of infrastructure that we would
be happy to contribute. But basically, I think we should start first with
requirements, then a schema for reporting (i.e. design the data first), go
next to interfaces, and then decide what existing code can be used.
Thoughts? Does anyone want to -1 this? If not, where should it live? (I
suspect the answer is in Jakarta, next to Ant). What should it be named?
What are the next steps? Who should be the founders? And what about
C-language integration testing, as well as other languages (which might
argue against a home in Jakarta?)?
Again, sorry for the large cross-posting, but I think it's time for this
issue to get full attention from all the projects.
-scott
Ross Burton
<ross.burton@ To: cocoon-dev@xml.apache.org
mail.com> cc: (bcc: Scott Boag/CAM/Lotus)
Sent by: Subject: Re: [C2] Unit testing.
ross@itzinter
active.com
02/10/2001
09:37 AM
Please
respond to
cocoon-dev
Paul Russell wrote:
>
> Guys,
>
> Having spent all day upto my arm pits in bits of cocoon, xerces and
> xalan, I've come to the conclusion that tracking down bugs in bits of
> cocoon can be nearly impossible at the moment. How would you guys feel
> about us starting to use JUnit to run unit tests over cocoon? That way
> we can be a lot more confident that we didn't just break something when
> we make a change? The tests can range from checking that a matcher
> matches, to checking that a classloader class loads, to checking that
> the output of a particular pipeline is as expected. I'm very happy to
> help in this respect, and I have minions (eh, boss?) that will put some
> work into developing tests, too. How do you guys feel about this?
>
> Does anyone know whether the IBM Public Licence is APL compatible, or
> would we have to keep JUnit out of the repository?
A small note - didn't the Avalon list seperate their testing code from
the core and put it on Sourceforge? arrowhead.sourceforge.net IIRC.
Ah - just went there. Yes, that is the site of the code (Kevin Burton
is the developer) but there is no downloads, no web page, no nothing.
Anyone know what happened? Did it get dropped?
Apart from that, +1 for the tests.
Ross Burton
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Test Infrastructure Project Proposal
Posted by Sam Ruby <ru...@nc.rr.com>.
Arved Sandstrom wrote:
>
> Don't mind at all (FOP being the poster child, I mean). I took a gander
and
> damned if I can reproduce the docs build failure...we don't have a FAQ
document
> and the current build.xml makes no reference to one at all.
It looks like I finally got someone to listen:
revision 1.20
date: 2001/02/08 14:09:25; author: fotis; state: Exp; lines: +0 -1
removes old reference to FAQ during build of distribution
Now I guess I need to look for another poster child. ;-)
> I won't lose sleep if it all stays in Jakarta, except that I'm
> wondering if it's desirable for GP testing code or build tool code to
reside in
> Jakarta when maybe XML project people want a strong say in those things
also,
> no less so than Jakarta. How would that work?
I'm not sure where it belongs yet, but many of us cross over, and hopefully
whereever the final home ends up will be a place where people who want to
participate and contribute good code will be welcome.
Re: Test Infrastructure Project Proposal
Posted by Arved Sandstrom <Ar...@chebucto.ns.ca>.
Hi, Sam
Don't mind at all (FOP being the poster child, I mean). I took a gander and
damned if I can reproduce the docs build failure...we don't have a FAQ document
and the current build.xml makes no reference to one at all. However, as late as
FOP 0.16.0 the build.xml "docs" target _does_ make reference to a FAQ file. I
can only assume that there was a period of time where the FAQ got removed but
the build.xml didn't reflect the change. And that was when your system updated
from CVS.
I'm guilty of what the real cause is here...not running an absolute complete
build when the distribution changes. There is a tendency to only check the
compile stage.
Wrt "stuff": yes, definitely. If the majority of committers want a thing in a
certain place then that is where it should go. As a single committer my "wants"
are to see codebases like Ant and jakarta-regexp in a location that indicates
that they are GP things. Particularly if we add _more_ GP stuff, i.e. testing
stuff, then maybe that now makes sense. That's just my opinion at the
moment...I won't lose sleep if it all stays in Jakarta, except that I'm
wondering if it's desirable for GP testing code or build tool code to reside in
Jakarta when maybe XML project people want a strong say in those things also,
no less so than Jakarta. How would that work?
Regards,
Arved
On Sat, 10 Feb 2001, you wrote:
> ----- Original Message -----
> Arved Sandstrom wrote:
> >
> > I'd say outside Jakarta _and_ XML. Ant no longer belongs in Jakarta
> either,
> > since everyone and their grandmother is using it. Actually, there's a
> bunch of
> > stuff in Jakarta (regexp, ORO, etc) which has nothing to do with Jakarta,
> per
> > se. I'd hate to see yet more stuff put in there.
> >
> > What about the possible role of Tigris? The main infrastructure tools
> projects
> > (Subversion, ArgoUML, etc) seem to very naturally fit there. Once we have
> > requirements, developing support tools can happen there, while a parallel
> > Apache project works on how Apache would use it, and the bits that pertain
> > directly to Apache testing. Just some thoughts - if everything lives in
> Apache
> > that's cool with me also.
>
> As the new chairman of Jakarta, I've been giving a lot of thought to this.
> I start with the premise that "stuff" belongs where a majority of
> committters want it to be put.
>
> By the way, xml-fop is still my poster child for why something like this
> will not work until people's habits change. I know I pointed it out before,
> but could you take a look again at the following build failure? It looks
> relatively easy to fix:
>
> http://oss.software.ibm.com/developerworks/opensource/jakarta/proto24/xml-fo
> p.html
>
> So the real question is, how do we get peoples habits to change?
>
>
> ---------------------------------------------------------------------
> In case of troubles, e-mail: webmaster@xml.apache.org
> To unsubscribe, e-mail: general-unsubscribe@xml.apache.org
> For additional commands, e-mail: general-help@xml.apache.org
--
Senior Developer
e-plicity.com (www.e-plicity.com)
Halifax, Nova Scotia
"B2B Wireless in Canada's Ocean Playground"
Re: Test Infrastructure Project Proposal
Posted by Sam Ruby <ru...@nc.rr.com>.
----- Original Message -----
Arved Sandstrom wrote:
>
> I'd say outside Jakarta _and_ XML. Ant no longer belongs in Jakarta
either,
> since everyone and their grandmother is using it. Actually, there's a
bunch of
> stuff in Jakarta (regexp, ORO, etc) which has nothing to do with Jakarta,
per
> se. I'd hate to see yet more stuff put in there.
>
> What about the possible role of Tigris? The main infrastructure tools
projects
> (Subversion, ArgoUML, etc) seem to very naturally fit there. Once we have
> requirements, developing support tools can happen there, while a parallel
> Apache project works on how Apache would use it, and the bits that pertain
> directly to Apache testing. Just some thoughts - if everything lives in
Apache
> that's cool with me also.
As the new chairman of Jakarta, I've been giving a lot of thought to this.
I start with the premise that "stuff" belongs where a majority of
committters want it to be put.
By the way, xml-fop is still my poster child for why something like this
will not work until people's habits change. I know I pointed it out before,
but could you take a look again at the following build failure? It looks
relatively easy to fix:
http://oss.software.ibm.com/developerworks/opensource/jakarta/proto24/xml-fo
p.html
So the real question is, how do we get peoples habits to change?
Re: Test Infrastructure Project Proposal
Posted by Arved Sandstrom <Ar...@chebucto.ns.ca>.
On Sat, 10 Feb 2001, Scott_Boag@lotus.com wrote:
[ SNIP ]
> What is happening in the XML/Web world is the integration of a lot of
> smaller components, plugged together via (hopefully) standard interfaces.
> This increases the need for unit testing, and integration testing in a big
> way. In systems such as we are building, robustness is everything, and
> fragility is becoming an increasing problem. I feel this is probably the
> most critical issue xml.apache.org and the Jakarta projects are facing,
> even above performance issues. The days have ended when Xerces can release
> without testing with Xalan, and Xalan can release without testing with
> Cocoon, etc.
>
> The Xalan project already has a fair amount of infrastructure that we would
> be happy to contribute. But basically, I think we should start first with
> requirements, then a schema for reporting (i.e. design the data first), go
> next to interfaces, and then decide what existing code can be used.
>
> Thoughts? Does anyone want to -1 this? If not, where should it live? (I
> suspect the answer is in Jakarta, next to Ant). What should it be named?
> What are the next steps? Who should be the founders? And what about
> C-language integration testing, as well as other languages (which might
> argue against a home in Jakarta?)?
I'm absolutely for this. Long overdue. One major point: we'll never get off the
ground with testing of this sort until we have configuration management
(including change management). It doesn't have to be a commercial tool like
Perforce, but we need something, even if it's just Apache-standard SDFs
(Software Description Files).
I'd say outside Jakarta _and_ XML. Ant no longer belongs in Jakarta either,
since everyone and their grandmother is using it. Actually, there's a bunch of
stuff in Jakarta (regexp, ORO, etc) which has nothing to do with Jakarta, per
se. I'd hate to see yet more stuff put in there.
What about the possible role of Tigris? The main infrastructure tools projects
(Subversion, ArgoUML, etc) seem to very naturally fit there. Once we have
requirements, developing support tools can happen there, while a parallel
Apache project works on how Apache would use it, and the bits that pertain
directly to Apache testing. Just some thoughts - if everything lives in Apache
that's cool with me also.
Regards,
Arved Sandstrom
--
Fairly Senior Software Type
e-plicity (www.e-plicity.com)
Halifax, Nova Scotia
Re: Test Infrastructure Project Proposal
Posted by Peter Donald <do...@apache.org>.
At 04:06 10/2/01 -0500, Scott_Boag@lotus.com wrote:
>I propose a project for testing infrastructure, that covers the needs of
>unit testing, stress testing, performance testing, negative testing (i.e.
>testing of error conditions), integration testing, and error logging and
>reporting.
Stress and performance testing belong in a separate project and we already
have one (jmeter) which could easily be extended to performace/stress test
other things. negative testing is usually considered a subset of unit
testing and should be covered by the same framework.
I am not sure what you mean by "integration testing" - I have heard
different meanings for it by different people. If you meaning integration
aka continuous integration then tinderbox/gump should do it for us. If you
mean "funcitonal testing" (ie test the integration of components and see if
they perform function) then this screams out for a customized set of ant
tasks/runtime or an amalgamation of things like gtest.
Could you expand what you mean by error logging/reporting - presumably of
tests? If so then I would log
* success
* failure - expected possible error condition
* error - unexpected error
* non-completion ( ie test required other tests to pass before it could be
run).
These could be spit out to a listener that could construct the appropriate
report.
>A small note - didn't the Avalon list seperate their testing code
It was always separate. it was a unit testing framework I built for a
separate project. It only ended up in Avalon as I needed unit testing in
avalon. It is still currently in avalon under package org.apache.testlet.
Cheers,
Pete
*-----------------------------------------------------*
| "Faced with the choice between changing one's mind, |
| and proving that there is no need to do so - almost |
| everyone gets busy on the proof." |
| - John Kenneth Galbraith |
*-----------------------------------------------------*
Re: Test Infrastructure Project Proposal
Posted by Jon Stevens <jo...@latchkey.com>.
on 2/10/01 1:06 PM, "Scott_Boag@lotus.com" <Sc...@lotus.com> wrote:
> We have analyzed JUnit and feel it doesn't address our needs, nor the
> integration testing needs, though we like it's Ant base.
What would it take to make JUnit address your needs?
I'm tired of hearing people start new projects without also giving details
about exactly why they can't do the work to make existing projects more
compatible for their needs.
thanks,
-jon
--
If you come from a Perl or PHP background, JSP is a way to take
your pain to new levels. --Anonymous
<http://jakarta.apache.org/velocity/> && <http://java.apache.org/turbine/>
Re: Test Infrastructure Project Proposal
Posted by Peter Donald <do...@apache.org>.
At 04:06 10/2/01 -0500, Scott_Boag@lotus.com wrote:
>I propose a project for testing infrastructure, that covers the needs of
>unit testing, stress testing, performance testing, negative testing (i.e.
>testing of error conditions), integration testing, and error logging and
>reporting.
Stress and performance testing belong in a separate project and we already
have one (jmeter) which could easily be extended to performace/stress test
other things. negative testing is usually considered a subset of unit
testing and should be covered by the same framework.
I am not sure what you mean by "integration testing" - I have heard
different meanings for it by different people. If you meaning integration
aka continuous integration then tinderbox/gump should do it for us. If you
mean "funcitonal testing" (ie test the integration of components and see if
they perform function) then this screams out for a customized set of ant
tasks/runtime or an amalgamation of things like gtest.
Could you expand what you mean by error logging/reporting - presumably of
tests? If so then I would log
* success
* failure - expected possible error condition
* error - unexpected error
* non-completion ( ie test required other tests to pass before it could be
run).
These could be spit out to a listener that could construct the appropriate
report.
>A small note - didn't the Avalon list seperate their testing code
It was always separate. it was a unit testing framework I built for a
separate project. It only ended up in Avalon as I needed unit testing in
avalon. It is still currently in avalon under package org.apache.testlet.
Cheers,
Pete
*-----------------------------------------------------*
| "Faced with the choice between changing one's mind, |
| and proving that there is no need to do so - almost |
| everyone gets busy on the proof." |
| - John Kenneth Galbraith |
*-----------------------------------------------------*
Re: Test Infrastructure Project Proposal
Posted by Peter Donald <do...@apache.org>.
At 04:06 10/2/01 -0500, Scott_Boag@lotus.com wrote:
>I propose a project for testing infrastructure, that covers the needs of
>unit testing, stress testing, performance testing, negative testing (i.e.
>testing of error conditions), integration testing, and error logging and
>reporting.
Stress and performance testing belong in a separate project and we already
have one (jmeter) which could easily be extended to performace/stress test
other things. negative testing is usually considered a subset of unit
testing and should be covered by the same framework.
I am not sure what you mean by "integration testing" - I have heard
different meanings for it by different people. If you meaning integration
aka continuous integration then tinderbox/gump should do it for us. If you
mean "funcitonal testing" (ie test the integration of components and see if
they perform function) then this screams out for a customized set of ant
tasks/runtime or an amalgamation of things like gtest.
Could you expand what you mean by error logging/reporting - presumably of
tests? If so then I would log
* success
* failure - expected possible error condition
* error - unexpected error
* non-completion ( ie test required other tests to pass before it could be
run).
These could be spit out to a listener that could construct the appropriate
report.
>A small note - didn't the Avalon list seperate their testing code
It was always separate. it was a unit testing framework I built for a
separate project. It only ended up in Avalon as I needed unit testing in
avalon. It is still currently in avalon under package org.apache.testlet.
Cheers,
Pete
*-----------------------------------------------------*
| "Faced with the choice between changing one's mind, |
| and proving that there is no need to do so - almost |
| everyone gets busy on the proof." |
| - John Kenneth Galbraith |
*-----------------------------------------------------*
Re: Test Infrastructure Project Proposal
Posted by Jon Stevens <jo...@latchkey.com>.
on 2/10/01 1:06 PM, "Scott_Boag@lotus.com" <Sc...@lotus.com> wrote:
> We have analyzed JUnit and feel it doesn't address our needs, nor the
> integration testing needs, though we like it's Ant base.
What would it take to make JUnit address your needs?
I'm tired of hearing people start new projects without also giving details
about exactly why they can't do the work to make existing projects more
compatible for their needs.
thanks,
-jon
--
If you come from a Perl or PHP background, JSP is a way to take
your pain to new levels. --Anonymous
<http://jakarta.apache.org/velocity/> && <http://java.apache.org/turbine/>
Re: Test Infrastructure Project Proposal
Posted by Peter Donald <do...@apache.org>.
At 04:06 10/2/01 -0500, Scott_Boag@lotus.com wrote:
>I propose a project for testing infrastructure, that covers the needs of
>unit testing, stress testing, performance testing, negative testing (i.e.
>testing of error conditions), integration testing, and error logging and
>reporting.
Stress and performance testing belong in a separate project and we already
have one (jmeter) which could easily be extended to performace/stress test
other things. negative testing is usually considered a subset of unit
testing and should be covered by the same framework.
I am not sure what you mean by "integration testing" - I have heard
different meanings for it by different people. If you meaning integration
aka continuous integration then tinderbox/gump should do it for us. If you
mean "funcitonal testing" (ie test the integration of components and see if
they perform function) then this screams out for a customized set of ant
tasks/runtime or an amalgamation of things like gtest.
Could you expand what you mean by error logging/reporting - presumably of
tests? If so then I would log
* success
* failure - expected possible error condition
* error - unexpected error
* non-completion ( ie test required other tests to pass before it could be
run).
These could be spit out to a listener that could construct the appropriate
report.
>A small note - didn't the Avalon list seperate their testing code
It was always separate. it was a unit testing framework I built for a
separate project. It only ended up in Avalon as I needed unit testing in
avalon. It is still currently in avalon under package org.apache.testlet.
Cheers,
Pete
*-----------------------------------------------------*
| "Faced with the choice between changing one's mind, |
| and proving that there is no need to do so - almost |
| everyone gets busy on the proof." |
| - John Kenneth Galbraith |
*-----------------------------------------------------*
Re: Test Infrastructure Project Proposal
Posted by Jon Stevens <jo...@latchkey.com>.
on 2/10/01 1:06 PM, "Scott_Boag@lotus.com" <Sc...@lotus.com> wrote:
> We have analyzed JUnit and feel it doesn't address our needs, nor the
> integration testing needs, though we like it's Ant base.
What would it take to make JUnit address your needs?
I'm tired of hearing people start new projects without also giving details
about exactly why they can't do the work to make existing projects more
compatible for their needs.
thanks,
-jon
--
If you come from a Perl or PHP background, JSP is a way to take
your pain to new levels. --Anonymous
<http://jakarta.apache.org/velocity/> && <http://java.apache.org/turbine/>
Re: Test Infrastructure Project Proposal
Posted by Peter Donald <do...@apache.org>.
At 04:06 10/2/01 -0500, Scott_Boag@lotus.com wrote:
>I propose a project for testing infrastructure, that covers the needs of
>unit testing, stress testing, performance testing, negative testing (i.e.
>testing of error conditions), integration testing, and error logging and
>reporting.
Stress and performance testing belong in a separate project and we already
have one (jmeter) which could easily be extended to performace/stress test
other things. negative testing is usually considered a subset of unit
testing and should be covered by the same framework.
I am not sure what you mean by "integration testing" - I have heard
different meanings for it by different people. If you meaning integration
aka continuous integration then tinderbox/gump should do it for us. If you
mean "funcitonal testing" (ie test the integration of components and see if
they perform function) then this screams out for a customized set of ant
tasks/runtime or an amalgamation of things like gtest.
Could you expand what you mean by error logging/reporting - presumably of
tests? If so then I would log
* success
* failure - expected possible error condition
* error - unexpected error
* non-completion ( ie test required other tests to pass before it could be
run).
These could be spit out to a listener that could construct the appropriate
report.
>A small note - didn't the Avalon list seperate their testing code
It was always separate. it was a unit testing framework I built for a
separate project. It only ended up in Avalon as I needed unit testing in
avalon. It is still currently in avalon under package org.apache.testlet.
Cheers,
Pete
*-----------------------------------------------------*
| "Faced with the choice between changing one's mind, |
| and proving that there is no need to do so - almost |
| everyone gets busy on the proof." |
| - John Kenneth Galbraith |
*-----------------------------------------------------*
Re: Test Infrastructure Project Proposal
Posted by Jon Stevens <jo...@latchkey.com>.
on 2/10/01 1:06 PM, "Scott_Boag@lotus.com" <Sc...@lotus.com> wrote:
> We have analyzed JUnit and feel it doesn't address our needs, nor the
> integration testing needs, though we like it's Ant base.
What would it take to make JUnit address your needs?
I'm tired of hearing people start new projects without also giving details
about exactly why they can't do the work to make existing projects more
compatible for their needs.
thanks,
-jon
--
If you come from a Perl or PHP background, JSP is a way to take
your pain to new levels. --Anonymous
<http://jakarta.apache.org/velocity/> && <http://java.apache.org/turbine/>