[IMPASSIONED PLEA] Naming and structuring test materials

[Benson Margulies <bi...@gmail.com>]

We have about 10 copies of the hello world schema floating around the
tree.

The idea is that wsdl-first and code-first test schemas and
implementation should live in testutils. The problem, from my jaundiced
point of view, is that we have been dumping things into there without
enough care in picking names.

For example, someone put a copy of the hello_world 'greeter' schema into
testutils, but modified it to use the XML binding. 

In my opinion, it should have been renamed and renamespaced to in the
process. The current state is unproductive in two ways.

First, new people looking for a test case schema are prone to do what I
did, which is see it, try to use it, and then belatedly run into the XML
binding specification. And then have to back up and turn around.

Second, it turns the testutils into a confusing arena in which no one
really knows what we have for inventory.

It looks to me as if another process is to take some example we get in a
test case, and check it in, 'as is'. I can see the reason to do this:
ultimately, the user's test is the regression test. However, there are
all kinds of opportunities for unexpected classpath and namespace
conflicts this way. I wonder if we should 'pom' these regressions as
individual sub-projects of systests so that they ran in comparative
isolation from each other.

While I'm inventing work with no one to do it, I'd propose that:

a) all the services used in the demos should be in testutils, verbatim,
and we should routinely run unit tests against those services. 

b) Some cleanup of the existing inventory would be helpful. I'll do at
least a little bit.

c) Some wiki-writing explaining the inventory would be helpful.

--benson

Re: [IMPASSIONED PLEA] Naming and structuring test materials

[Benson Margulies <bi...@gmail.com>]

I'm all in favor of #1. In my perhaps unlucky experience, I keep running
into 'standard' impls that are to quirky to reuse. 


On Mon, 2007-12-17 at 09:49 -0500, Daniel Kulp wrote:
> Fred,
> 
> On Friday 14 December 2007, Fred Dushin wrote:
> > I understand why the testutils stuff is there, but I don't like it.
> > The idea is to shorten build times, by putting a common set of
> > functionality in a JAR, and re-use to your heart's content.
> 
> The idea isn't just to shorten the build time, although that's a great 
> benifit.  There are two main reasons:
> 
> 1) Provide a set of standard "Impls" that could be reused by tests so you 
> don't need to write a bunch of new Impls for everything.   For example, 
> simple, jaxws, js, etc...  all need some simple server side impls to do 
> various testing.   I don't believe in copying the same code all over the 
> place.   That kind of destroys the whole idea of code reuse.
> 
> 2) Related to (1) is the problems that copying the code all over the 
> place has on IDE's, specifically eclipse.   When building with maven, it 
> doesn't put test classes from one project onto the paths of other 
> projects.   However, the way eclipse:eclipse sets up the eclipse 
> projects, it does in eclipse.  Thus, you end up with two classes with 
> the same names on the classpath and if they get modified, you get random 
> failures and such.    Thus, when you copy, you would also need to make 
> sure to change all the namespaces, change the package names, etc... 
> Thus, creating even more work for your self.   You also have that issue 
> with the /wsdl directory.   If there is a /wsdl/hello_world.wsdl in 
> testutils, and you try to have a /wsdl/hello_world.wsdl in your own set 
> of tests, you may or may not get the correct one when you run your 
> tests.  (which is why I went through an renamed a bunch of /wsdl dir 
> to /wsdl_MODULE for /wsdl dirs outside of testutils).
> 
> 
> > My problem with that is re-use doesn't always work, so you get what
> > you observe -- lots of test-specific functionality in testutils.
> > (Look at hello_world_secure.wsdl, for an example).
> 
> Well, if it's specific to just the system tests, put it in the system 
> tests.   Don't put it in testutils.     The point of testutils was to be 
> a place for SHARED stuff.   If it's not shared, don't put it there.   
> Or, import the stuff from testutils.   (The catalog stuff should all be 
> working now.   You can import via testutils/hello_world.wsdl or 
> similar).
> 
> 

Re: [IMPASSIONED PLEA] Naming and structuring test materials

[Daniel Kulp <dk...@apache.org>]

Fred,

On Friday 14 December 2007, Fred Dushin wrote:
> I understand why the testutils stuff is there, but I don't like it.
> The idea is to shorten build times, by putting a common set of
> functionality in a JAR, and re-use to your heart's content.

The idea isn't just to shorten the build time, although that's a great 
benifit.  There are two main reasons:

1) Provide a set of standard "Impls" that could be reused by tests so you 
don't need to write a bunch of new Impls for everything.   For example, 
simple, jaxws, js, etc...  all need some simple server side impls to do 
various testing.   I don't believe in copying the same code all over the 
place.   That kind of destroys the whole idea of code reuse.

2) Related to (1) is the problems that copying the code all over the 
place has on IDE's, specifically eclipse.   When building with maven, it 
doesn't put test classes from one project onto the paths of other 
projects.   However, the way eclipse:eclipse sets up the eclipse 
projects, it does in eclipse.  Thus, you end up with two classes with 
the same names on the classpath and if they get modified, you get random 
failures and such.    Thus, when you copy, you would also need to make 
sure to change all the namespaces, change the package names, etc... 
Thus, creating even more work for your self.   You also have that issue 
with the /wsdl directory.   If there is a /wsdl/hello_world.wsdl in 
testutils, and you try to have a /wsdl/hello_world.wsdl in your own set 
of tests, you may or may not get the correct one when you run your 
tests.  (which is why I went through an renamed a bunch of /wsdl dir 
to /wsdl_MODULE for /wsdl dirs outside of testutils).


> My problem with that is re-use doesn't always work, so you get what
> you observe -- lots of test-specific functionality in testutils.
> (Look at hello_world_secure.wsdl, for an example).

Well, if it's specific to just the system tests, put it in the system 
tests.   Don't put it in testutils.     The point of testutils was to be 
a place for SHARED stuff.   If it's not shared, don't put it there.   
Or, import the stuff from testutils.   (The catalog stuff should all be 
working now.   You can import via testutils/hello_world.wsdl or 
similar).


-- 
J. Daniel Kulp
Principal Engineer, IONA
dkulp@apache.org
http://www.dankulp.com/blog

Re: [IMPASSIONED PLEA] Naming and structuring test materials

[Fred Dushin <fr...@dushin.net>]

On Dec 14, 2007, at 11:01 AM, Benson Margulies wrote:

> We have about 10 copies of the hello world schema floating around the
> tree.
>
> The idea is that wsdl-first and code-first test schemas and
> implementation should live in testutils. The problem, from my  
> jaundiced
> point of view, is that we have been dumping things into there without
> enough care in picking names.

I agree.

>
> For example, someone put a copy of the hello_world 'greeter' schema  
> into
> testutils, but modified it to use the XML binding.
>
> In my opinion, it should have been renamed and renamespaced to in the
> process. The current state is unproductive in two ways.
>
> First, new people looking for a test case schema are prone to do  
> what I
> did, which is see it, try to use it, and then belatedly run into  
> the XML
> binding specification. And then have to back up and turn around.
>
> Second, it turns the testutils into a confusing arena in which no one
> really knows what we have for inventory.

I understand why the testutils stuff is there, but I don't like it.   
The idea is to shorten build times, by putting a common set of  
functionality in a JAR, and re-use to your heart's content.

My problem with that is re-use doesn't always work, so you get what  
you observe -- lots of test-specific functionality in testutils.   
(Look at hello_world_secure.wsdl, for an example).

What would be better (IMO) would be to possible put base schema and  
logical WSDL types in testutils, and then have the actual tests  
import these schema and WSDL from system tests, but only if it makes  
sense.  You'd still "pay" for code generation during the build, but  
that's life in the fast lane.  The current approach leads to spaghetti.


>
> It looks to me as if another process is to take some example we get  
> in a
> test case, and check it in, 'as is'. I can see the reason to do this:
> ultimately, the user's test is the regression test. However, there are
> all kinds of opportunities for unexpected classpath and namespace
> conflicts this way. I wonder if we should 'pom' these regressions as
> individual sub-projects of systests so that they ran in comparative
> isolation from each other.
>
> While I'm inventing work with no one to do it, I'd propose that:
>
> a) all the services used in the demos should be in testutils,  
> verbatim,
> and we should routinely run unit tests against those services.

Wouldn't there me a maintenance issue there?  I'd rather see the  
sample programs run as part of a build.

>
> b) Some cleanup of the existing inventory would be helpful. I'll do at
> least a little bit.

That would be good.  I'd offer to help with the security stuff, but  
I'm flat out right now.

>
> c) Some wiki-writing explaining the inventory would be helpful.

RoTR would be good in this area.

>
> --benson
>
>
>