You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by to...@apache.org on 2005/01/27 10:43:07 UTC
cvs commit: ws-axis/site/src4forrest-060/java/src/documentation/content/xdocs/java/docbook testing-again.dbk
toshi 2005/01/27 01:43:07
Added: site/src4forrest-060/java/src/documentation/content/xdocs/java/docbook
testing-again.dbk
Log:
Migration for Forrest 0.6
Revision Changes Path
1.1 ws-axis/site/src4forrest-060/java/src/documentation/content/xdocs/java/docbook/testing-again.dbk
Index: testing-again.dbk
===================================================================
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.0/docbookx.dtd">
<article>
<title>Testing (again)</title>
<authorblurb>
<para><author><lineage>Steve Loughran</lineage></author></para>
</authorblurb>
<sect1>
<title>Where we are today?</title>
<para>We are now at the second revision of the Axis test framework. The
original design had all classes in a package under java/test, built them
in one big <javac> and then executed them. While it ran the tests,
it was not that flexible. It was hard to run individual tests, and it
was hard to maintain. A major effort by Matt Siebert refactored the test
process to address these.</para>
<para>The revision attempted to address this with a modular design, based
on separate Ant build files for each test package, using common XML entity
references to share build file fragments between the files. This gave us
isolated builds of each subcomponents, the ability to build and run tests
on their own, and the flexiblity to have very different build processes
for each of the tests.</para>
<para>The many build files compile the source, all of java/test/*.java
into build/classes, putting them into the hierarchy build/classes/test.
Test packages which have special dependencies can make their builds
conditional, so only those tests for which we have support get compiled
down. This architecture makes it easy to add a new test package, without
having to edit shared build files.</para>
<para>We have a separation between "unit tests" and
"functional" tests; the latter includes the interop and attachment
tests. There are separate targets in build test to build them, as the
choice of which tests to execute is primarily driven by the compilation
process. Nearly all the tests in the class tree will get executed, so to
select which tests to run, you control which tests get built. This is a
simple way of letting the test package-specific build files control which
test to run.</para>
<sect2>
<title>The Tests</title>
<para>The many build files compile the source, all of java/test/*.java
into build/classes, putting them into the hierarchy build/classes/test.
Test packages which have special dependencies can make their builds
conditional, so only those tests for which we have support get compiled
down.</para>
<para>We have a separation between "unit tests" and
"functional" tests; the latter includes the interop and
attachment tests. There are separate targets in build test to build
them, as the choice of which tests to execute is primarily driven by the
compilation process. Nearly all the tests in the class tree will get
executed, so to select which tests to run, you control which tests get
built. This is a simple way of letting the test package-specific build
files control which test to run.</para>
</sect2>
<sect2>
<title>WSDL</title>
<para>A core component of many of the tests is generating Java source
from WSDL files, both local test case WSDL and remote interop WSDL. The
latter introduces a requirement to be on line, and on-line through a
transparent firewall -we dont look after proxy settings enough to run
behind a firewall whose sole net access is via a proxy 80. This is
somewhat ironic, given that such a facility is the selling point of the
transport-stack-atop-port-80 that is the SOAP and WS-* specification
suite.</para>
<para>As well as lacking off-line support for generating WSDL, we cant
(obviously) run the interop tests without a network connection. This
means that when a remote interop server goes down, the build fails.</para>
</sect2>
<sect2>
<title>Execution</title>
<para>After compiling all the code down, we run the tests. This is done
by batch JUnit execution of all the test suites in all the packages with
a PackageTests.class class in their package (i.e. all of
build/classes/**/PackageTests.class).</para>
<para>Functional tests are all of **/FunctionalTests.class and
**/*TestCase.class; the latter are those test cases which are
auto-created by the Wsdl2Java routine, often with manual editing to make
the test complete.</para>
<para>When the tests need a functional servlet engine to host the web
services, we bring up the simple axis server; a minimal implementation
of the servlet API that omits the production-quality aspects of a web
server, including JSP support. The <runaxisfunctionaltests> task
starts and stops the server, using an execution process borrowed from
Cactus: we supply the task with the target names of the start and stop
targets, and the task executes them before and after running all the
functional tests.</para>
</sect2>
<sect2>
<title>Result Processing</title>
<para>In a Gump build, the build stops after the first failure, and the
team notified. The property <varname>test.functional.fail</varname> sets
the <symbol>haltonfailure</symbol> attribute of the <junit>
task; set it to true and the test suite runs all tests before
completing. Either way, the <symbol>create-test-report</symbol> target
will, if Xalan or other XSLT engine is present, convert the XML reports
of the test run into an HTML report, package by package.</para>
</sect2>
</sect1>
<sect1>
<title>What do we want from a test suite?</title>
<sect2>
<title>Basic Improvements to the current status quo</title>
<itemizedlist>
<listitem>
<para>All the tests to pass :)</para>
</listitem>
<listitem>
<para>Faster tests</para>
</listitem>
<listitem>
<para>Scalability: easy to add new tests</para>
</listitem>
<listitem>
<para>Offline support, and robustness against unavailable interop
servers.</para>
</listitem>
</itemizedlist>
</sect2>
<sect2>
<title>Functional testing on production app servers</title>
<para>If we look at a lot of bugreps they are related to config and
operations on app servers. "Weblogic doesnt save
server-config.wsdd", "SunOne server has the wrong
classpath", "Jboss.net won't compile .JWS pages that import
javax.servlet.*", etc, etc. We need to run more tests on production
systems, rather than wait for user feedback after we ship. Now everybody
runs their apps on some such systems, so we have implicit testing, but
it is not part of the daily Gump or any other regular, controlled test
process.</para>
<para>We could modify the test system so that instead of starting the
SimpleAxisServer servlet routine, we can deploy to a local web server or
app server. This would verify that the core test suite runs on different
systems.</para>
</sect2>
<sect2>
<title>Test more than SOAP</title>
<para>We need more tests to validate the configuration; extending the
httpunit tests to have more tests of not-quite-SOAP requests. What
happens when the server gets less than they were promised? What about
more than promised? near-infinite recursive XML? xsd:import statements
in the XML? What happens when a client starts parsing from a socket that
doesnt close its connection, or lies about the amount it is sending?</para>
<para>These are the security and robustness categories we aren't
testing for today.</para>
</sect2>
<sect2>
<title>Automated invocation of compliance test suites: JAX-RPC TCK, WS-I
Basic</title>
<para>We have one test suite, JAX-RPC, that is only available under
restricted conditions. We need someone with access to the test suite to
run it.</para>
</sect2>
<sect2>
<title>Understandig that Interop servers are regularly unavailable</title>
<para>If Axis depends on everyone's interop server to be present,
then we have a global build system that breaks every time somebody in
their machine off -"the light switch in belgium problem". This
is too brittle. We need to cache the external WSDL in CVS, then probe
the servers to see if that has changed, downloading it only if it is
different. It would be nice to only regenerate the java proxy classes
from the WSDL when such a change has occurred.</para>
</sect2>
<sect2>
<title>Load testing</title>
<para>What happens to the system under load? This is very dependent upon
the app server; having tests running on the production server is a first
step to this. Traditional load testing has N clients each making M
requests, for N*M simultaneous requests. The facilities for individuals
to perform aggressive load tests are limited, but there is strength in
numbers; many Axis developers could have their test systems sychronised
to test an externally visible server together. This co-ordination could
be though a P2P infrastructure, such as JXTA or Chord, but as we are not
trying to design a stealth DDoS system, we could do it client-server
with a (separate) SOAP service choreographing the clients.</para>
<para>This would seem to be a long term project.</para>
</sect2>
<sect2>
<title>Performance testing</title>
<para>This is related to load testing, but doesnt need as many clients.
We need to know how long it takes for actions to be executed, and we
need to log this over time so that major regressions can get picked up.
If on the daily run one test takes 50% longer than usual, it is
immediately obvious that one of the day's changes caused the
problem. If the performance of the system doesn't get looked at till the
next version goes into RC phase, performance slippages get missed, and
even institutionalised.</para>
</sect2>
<sect2>
<title>Coverage Analysis</title>
<para>We should be able to use quilt
(<ulink url="http://quilt.sourceforge.net/">http://quilt.sourceforge.net/</ulink>)
to do this today. As with performance tests, tracking coverage changes
over time is informative; it shows whether things are improving or
getting worse.</para>
</sect2>
<sect2>
<title>Local interop testing with non-Axis clients and servers</title>
<para>We already have some examples of .net client tests against axis,
with an Ant build but sadly no automatic .NUnit test case generation. We
can also build axis clients against .Net and other servers, where we can
create JUnit stubs for ease of test generation. If such tests were part
of the main test suite, then suitably equipped systems could run our own
interop tests. These would be an extension of the SoapBuilders main
suite. Here we"d want to verify that fixes worked and continued to
work (e.g. the .NET1.0 empty array bugfix). We can also add stuff that
isn't in the SoapBuilders: Cookie Sessions, Http Header sessions,
is-date-in-the-right-TZ-at-the-far-end tests, and so on.</para>
<para>There are logistical/tactical and strategic arguments against
doing this. Logistical: even for the example of one client platform is
daunting; we don't want to expose a .NET server to the internet for
anyone to hit it, so the tests will only run on localhost when
localhost=windows, which excludes the Gump builds.</para>
<para>The strategic argument is that the combinatorial explosion of
local interop testing against multiple clients and servers is too big;
that is what the SoapBuilders are for. Either we focus on one or two key
platforms to interop test against -.net and MSSTK, or we raise the
problem back to SoapBuilders.</para>
<para>What would we want from SoapBuilders, to help our regression and
interop problems? I'd argue for extra tests, above and beyond the
formal "rounds", wherever someone has an interop issue. We
should be able to announce that we have a problem and the URL of a test
endpoint, and everyone can add it to the things we can test against.
Similarly, other platforms should not just fix things, they should
provide means for outsiders to test the system.</para>
<para>Glen Daniels has proposed a pattern-matching server that waits for
expected wire traces, and generates preprogrammed responses, simulating
part of the behaviour of a foreign server. This has the advantage of
being standalone, but with the disadvantage of not being as thorough as
a live test. You also have the challenge of keeping the datasets up to
date.</para>
</sect2>
<sect2>
<title>Wiretrace logging in the test case results</title>
<para>This is just an extra little feature for diagnosis: can we record
the wire traces of sent and received messages, and whenever we get a
test failure, save the results to the JUnit log for an easier
post-mortem. Just a thought :)</para>
<para></para>
</sect2>
<sect2>
<title>Ease of learning, installation, use</title>
<para>We are an open source project where anyone can download the
source, build it and run the tests. Therefore the test framework must be
easy to run, easy to work with, and easy to maintain by people other
than the original authors. We also want to keep effort minimised by
re-using as much as possible of other OSS projects.</para>
</sect2>
</sect1>
<sect1>
<title>Options</title>
<para>Here are some of the things we can do</para>
<sect2>
<title>Nothing</title>
<para>Leave things as they are. Maybe move to Ant1.6 alpha builds to get
better memory management, the faster build.xml parser and the refactored
.NET tasks, or just up to 1.5.3/1.5.4 to get the memory leak fix.</para>
<para>Costs: nothing at first; a gradual increase in longer term costs.</para>
</sect2>
<sect2>
<title>Improve build.xml reuse</title>
<para>We don't necessarily need a separate build file in every test
package. Instead we can have a properties file in there that sets well
known properties</para>
<programlisting>package=test/example
conditions=httpunit.present
online=false
needsserver=true
functional=true</programlisting>
<para>This can be read in by something (ant or custom java) and used to
control the build. Reading it into pure ant (i.e. without writing our
own task) would be tricky as condition expressions are tricky. An XML
description might be better, and could be XSLT'd into the build
files.</para>
</sect2>
<sect2>
<title>Caching WSDL Generation</title>
<para>This is a trick we can do with any of these.</para>
<para>Write a new <axis-importwsdl> task that implements
dependency awareness around the inner generation process. We could go
one step further and integrate dependency logic into the generation
process, but as that is more fundamental, I am a bit leery of that.
</para>
<orderedlist>
<listitem>
<para>caches the results of the fetch.</para>
</listitem>
<listitem>
<para>uses the if-modified-since tag to conditionally retrieve
content</para>
</listitem>
<listitem>
<para>even if content is returned, compares it byte-for-byte against
the cached copy</para>
</listitem>
<listitem>
<para>only imports the wsdl if the wsdl file is newer than a
timestamp marker file in the generated dir (and a force option is
false)</para>
</listitem>
<listitem>
<para>if the server is unreachable, but the cached copy is present,
don't fail the build, just set a property saying the server isn't
there and continue with the WSDL generation.</para>
</listitem>
<listitem>
<para>if the server is unreachable and the cached copy isn't there,
only fail the build if some attribute is set, otherwise a
wsdlnotpresent property is set</para>
<para>We could do this over a fairly convoluted set of ant 1.6
tasks, but only if the httptasks in the Ant sandbox were pulled in for
more graceful handling of missing servers. Pulling it in to one Axis
task gives us more control, ant1.5 support and wouldn't be that
hard to implement.</para>
</listitem>
</orderedlist>
</sect2>
<sect2>
<title>Write our own test harness</title>
<para>This deserves a mention: could we do our own complete test
harness? Why? is the reponse. What would that add?</para>
<para>In theory, having our own hosting app would let us run tests
differently from core JUnit, doing more SOAP related things like post
XML and expect responses.</para>
</sect2>
<sect2>
<title>Return to being JUnit-centric</title>
<para>The advantage of an ant-centric test system is that we can use Ant
to build stuff during testing. The disadvantage is in complexity and
time in running the tests. Is there a better compromise? Maybe. </para>
<para>It is possible to run Ant from inside JUnit. This is how Ant runs
its many self tests; by invoking Ant from JUnit itself. If we
put JUnit in charge, then those tests that do need a complex Ant-based
test system can call Ant to aid it, the rest run straight from JUnit.</para>
<para>We may be able to take advantage of this by categorising tests
into various patterns that we can build and run differently.</para>
<orderedlist>
<listitem>
<para>Pure unit tests that compile and run without needing any
server at all</para>
</listitem>
<listitem>
<para>WSDL-importing tests that need to import WSDL and generate
code before the tests run</para>
</listitem>
<listitem>
<para>Local functional tests that run against an instance of the
Axis running on a servlet engine</para>
</listitem>
<listitem>
<para>Local functional tests that only run on a full J2EE app server</para>
</listitem>
<listitem>
<para>Remote interop tests</para>
</listitem>
</orderedlist>
<para>Clearly this stuff is not 100% exclusive: a lot of the local
functional tests generate WSDL first, as do all the interop tests. And
there are other flags which will include/exclude test items: the
presence/absence of needed libraries, such as attachment support, and an
online/offline flag to distinguish tests that need a full internet
connection, from those that don't. All the interop tests are online, but
so are a few of the others.</para>
<para>In a JUnit-centric world, first the local unit tests would get
built and run, all in in a couple of big <javac> and
<junit> task. Then the WSDL import process can take place, using
something like the new dependency-aware wsdl import task proposed
earlier. </para>
</sect2>
</sect1>
</article>