You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commons-dev@ws.apache.org by ve...@apache.org on 2008/11/11 01:11:31 UTC
svn commit: r712883 -
/webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java
Author: veithen
Date: Mon Nov 10 16:11:31 2008
New Revision: 712883
URL: http://svn.apache.org/viewvc?rev=712883&view=rev
Log:
Transport test kit: Added Javadoc with an introduction and high level overview of the test kit.
Added:
webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java (with props)
Added: webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java
URL: http://svn.apache.org/viewvc/webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java?rev=712883&view=auto
==============================================================================
--- webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java (added)
+++ webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java Mon Nov 10 16:11:31 2008
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * Transport test kit base package.
+ *
+ * <h2>Introduction and high level overview</h2>
+ *
+ * In general a good test suite for an Axis2 transport should contain test cases that
+ * <ul>
+ * <li>test the transport sender in isolation, i.e. with non Axis2 endpoints;</li>
+ * <li>test the transport listener in isolation, i.e. with non Axis2 clients;</li>
+ * <li>test the interoperability between the transport sender and the transport listener.</li>
+ * </ul>
+ * In addition, the test suite should cover
+ * <ul>
+ * <li>different message exchange patterns (at least one-way and request-response);</li>
+ * <li>different content types (SOAP 1.1/1.2, POX, SOAP with attachments, MTOM, plain text, binary, etc.).</li>
+ * </ul>
+ * Also for some transports it is necessary to execute the tests with different transport
+ * configurations or with different protocol providers. For example, HTTP transport implementations
+ * are tested in HTTP 1.0 and HTTP 1.1 mode, and the JMS transport is tested with different
+ * JMS providers (currently Qpid and ActiveMQ).
+ * <p>
+ * The test kit grew out of the idea that is should be possible to apply a common set of tests
+ * (with different MEPs and content types) to several transports with a minimum of code duplication.
+ * By providing non Axis2 test clients and endpoints as well as the code that sets up the
+ * necessary environment as input, the framework should then be able to build a complete test suite
+ * for the transport.
+ * <p>
+ * It is clear that since each transport protocol has its own specificities, a high level of abstraction
+ * is required to achieve this goal. The following sections give a high level overview of the
+ * various abstractions that have been introduced in the test kit.
+ *
+ * <h3>Integration with JUnit</h3>
+ *
+ * One of the fundamental requirements for the test kit is to integrate well with JUnit.
+ * This requirement ensures that the tests can be executed easily as part of the Maven
+ * build and that other available tools such as test report generators and test coverage
+ * analysis tools can be used.
+ * <p>
+ * The usual approach to write JUnit tests is to extend {@link junit.framework.TestCase.TestCase}
+ * and to define a set of methods that implement the different test cases. Since the goal of the framework
+ * is to build test suites in an automated way and the number of test cases can be fairly high, this
+ * approach would not be feasible. Fortunately JUnit supports another way to create a test suite
+ * dynamically. Indeed JUnit scans the test code for methods with the following signature:
+ *
+ * <pre>public static TestSuite suite()</pre>
+ *
+ * A typical transport test will implement this method and use {@link org.apache.axis2.transport.testkit.TransportTestSuiteBuilder}
+ * to let the framework create the test suite.
+ *
+ * <h3>Test case naming</h3>
+ *
+ * One problem that immediately arises when building a test suite dynamically is that each test
+ * case must have a name (which should be unique) and that this name should be sufficiently meaningful
+ * so that when it appears in a report a human should be able to get a basic idea of what the test case does.
+ * The names generated by the test kit have two parts:
+ * <ul>
+ * <li>A numeric ID which is the sequence number of the test case in the test suite.</li>
+ * <li>A set of key-value pairs describing the components that are used in the test case.</li>
+ * </ul>
+ * Example:
+ *
+ * <pre>0076:test=REST,client=java.net,endpoint=axis</pre>
+ *
+ * The algorithm used by the test kit to collect the key-value pairs is described in the documentation of
+ * the {@link org.apache.axis2.transport.testkit.name} package.
+ *
+ * <h3>Resource management</h3>
+ *
+ * In general setting up the environment in which a given test case is executed may be quite expensive.
+ * For example, running a test case for the JMS transport requires starting a message broker. Also
+ * every test case requires at least an Axis2 client and/or server environment to deploy the transport.
+ * Setting up and tearing down the entire environment for every single test case would be far too
+ * expensive. On the other hand the environments required by different test cases in a single test suite
+ * are in general very different from each other so that it would not possible to set up a common
+ * environment used by all the test cases.
+ * <p>
+ * To overcome this difficulty, the test kit has a mechanism that allows a test case to reuse resources
+ * from the previous test case. This is managed in an entirely transparent way by a lightweight
+ * dependency injection container (see [TODO: need to regroup this code in a single package]), so that
+ * the test case doesn't need to care about it.
+ * <p>
+ * The mechanism is based on a set of simple concepts: [TODO: this is too detailed for a high level overview and
+ * should be moved to the Javadoc of the relevant package]
+ * <ul>
+ * <li><p>Every test case is linked to a set of <em>resources</em> which are plain Java objects (that are not
+ * required to extend any particular class or implement any particular interface).
+ * These objects define the <em>resource set</em> of the test case (which is represented
+ * internally by a {@link org.apache.axis2.transport.testkit.tests.TestResourceSet}
+ * object).</p></li>
+ * <li><p>The lifecycle of a resource is managed through methods annotated by
+ * {@link org.apache.axis2.transport.testkit.tests.Setup} and {@link org.apache.axis2.transport.testkit.tests.TearDown}.
+ * These annotations identify the methods to be called when the framework sets up and tears down the resource.
+ * The arguments of the methods annotated using {@link org.apache.axis2.transport.testkit.tests.Setup} also
+ * define the <em>dependencies</em> of that resource.</p>
+ * <p>Example:</p>
+ * <pre>public class MyTestClient {
+ * \@Setup
+ * private void setUp(MyProtocolProvider provider) throws Exception {
+ * provider.connect();
+ * }
+ *}</pre>
+ * <p>As shown in this example, dependencies are specified by class (which may be abstract). The actual
+ * instance that will be injected is selected during <em>resource resolution</em>.</p></li>
+ * <li><p>Resources are (in general) resolved from the resource set of the test case. For example an instance
+ * of the <code>MyTestClient</code> class can only be used as a resource for a given test case
+ * if the resource set of this test case also contains an instance of <code>MyProtocolProvider</code>
+ * (more precisely an object that is assignment compatible with <code>MyProtocolProvider</code>).</p></li>
+ * <li><p>A resource will be reused across two test cases if it is part of the resource sets of both
+ * test cases and all its dependencies (including transitive dependencies) are part of both resource sets.
+ * The precise meaning of "reusing" in this context is using the same instance without calling the
+ * tear down and set up methods.</p>
+ * <p>For example, consider the following test cases and resource sets:</p>
+ * <table border="1">
+ * <tr><th>Test case</th><th>Resource set</th></tr>
+ * <tr><td>T1</td><td><code>c:MyTestClient</code>, <code>p1:MyProtocolProvider</code></td></tr>
+ * <tr><td>T2</td><td><code>c:MyTestClient</code>, <code>p1:MyProtocolProvider</code>, <code>r:SomeOtherResourceType</code></td></tr>
+ * <tr><td>T3</td><td><code>c:MyTestClient</code>, <code>p2:MyProtocolProvider</code>, <code>r:SomeOtherResourceType</code></td></tr>
+ * </table>
+ * <p>Assuming that <code>SomeOtherResourceType</code> is independent of <code>MyTestClient</code> and
+ * <code>MyProtocolProvider</code>, the lifecycle of the different resources will be as follows:</p>
+ * <table border="1">
+ * <tr><th>Transition</th><th>Lifecycle actions</th></tr>
+ * <tr><td>• → T1</td><td>set up <code>p1</code>, set up <code>c</code></td></tr>
+ * <tr><td>T1 → T2</td><td>set up <code>r</code></td></tr>
+ * <tr><td>T2 → T3</td><td>tear down <code>c</code>, tear down <code>p1</code>, set up <code>p2</code>, set up <code>c</code></td></tr>
+ * <tr><td>T3 → •</td><td>tear down <code>c</code>, tear down <code>p2</code>, tear down <code>r</code></td></tr>
+ * </table>
+ * <p>Even if T2 and T3 use the same instance <code>c</code> of <code>MyTestClient</code>, this resource
+ * is not reused (in the sense defined above) since the <code>MyProtocolProvider</code> dependency
+ * resolves to different instances.</p></li>
+ * </ul>
+ *
+ * <h3>Resources required by a transport test case</h3>
+ *
+ * Every transport test case (extending {@link org.apache.axis2.transport.testkit.tests.MessageTestCase})
+ * at least requires three resources:
+ * <ul>
+ * <li>A test client ({@link org.apache.axis2.transport.testkit.client.AsyncTestClient}
+ * or {@link org.apache.axis2.transport.testkit.client.RequestResponseTestClient}) that
+ * allows the test case to send messages (and receive responses).</li>
+ * <li>A test endpoint ({@link org.apache.axis2.transport.testkit.endpoint.AsyncEndpoint}
+ * or {@link org.apache.axis2.transport.testkit.endpoint.InOutEndpoint}). In the one-way case,
+ * this resource is used to receive requests send by the test client. In the request-response
+ * case its responsibility is to generate well defined responses (typically a simple echo).</li>
+ * <li>A channel ({@link org.apache.axis2.transport.testkit.channel.AsyncChannel} or
+ * {@link org.apache.axis2.transport.testkit.channel.RequestResponseChannel}. This resource
+ * manages everything that it necessary to transport a message from a client to an endpoint.
+ * Depending on the transport this task can be fairly complex. For example, in the JMS case,
+ * the channel creates the required JMS destinations and registers them in JNDI, so that
+ * they can be used by the client and by the endpoint. On the other hand, for HTTP the
+ * channel implementation is very simple and basically limited to the computation of the
+ * endpoint reference.</li>
+ * </ul>
+ * <p>The test kit provides the following Axis2 based test client and endpoint implementations:</p>
+ * <table border="1">
+ * <tr>
+ * <th></th>
+ * <th>One-way</th>
+ * <th>Request-response</th>
+ * </tr>
+ * <tr>
+ * <th>Client</th>
+ * <td>{@link org.apache.axis2.transport.testkit.axis2.client.AxisAsyncTestClient}</td>
+ * <td>{@link org.apache.axis2.transport.testkit.axis2.client.AxisRequestResponseTestClient}</td>
+ * </tr>
+ * <tr>
+ * <th>Endpoint</th>
+ * <td>{@link org.apache.axis2.transport.testkit.axis2.endpoint.AxisAsyncEndpoint}</td>
+ * <td>{@link org.apache.axis2.transport.testkit.axis2.endpoint.AxisEchoEndpoint}</td>
+ * </tr>
+ * </table>
+ *
+ * <h3>Message encoders and decoders</h3>
+ *
+ * Different clients, endpoints and test cases may have fairly different ways to "naturally" represent
+ * a message:
+ * <ul>
+ * <li>To test the listener of an HTTP transport, an obvious choice is to build a test client
+ * that relies on standard Java classes such as {@link java.net.URLConnection}. For that
+ * purpose the most natural way to represent a message is as a byte sequence.</li>
+ * <li>All Axis2 based test clients and endpoints already have a canonical message
+ * representation, which is the SOAP infoset retrieved by
+ * {@link org.apache.axis2.context.MessageContext#getEnvelope()}.</li>
+ * <li>A test case for plain text messages would naturally represent the test message
+ * as a string.</li>
+ * </ul>
+ * Since defining a message representation that would be suitable for all clients, endpoints and test
+ * cases (and make their implementation simple) is impossible, a different approach has been chosen
+ * in the framework. Every client, endpoint or test case implementation chooses the Java type that is
+ * considers as best suited to represent the message. When invoking the test client, a test case
+ * uses a {@link org.apache.axis2.transport.testkit.message.MessageEncoder} to transform the message
+ * from its own representation to the representation used by the test client. In the same way,
+ * a {@link org.apache.axis2.transport.testkit.message.MessageDecoder} is used to transform the message
+ * intercepted by the endpoint (in the one-way case) or the response message received by the test client
+ * (in the request-response case).
+ * <p>
+ * [TODO: currently message encoders and decoders are chosen at compile time and the transformation is
+ * is invoked indirectly by adapters; this will change in the future so that encoders and decoders are
+ * selected dynamically at runtime]
+ *
+ * <h3>Exclusion rules</h3>
+ *
+ * Sometimes it is necessary to exclude particular test cases (or entire groups of test cases) from the
+ * test suite generated by the test kit. There are various reasons why one would do that:
+ * <ul>
+ * <li>A test case fails because of some known issue in the transport. In that case it should be excluded
+ * until the issue is fixed. This is necessary to distinguish this type of failure from regressions.
+ * In general the tests checked in to source control should always succeed unless there is a regression.</li>
+ * <li>Sometimes a particular test case doesn't make sense for a given transport. For example a test
+ * case that checks that the transport is able to handle large payloads would not be applicable
+ * to the UDP transport which has a message size limitation.</li>
+ * <li>The test suite builder generates test cases by computing all possible combinations of MEPs, content types,
+ * clients, endpoints and environment setups. For some transports this results in a very high number of test
+ * cases. Since these test cases generally have a high degree of overlap, one can use exclusion rules
+ * to reduce the number of test cases to a more reasonable value.</li>
+ * </ul>
+ * The test kit allows to specify exclusion rules using LDAP filter expressions. It takes advantage of the
+ * fact that each test case has a set of key-value pairs used to build the test case name. The LDAP filters
+ * are evaluated against this set.
+ * For example, {@link org.apache.axis2.transport.testkit.TransportTestSuiteBuilder} defines the following
+ * default exclusion rule:
+ *
+ * <pre>(&(client=*)(endpoint=*)(!(|(client=axis)(endpoint=axis))))</pre>
+ *
+ * This rule excludes all test cases that would use a non Axis2 client and a non Axis2 endpoint.
+ *
+ * <h3>Logging</h3>
+ *
+ * Transport test cases generally involve several interacting components and some of these components
+ * may use multithreading. Also experience has shown that some test cases may randomly fail (often with
+ * a failure probablity highly dependent on the execution platform) because of subtle problems in the
+ * transport under test or in the tests themselves. All this can make debugging extremely difficult.
+ * To simplify this task, the test kit collects (or provides the necessary infrastructure to collect)
+ * a maximum of information during the execution of each test case.
+ * <p>
+ * The collected information is written to a set of log files managed by
+ * {@link org.apache.axis2.transport.testkit.util.LogManager}. An instance is added automatically to
+ * the resource set of every test case and other resources can acquire a reference through the dependency
+ * injection mechanism described above. This is the recommended approach. Alternatively, the log manager
+ * can be used as a singleton through {@link org.apache.axis2.transport.testkit.util.LogManager#INSTANCE}.
+ * <p>
+ * Logs files are written to subdirectories of <tt>target/testkit-logs</tt>. The directory structure has
+ * a two level hierarchy identifying the test class (by its fully qualified name) and the test case
+ * (by its ID). It should be noted that the test results themselves (in particular the exception in case
+ * of failure) are still written to the standard JUnit/Surefire logs and that these logs should be
+ * consulted first. The test kit specific log files are only meant to provide additional information.
+ * <p>
+ * Each test case at least produces a <tt>01-debug.log</tt> file with the messages that were logged
+ * (using JCL) at level DEBUG during the execution of the test case. In addition, depending on the
+ * components involved in the test, the test kit will produce the following logs (<tt>XX</tt>
+ * denotes a sequence number which is generated automatically):
+ * <dl>
+ * <dt><tt>XX-formatter.log</tt></dt>
+ * <dt><tt>XX-builder.log</tt></dt>
+ * <dd><p>These files are produced when Axis2 test clients and endpoints are used.
+ * <tt>XX-formatter.log</tt> will contain the payload of an incoming message as seen by the
+ * {@link org.apache.axis2.transport.MessageFormatter}. <tt>XX-builder.log</tt> on the other
+ * hand will contain the payload of an outgoing message as produced by the
+ * {@link org.apache.axis2.builder.Builder}. Note that the number of log files depends on
+ * serveral factors, such as the MEP, whether the client or endpoint is Axis2 based or not and
+ * whether the transport chooses to use message builders and formatters or not.</p>
+ * <p>These files provides extremely valuable information since it is very difficult to get this
+ * data using other debugging techniques. Note that the files are created by
+ * {@link org.apache.axis2.transport.testkit.axis2.LogAspect} which relies on Aspect/J to
+ * intercept calls to message formatters and builders. This will only work if the tests are
+ * run with the Aspect/J weaver.</p></dd>
+ * <dt><tt>XX-service-parameters.log</tt></dt>
+ * <dd><p>If the test case uses an Axis2 based endpoint, this file will contain the parameters
+ * of the {@link org.apache.axis2.description.AxisService} implementing this endpoint.
+ * This information is useful since the service configuration is in general determined
+ * by different components involved in the test.</p></dd>
+ * </dl>
+ */
+package org.apache.axis2.transport.testkit;
\ No newline at end of file
Propchange: webservices/commons/trunk/modules/transport/modules/testkit/src/main/java/org/apache/axis2/transport/testkit/package-info.java
------------------------------------------------------------------------------
svn:eol-style = native