You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Al Byers <by...@automationgroups.com> on 2006/11/01 14:24:49 UTC
OFBiz Testing Initiative
One of the presentations at the Users Conference will be me talking on my
experiences with Grinder. With David there, we hope to go far past that to
discuss strategies for adding testing functionality to OFBiz. I have no
experience that qualifies me to be an expert in this area - just have a
need. Rather than just wait for everyone to show up at the conference, I
felt it would be helpful to try to treat some of the subjects ahead of time.
Of course, not everything will be decided at the conference, so as much
input as possible from everyone will be needed. Also, if there is anyone
with strong Python and/or Grinder skills, their help would also be
appreciated.
I have attached the whitepaper HTML file, but I am also going to paste it in
here.
-Al Byers
<H1 align="center">OFBiz Testing Initiative</H1>
<ul>
<li><a href="#goal">GOAL</a></li>
<li><a href="#background">BACKGROUND</a></li>
<ul>
<li><a href="#specs">Problem with Design Documents</a></li>
<li><a href="#features">Applicable Features of OFBiz</a></li>
<li><a href="#grinder">Applicable Features of Grinder</a></li>
<li><a href="#xp">Applicable Features of Extreme Programming</a></li>
</ul>
<li><a href="#requirements">REQUIREMENTS</a></li>
<ul>
<li><a href="#generation">Easy Generation</a></li>
<li><a href="#complex">Handle Complex Tests</a></li>
<li><a href="#adaptable">Adaptable per Application</a></li>
<li><a href="#rugged">Rugged</a></li>
<li><a href="#iterations">Handle Iterations</a></li>
<li><a href="#proficiency">Usable at Multiple Proficiency Levels</a></li>
<li><a href="#output">Simplify Output Testing</a></li>
</ul>
<li><a href="#plan">STRAWMAN PLAN</a></li>
<ul>
<li><a href="#modify">Modify Test Script Generation</a></li>
<li><a href="#higher">Add Test Overlay Config File</a></li>
<li><a href="#scc">Source Code Control to Handle Iterations</a></li>
</ul>
<li><a href="#appendices">Appendices</a></li>
<ul>
<li><a href="#stdcode">Standard Generated Grinder Test Script</a></li>
<li><a href="#customcode">Possible Custom Generated Grinder Test
Script</a></li>
<li><a href="#overlay">Possible Overlay XML File</a></li>
</ul>
</ul>
<H2><a name="goal">GOAL</a></H2>
The goal of the OFBiz Testing Initiative (OFBTI) is to add functionality to
OFBiz that will streamline
the functional testing process to the point that it will be cost-effective
and beneficial to write
comprehensive tests. Ideally, the tools could be used by non-programmers.
<p/>
A secondary goal would be to add tools to make design specs easier to write
and
more useful.
<H2><a name="background">BACKGROUND</a></H2>
<H3><a name="specs">Problem with Design Documents</a></H3>
Design documents, though necessary, have serious drawbacks.
<ul>
<li><b>Expensive to create</b> - should probably take as much time as the
implementation.<li>
<li><b>Never complete enough</b> - always need to go back to client,
anyway.<li>
<li><b>No automation help in implementation and validation</b> - as text
documents,
they must be manually translated into architecture documents.<li>
<li><b>Easy to get out-of-date</b> - while many times they are supposed to
be "living" documents,
in reality, they almost never are.<li>
</ul>
<H3><a name="xp">Applicable Features of Extreme Programming</a></H3>
Extreme programming addresses many of the shortcomings of the more
conventional design document approach.
Instead of a complete design doc or even use case up front, all that is
required is a "story".
The story needs only be complete enough to generate the first test and the
first test is usually just a placeholder.
Then in the course of many iterations, the functional test is enhanced, not
the design document.
The tests are living and serve the useful purpose of verifying that things
have not broken as code gets changed.
<p/>
One of the very useful things about this approach is that when bugs are
discovered, a test or subtest is added to
guard against it ever creeping back into the code.
<H3><a name="features">Applicable Features of OFBiz</a></H3>
OFBiz is different from other development environments, and it would be good
to identify the features that could be used
advantageously or which must be dealt with.
<p/>
OFBiz is highly configurable via XML formatted files and it would be good to
continue that pattern.
XML config files, because they have associated schema, make it feasible for
non-programmers and those not intimately
familiar with OFBiz to make changes and perform certain programming tasks.
<p/>
OFBiz has a general pattern of offering easily configurable options via data
parameters, but always offering easy
access to the lowest level of programming for those needed cases without
requiring a huge environment adaptation.
For instance, the screen widget config files allow screens to be created
with miminal data and allow the user to be
prompted via the schema in an XML editor, but if there is a construct that
the widgets do not handle, the user
can just throw in a call to an HTML component or a FreeMarker template -
they do not need to abandon the screen widget framework.
<p/>
OFBiz is a service oriented framework and there are many tools that can be
used aid in that process.
If a request is processed by a service (in lieu of an event) then the input
parameters can be automatically taken from
the HTTP stream and automatically converted to the right type. Also, the
form widget can build forms from the service definition.
<H3><a name="grinder">Applicable Features of Grinder</a></H3>
There are multiple options for testing frameworks, but Grinder offers the
following unique advantages.
<ul>
<li><b>A robust test script generator</b> - TCPProxy attempts to assign
return parameters to variables and
reuse them, rather than passing literals around. This means that it does
not instantly break when keys are autogenerated.</li>
<li><b>Grinder has convenient customization point</b> - script generation is
done via an XSLT stylesheet that makes use of Java extensions.
This would be a natural place to make modifications to suit OFBTI. There
are also filters for the requests and responses that can be
swapped out via command line parameters.</li>
<li><b>Jython</b> - allows seamless switching between a highly productive
scripting environment and regular Java code.
Because of this feature, Jython has a lot of possibilities within OFBiz.
</li>
<li><b>Integrated functional and jUnit testing</b> - the use of Jython
allows for the same environment to be used to run HTTP client system tests
and jUnit functional tests.</li>
</ul>
<H2><a name="requirements">REQUIREMENTS</a></H2>
<H3><a name="generation">Easy Generation</a></H3>
The use of a test script generator, such as Grinder's TCPProxy, allows test
scripts to be created by capturing the
input of a user at a browser.
<H3><a name="complex">Handle Complex Tests</a></H3>
A further enhancement would be to allow the creation of scripts by supplying
a few data parameters or by
easily modifying automatically generated scripts.
One of the biggest drawbacks of end user testing environments is that when
something changes, the associated test script
is usually unusable.
A big improvement would be to allow tests to be created by chaining together
subtests.
<H3><a name="adaptable">Adaptable per Application</a></H3>
Many applications will have special characteristics that need to be handled
specially.
In one instance, I found that Grinder was using the '$' character to form
Python variable names (which doesn't work) because
that is how the HTTP parameters were named.
<H3><a name="rugged">Rugged</a></H3>
<H3><a name="iterations">Handle Iterations</a></H3>
The general idea of XP is that the tests become more complex and
comprehensive as the code becomes more complete.
It would probably be a good idea to retain visibility to past test setups -
though, at this time, I am not sure why.
<H3><a name="proficiency">Usable at Multiple Proficiency Levels</a></H3>
It would be good to have the testing environment useable, or at least
understandable, by non-programmers.
JUnit tests would not meet this requirement.
Neither would Python scripts (though it could come close).
For project managers, an XML-based configuration environment would be
needed.
<H3><a name="output">Simplify Output Testing</a></H3>
The analysis of the HTML pages that are returned by Grinder tests is one of
the more problematic areas of end-user testing.
The more common scenarios would need to be handled by the XML-based
configuration environment mentioned above.
The foreseeable cases would be matching of form values, existence or
non-existence anywhere of a phrase or something
that can be tested by an XPath expression.
<H2><a name="plan">STRAWMAN PLAN</a></H2>
What is discussed below is how I see tackling the testing problem.
I am only interested in what will be supported by the community, so it can
be changed.
<p/>
The overall approach is to use as much of Grinder as possible, and allow for
the generated Grinder test scripts to be
overwritten by a user generated XML test configuration file.
<H3><a name="modify">Modify Test Script Generation</a></H3>
The Grinder TCPProxy program will be used, but the XSLT stylesheet and the
associated Java extension classes will be enhanced.
The main change will be that the individual request tests that Grinder
generates for each round trip to the server will be
wrapped by code that will check to see if there are data overrides to be
made coming from the user generated high level test config file.
Each wrapper will also have a dictionary (ie. map) that defines tests to be
made on each low level result.
These also will be modified by the higher level config file.
If there is not higher level config file, the script as generated by
TCPProxy will run with no modifications.
<H4><a name="inoutmaps">Input Dictionary</a></H4>
The input directory will do more than just allow the user to supply literal
values as test input.
There will be helper functions for allowing the user to randomly pick values
from a list.
Also there will be helper functions for generating reasonably looking
addresses.
The user will be able to create Python scripts to generated special input.
See the appendix for samples of what this would look like.
<H4><a name="inoutmaps">Output Dictionary</a></H4>
The output dictionary, which contains the test criteria, would have test
helper functions of the following sort:
<ul>
<li><b>Literal value</b></li>
<li><b>Regular expression</b></li>
<li><b>Form value</b></li>
<li><b>XPath</b></li>
<li><b>Custom Jython scripts</b></li>
</ul>
Many of the test methods will be suitable for non-programmers, but the use
of Jython scripts follows the OFBiz pattern
of providing simple methods for simple tasks, but making it easy to drop
down to the level needed to solve the problem.
At some point (right away?), it will be necessary to allow complex joing of
tests with AND, OR and NOT operators.
<H3><a name="higher">Add Test Overlay Config File</a></H3>
One of the modifications to the OOTB test script generated by TCPProxy will
be that the script will look for the existence
a file path as a parameters and use it to overwrite the default test script
values (ie. the values typed in by the user
when the script was being generated).
The script could have multiple levels of complex scripts, but eventually,
they must call one of the "request" tests.
Keep in mind that the base test script could have a large number of
"request" scripts, but they would not all need
to be used by the user-defined config file; it could just use a subset. So
the base test script may not, in fact, be totally
generated by one session of user interaction with the system under test; it
could be built up over time as new functionality
is added without having to run thru the test. The OOTB behavior is to
sequentially number the tests, but in this case,
we may wish to name them with the request name that they interact with.
<p/>
These high level test scripts would need "include" functionality so that
they could used standard building blocks.
<H3><a name="scc">Source Code Control to Handle Iterations</a></H3>
<H2><a name="appendices">Appendices</a></H2>
<H3><a name="stdcode">Standard Generated Grinder Test Script</a></H3>
This is the actual output from a TCPProxy session:
<p/>
<pre>
# The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)
# These definitions at the top level of the file are evaluated once,
# when the worker process is started.
connectionDefaults.defaultHeaders = \
( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
NVPair('Accept-Encoding', 'gzip,deflate'),
NVPair('Accept-Language', 'en-us,en;q=0.5'),
NVPair('Accept-Charset', 'UTF-8,*'),
NVPair('Accept',
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
,text/plain;q=0.8,image/png,*/*;q=0.5'), )
headers0= \
( )
headers1= \
( NVPair('Referer', 'https://localhost:8443/webtools/control/main'), )
headers2= \
( NVPair('Referer', '
https://localhost:8443/webtools/control/checkLogin/main'), )
url0 = 'https://localhost:8443'
# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)
request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)
request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)
request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)
class TestRunner:
"""A TestRunner instance is created for each worker thread."""
# A method for each recorded page.
def page1(self):
"""GET / (requests 101-102)."""
# Expecting 302 'Moved Temporarily'
result = request101.GET('/webtools/')
grinder.sleep(16)
request102.GET('/webtools/control/main')
return result
def page2(self):
"""GET main (request 201)."""
result = request201.GET('/webtools/control/checkLogin/main')
return result
def page3(self):
"""POST login (request 301)."""
result = request301.POST('/webtools/control/login',
( NVPair('USERNAME', 'admin'),
NVPair('PASSWORD', 'ofbiz'), ),
( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
return result
def __call__(self):
"""This method is called for every run performed by the worker
thread."""
self.page1() # GET / (requests 101-102)
grinder.sleep(4328)
self.page2() # GET main (request 201)
grinder.sleep(2109)
self.page3() # POST login (request 301)
def instrumentMethod(test, method_name, c=TestRunner):
"""Instrument a method with the given Test."""
unadorned = getattr(c, method_name)
import new
method = new.instancemethod(test.wrap(unadorned), None, c)
setattr(c, method_name, method)
# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
</pre>
<H3><a name="custmcode">Possible Custom Generated Grinder Test
Script</a></H3>
<pre>
# The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
import AGrinderTest
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)
# These definitions at the top level of the file are evaluated once,
# when the worker process is started.
connectionDefaults.defaultHeaders = \
( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
NVPair('Accept-Encoding', 'gzip,deflate'),
NVPair('Accept-Language', 'en-us,en;q=0.5'),
NVPair('Accept-Charset', 'UTF-8,*'),
NVPair('Accept',
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
,text/plain;q=0.8,image/png,*/*;q=0.5'), )
headers0= \
( )
headers1= \
( NVPair('Referer', 'https://localhost:8443/webtools/control/main'), )
headers2= \
( NVPair('Referer', '
https://localhost:8443/webtools/control/checkLogin/main'), )
url0 = 'https://localhost:8443'
# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)
request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)
request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)
request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)
class TestRunner:
"""A TestRunner instance is created for each worker thread."""
# A method for each recorded page.
def page1(self):
"""GET / (requests 101-102)."""
# Expecting 302 'Moved Temporarily'
result = request101.GET('/webtools/')
grinder.sleep(16)
request102.GET('/webtools/control/main')
return result
def page2(self):
"""GET main (request 201)."""
result = request201.GET('/webtools/control/checkLogin/main')
return result
def page3(self):
"""POST login (request 301)."""
result = request301.POST('/webtools/control/login',
( NVPair('USERNAME', 'admin'),
NVPair('PASSWORD', 'ofbiz'), ),
( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
return result
"""This is not working code. Just an idea of how the generated code
would look"""
def webtoolsWrap(self, overlayMap):
inMap = {}
outMap = {}
self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap)
def mainWrap(self, overlayMap):
inMap = {}
outMap = {}
self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)
def loginWrap(self, overlayMap):
inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'}
outMap = {}
self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)
def __call__(self):
"""This method is called for every run performed by the worker
thread."""
"""I am going to do some handwaving here because I want to get this doc
out today,
but there would be code here to read in an argument from the command
line and
use it as a file path to read in the "test overlay" XML doc"""
self.agrinder = AGrinderTest.AGrinder()
testNode = self.agrinder.getOverlayTest("webtools", sys.argv)
self.webtoolsWrap(testNode) # GET / (requests 101-102)
grinder.sleep(4328)
self.mainWrap(testNode) # GET main (request 201)
grinder.sleep(2109)
self.loginWrap(testNode) # POST login (request 301)
def instrumentMethod(test, method_name, c=TestRunner):
"""Instrument a method with the given Test."""
unadorned = getattr(c, method_name)
import new
method = new.instancemethod(test.wrap(unadorned), None, c)
setattr(c, method_name, method)
# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
</pre>
<H3><a name="overlay">Possible Overlay XML File</a></H3>
This example is probably not even close to what the final test overlay XML
script will look like,
but I think it is better to have something to go from.
<pre>
<tests>
<test name="webtools">
<subtest name="login">
<input>
<field name="USERNAME">jdoe</field>
<field name="PASSWORD">id9Ed3jk</field>
</input>
<match>
<field type="regex" op="not">not found</field>
<field type="script" op="true">isSuccess()</field>
</match>
</subtest>
</test>
</tests>
</pre>