AntEater: a functional testing framework for HTTP/SOAP apps

[Ovidiu Predescu <ov...@cup.hp.com>]

Hi,

As I mentioned last week, thanks to Jeff Turner who pointed to me the
httpClient testing framework from Tomcat, I was able to do some very good
progress in implementing a functional testing framework for Cocoon. With some
more requirements from a team here, I actually extended it quite a bit to also
be able to handle incoming HTTP requests.

I called this tool AntEater (pun intended), as it's heavily based on Ant.

Anteater allows one to write tests that fire HTTP requests to a Cocoon
installation and test the incoming response, using simple matching operations.
If Anteater and Cocoon are on the same machine, you can use normal Ant tasks to
modify Cocoon files while the system is running, send another request to the
server and test in the response the modified Cocoon behavior.

Using Anteater we should be able to write functional tests that automatically
check Cocoon after any change we do to the source code. The only testing that
happens right now seems to be a manual one.

The intent is to have tests written for AntEater, along with the junit tests,
and perhaps a Cactus based framework (still looking at it), part of a testing
suite for Cocoon.

I was wondering what would be a good way to release it. As it is right now it's
a standalone package, much like Ant. You install it on the local filesystem,
and you get an 'anteater' script, that behaves just like the 'ant' script from
ant, while adding additional tasks. The reason for a separation from Cocoon is
that it's quite generic, not tied to Cocoon in any way. In fact one of the team
in my group plans to use it to test generic XML based applications, involving
synchronous and asynchronous SOAP messages.

I append the current documentation I wrote for it. I would appreciate any
comments from you on Anteater.

Regards,
-- 
Ovidiu Predescu <ov...@cup.hp.com>
http://orion.rgv.hp.com/ (inside HP's firewall only)
http://sourceforge.net/users/ovidiu/ (my SourceForge page)
http://www.geocities.com/SiliconValley/Monitor/7464/ (GNU, Emacs, other stuff)

Re: AntEater: a functional testing framework for HTTP/SOAP apps

[Stefano Mazzocchi <st...@apache.org>]

Ovidiu Predescu wrote:
> 
> Hi,
> 
> As I mentioned last week, thanks to Jeff Turner who pointed to me the
> httpClient testing framework from Tomcat, I was able to do some very good
> progress in implementing a functional testing framework for Cocoon. With some
> more requirements from a team here, I actually extended it quite a bit to also
> be able to handle incoming HTTP requests.

Great news!
 
> I called this tool AntEater (pun intended), as it's heavily based on Ant.

I think there is already a proposal called AntEater for Ant 2.0, you
might want to check in the Ant CVS under the 'proposal' folder.

> Anteater allows one to write tests that fire HTTP requests to a Cocoon
> installation and test the incoming response, using simple matching operations.
> If Anteater and Cocoon are on the same machine, you can use normal Ant tasks to
> modify Cocoon files while the system is running, send another request to the
> server and test in the response the modified Cocoon behavior.

Hmmm, can you write me an example of why you think something like this
is helpful? i mean, how would the tester suite know what to modify? hey,
if it fixed cocoon bugs by itself, we can let it run for a couple of
days and make the final release :)

Just kidding, of course.
 
> Using Anteater we should be able to write functional tests that automatically
> check Cocoon after any change we do to the source code. The only testing that
> happens right now seems to be a manual one.

Correct, something like this would be "extremely" beneficial.
 
> The intent is to have tests written for AntEater, along with the junit tests,
> and perhaps a Cactus based framework (still looking at it), part of a testing
> suite for Cocoon.

That would be killer. We definately need more testing to make sure we
don't introduce back incompatibilities as we patch along.
 
> I was wondering what would be a good way to release it. As it is right now it's
> a standalone package, much like Ant. You install it on the local filesystem,
> and you get an 'anteater' script, that behaves just like the 'ant' script from
> ant, while adding additional tasks. The reason for a separation from Cocoon is
> that it's quite generic, not tied to Cocoon in any way. In fact one of the team
> in my group plans to use it to test generic XML based applications, involving
> synchronous and asynchronous SOAP messages.
> 
> I append the current documentation I wrote for it. I would appreciate any
> comments from you on Anteater.

Question: how hard would it be to integrate it with Ant? I mean,
creating an optional, but coherent, set of tasks that extend the basic
ant functionality into this web-testing suite?

I know there is very little functional overlap between the two, but this
can be said for almost any Ant task: Ant power comes from exactly from
the availability of so many useful tasks and adding new ones is a really
good thing to do.

But the SOAP incoming requests things might be a little oftopic for
Ant... hmmm, let's see what you guys thing first and then we can ask
around where the best location is.

Anyway, if it doesn't end up under Ant, please change that name since it
might appear a little rude on them.

-- 
Stefano Mazzocchi      One must still have chaos in oneself to be
                          able to give birth to a dancing star.
<st...@apache.org>                             Friedrich Nietzsche
--------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org

Re: AntEater: a functional testing framework for HTTP/SOAP apps

[Jeff Turner <je...@socialchange.net.au>]

Wow, congratulations :) It sounds amazingly comprehensive, given how quickly it
was written. Apart from Cocoon, I can see this being very useful when
developing plain old servlets.

Specific comments:

Going through Anteater.txt, I couldn't find any way to provide an error message
if the test failed. How about adding a <failMessage> block, to say what it
means when the test fails? Does <comment> fulfil this role?

Also, how about using the standard "description" attribute instead of
<comment>? In Ant 1.4, "description" can be attached to just about everything,
and has rather become a standard.

Likewise to keep with Ant naming conventions, "assign" could be "id", and
"href" become "refid".

I saw a few references to namespace-scoped elements there ("soap:"), but no
namespace declarations. Are namespaces ignored?

I was thinking.. you currently have a <http> task. Would it be easy to add a
<file> task, so that validations can be applied to things like sitemap.xmap,
cocoon.xconf, or a site's web.xml? That would allow elimination of
misconfigurations as a source of apparent failures. For example, to ensure that
the Cocoon servlet is registered:

<file path="build/cocoon/webapp/WEB-INF/web.xml">
  <match>
    <xpath select="web-app/servlet/servlet-class"
    value="org.apache.cocoon.servlet.CocoonServlet"/>
  </match>
</file>

And to ensure that the sitemap is present and contains a mapping for /index.html:

<file path="build/cocoon/webapp/sitemap.xmap">
  <match>
    <xpath select="map:sitemap/map:pipeline/map:match[pattern='hello.html']"/>
  </match>
</file>

This then leads on to your suggestion:

> If Anteater and Cocoon are on the same machine, you can use normal Ant tasks to
> modify Cocoon files while the system is running, send another request to the
> server and test in the response the modified Cocoon behavior.

I don't think those "normal Ant tasks" exist yet ;) What would be really nice
is a version of James Strachan's xtags taglib, for Ant instead of JSP.

On Thu, Oct 04, 2001 at 09:20:17AM -0700, Ovidiu Predescu wrote:
> Hi,
> 
[..]
> I was wondering what would be a good way to release it. As it is right now it's
> a standalone package, much like Ant. You install it on the local filesystem,
> and you get an 'anteater' script, that behaves just like the 'ant' script from
> ant, while adding additional tasks.

Have you needed to make any modifications to the Ant core? If not, why not just
put the jar in lib/, and create a new Ant file 'tests.xml' with a bunch of
<taskdef>s to declare the tasks?

Anyway, well done, and I look forward to playing, and sponging off your
hard work for private projects ;)


--Jeff

[..]
> Regards,
> -- 
> Ovidiu Predescu <ov...@cup.hp.com>
> http://orion.rgv.hp.com/ (inside HP's firewall only)
> http://sourceforge.net/users/ovidiu/ (my SourceForge page)
> http://www.geocities.com/SiliconValley/Monitor/7464/ (GNU, Emacs, other stuff)
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org