Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[William McKee <wi...@knowmad.com>]

On Fri, Jan 16, 2004 at 02:30:45PM -0500, Geoffrey Young wrote:
> basically, there are two ways to approach Apache-Test: do it all yourself or
> let Apache-Test add some highly magical stuff.  the article I wrote takes
> the first route - it assumes that you already know how to edit an httpd.conf
> and are somewhat familiar with Test.pm or Test::More.  the testing tutorial
> shows you other parts of Apache-Test, such as letting it autogenerate tests
> and configurations.
> 

Thanks to Stas' tireless efforts to educate me as well as your articles
and sample, I think that I've about got my confusion cleared up (part of
which was due to a valid bug).

There's one last piece that I'm still unclear on which your article did
not mention--reponse handlers. Is this where the two paths diverge? I'm
not sure that I see much difference between the way you do it and the
highly magical stuff.


> personally, I find my approach much more managable for first time users, but
> YMMV.

My mileage starting improving once I began to use your bug reporting
skeleton. Starting with a working skeleton that implements correct
practices is no doubt the best way to get started with this testing
environment. Your article makes mention that this framework is easy--it
is if you know how all the pieces work together (and there are several
pieces).


Cheers!
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[Geoffrey Young <ge...@modperlcookbook.org>]

> I would appreciate some recommendations
> for books that explain the big picture view of testing. Have any titles
> on your bookshelf that you'd suggest for someone like myself? I don't
> know of any books that discuss testing specifically from a Perl
> perspective. Do you?

ORA had one in the works, but I think it fell through, which is a shame.

--Geoff

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[William McKee <wi...@knowmad.com>]

On Sun, Jan 18, 2004 at 10:57:25AM -0500, Geoffrey Young wrote:
> it really depends on what you're trying to accomplish.  Apache-Test is for
> what is generally called 'unit tests' or 'functional tests' - given known
> input, make sure your code gives you the correct output.

Now that you mention it, I had thought Apache-Test would be performing
system tests since it utilizes a server similar to a production
environment in order to do the testing. Obviously, I'm a newbie to
testing and have lots to learn! I would appreciate some recommendations
for books that explain the big picture view of testing. Have any titles
on your bookshelf that you'd suggest for someone like myself? I don't
know of any books that discuss testing specifically from a Perl
perspective. Do you?


Thanks,
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[Geoffrey Young <ge...@modperlcookbook.org>]

>>well, it's important if you're writing an API, not merely testing CGI
>>scripts or the like.  for instance, suppose you have a class that subclasses
>>mod_perl.  if you plan'd and ok'd things from a PerlHandler then you could
>>do things like
>>
>>  isa_ok($r, 'My::Class');
>>
>>which you can't really do otherwise.
> 
> 
> OK, let me see if I'm understanding. The response handler acts as a kind
> of subclass that sits between modules in the project's lib/ directory
> and requests. It's like a back-door into the code that is running on the
> server which allows you to make your tests even more complete.

pretty much, but I wouldn't call it a back door - it's web application
programming through the Apache/mod_perl API.

> 
> If this is correct, it's a very subtle distinction. The request test is
> essentially testing the response. However, it doesn't have direct access
> to the $r object that got created in response to the request--it just
> gets the final data being sent back. 

exactly.  you could, say, die() after every API call and test for 500 in
your foo.t, in which case you really don't know where the issue is.  or you
can just have foo.t call a PerlResponseHandler that does all the work for you.

it really depends on what you're trying to accomplish.  Apache-Test is for
what is generally called 'unit tests' or 'functional tests' - given known
input, make sure your code gives you the correct output.

so, you can use Apache-Test merely to control Apache activity - configure,
start, and stop the server - which gives you a pristine environment to test
against.  then you can unit test whatever you want, say CGI scripts (given a
certain query string produce a proper login screen).

in this case, mod_cgi is the response handler.

if you use mod_perl as the response handler for your application, then you
have more available in terms of the Apache API.  and you can use Apache-Test
to write unit tests against an API that you write to abstract out components.

> In this sense, it seems like the
> response test is like an Apache filter. If I'm grokking this correctly,
> this seems like an amazingly powerful tool which would be useful to
> testing an API but not as necessary for testing a web application unless
> you're getting some hard-to-track bugs happening.

it is incredibly powerful, yes :)

> 
> In Stas' example at perl.apache.org, he uses the url
> '/TestApache__cool' in the request test, cool.t. I saw the paragraph at
> the end of the section about developing response and request parts of a
> test which described the directory layout. However, I didn't see a
> discussion of how the url is created. I'm guessing that A::T takes all
> the directories and files in t/response and creates Locations which
> the test apache server can handle. Would the file
> t/response/Mod/hot.pm create a url of '/Mod__hot'?

this is the magic part.  but you don't need to worry about the magic to make
it all happen.  just configure extra.conf.in as you ordinarly would.  then
define your PerlResponseHandlers outside of lib/ - like in t/My/ :)  the
packages in t/My can call the API living in lib/ so you can test them.

using the magic is the next step, and is particularly helpful in largish
applications.  for me, I tend to use the simple approach most of the time.

--Geoff

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[William McKee <wi...@knowmad.com>]

On Sat, Jan 17, 2004 at 02:35:17PM -0500, Geoffrey Young wrote:
> cool.  actually, I've been meaning for a while to create a :testmore import
> target that exports everything but ok() and plan() (the two that collide
> with Test::More). 

That'd be very useful!


> well, it's important if you're writing an API, not merely testing CGI
> scripts or the like.  for instance, suppose you have a class that subclasses
> mod_perl.  if you plan'd and ok'd things from a PerlHandler then you could
> do things like
> 
>   isa_ok($r, 'My::Class');
> 
> which you can't really do otherwise.

OK, let me see if I'm understanding. The response handler acts as a kind
of subclass that sits between modules in the project's lib/ directory
and requests. It's like a back-door into the code that is running on the
server which allows you to make your tests even more complete.

If this is correct, it's a very subtle distinction. The request test is
essentially testing the response. However, it doesn't have direct access
to the $r object that got created in response to the request--it just
gets the final data being sent back. In this sense, it seems like the
response test is like an Apache filter. If I'm grokking this correctly,
this seems like an amazingly powerful tool which would be useful to
testing an API but not as necessary for testing a web application unless
you're getting some hard-to-track bugs happening.

In Stas' example at perl.apache.org, he uses the url
'/TestApache__cool' in the request test, cool.t. I saw the paragraph at
the end of the section about developing response and request parts of a
test which described the directory layout. However, I didn't see a
discussion of how the url is created. I'm guessing that A::T takes all
the directories and files in t/response and creates Locations which
the test apache server can handle. Would the file
t/response/Mod/hot.pm create a url of '/Mod__hot'?


> I've always used environment variables.  for instance, when I was testing a
> SOAP client API, I defaulted my SOAP server to localhost, but used
> %ENV{SOAP_SERVER} (or somesuch) if available.  this let me run the same
> tests against both my local extra.conf.in and the production box I was
> setting up.

Coming from a DOS/Win* background (I've been using a Linux workstation
for about a year now), I'm not as familiar with the wide range of uses
for environment variables. It's good to know that others are using them
in this way. I'll continue to forge ahead.


> HTH

++


Regards,
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[Geoffrey Young <ge...@modperlcookbook.org>]

> I see why you call this code magical. It's definitely a step beyond my
> understanding and probably beyond my needs for now.

:)


> OK, I'm beginning to see now. I was able to take my revised skeleton and
> replace Apache::Test with Test::More. I had to modify the plan line to
> remove the have_lwp() as that must be part of the Apache::Test module.
> After that, everything worked!

:)

> 
> Then I discovered that I can use both Apache::Test and Test::More
> together with the following calls:
> 
>     use Apache::Test qw('have_lwp');
>     use Test::More;
> 
> As you're no doubt aware, without the qw('have_lwp'), T::M overrides
> subroutines like 'ok' in A::T. I was even able to add my usual test
> module into the mix--HTTP::WebTest. Hey, I think things are gel'ing!

cool.  actually, I've been meaning for a while to create a :testmore import
target that exports everything but ok() and plan() (the two that collide
with Test::More).  but other than that, so long as you call Test::More::plan
 you can use all the Test::More comparison functions, as well as things like
t_cmp() from Apache::TestUtil.

> 
> So each developer can run their own Apache process and test against it
> as if it were the production server. You could even use the same binary
> that is running the production server though you probably wouldn't want
> to do intensive testing on a production platform. This test environment
> rocks!

that's what we've been saying all along.

;)

> 
> I still don't understand the driving need for writing tests that use
> PerlHandlers but it seems best for me to leave that part of the test
> framework alone until my knowledge matures.

well, it's important if you're writing an API, not merely testing CGI
scripts or the like.  for instance, suppose you have a class that subclasses
mod_perl.  if you plan'd and ok'd things from a PerlHandler then you could
do things like

  isa_ok($r, 'My::Class');

which you can't really do otherwise.

> 
> One last question for now. Is there a way to pass additional arguments
> into my scripts besides using environment variables? I use the
> HTTP::WebTest module to do both local and remote testing. It'd be nice
> to be able to pass an option to it to tell it to go into remote mode. I
> guess doing something along the lines of the following is good:
> 
>     TEST_REMOTE=1 t/TEST -run
> 
> I suppose the other option would be to write another test that would
> only run if the above environment parameter is passed. Is there a better
> way that folks here are using? I suspect this is rather non-standard
> usage of test suites so it may be better to put my remote tests into
> another location.

I've always used environment variables.  for instance, when I was testing a
SOAP client API, I defaulted my SOAP server to localhost, but used
%ENV{SOAP_SERVER} (or somesuch) if available.  this let me run the same
tests against both my local extra.conf.in and the production box I was
setting up.

HTH

--Geoff

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[William McKee <wi...@knowmad.com>]

On Sat, Jan 17, 2004 at 09:59:03AM -0500, Geoffrey Young wrote:
> > There's one last piece that I'm still unclear on which your article did
> > not mention--reponse handlers. Is this where the two paths diverge? I'm
> > not sure that I see much difference between the way you do it and the
> > highly magical stuff.
> 
> well, that's the next step.  instead of calling plan() from within the test
> script, you can call
> 
>   plan $r, tests => 5;
> 
> from within a handler() subroutine in a PerlHandler.  but again, you don't
> need to use the magical parts of Apache-Test to autogenerated all the foo.
> that is, you can still write your own foo.t and extra.conf.in or whatever
> else you need.

I see why you call this code magical. It's definitely a step beyond my
understanding and probably beyond my needs for now.


> if you understand that no matter what wrapper you use - Apache-Test,
> Test::More, etc - in the end they are all just feeding "1..5" and "ok" to
> Test::Harness, then it all kinda makes sense - of course you can plan() and
> ok() from either a PerlHandler or *.t, so long as the proper sequence of
> data is printed to STDOUT.

OK, I'm beginning to see now. I was able to take my revised skeleton and
replace Apache::Test with Test::More. I had to modify the plan line to
remove the have_lwp() as that must be part of the Apache::Test module.
After that, everything worked!

Then I discovered that I can use both Apache::Test and Test::More
together with the following calls:

    use Apache::Test qw('have_lwp');
    use Test::More;

As you're no doubt aware, without the qw('have_lwp'), T::M overrides
subroutines like 'ok' in A::T. I was even able to add my usual test
module into the mix--HTTP::WebTest. Hey, I think things are gel'ing!

So each developer can run their own Apache process and test against it
as if it were the production server. You could even use the same binary
that is running the production server though you probably wouldn't want
to do intensive testing on a production platform. This test environment
rocks!

I still don't understand the driving need for writing tests that use
PerlHandlers but it seems best for me to leave that part of the test
framework alone until my knowledge matures.

One last question for now. Is there a way to pass additional arguments
into my scripts besides using environment variables? I use the
HTTP::WebTest module to do both local and remote testing. It'd be nice
to be able to pass an option to it to tell it to go into remote mode. I
guess doing something along the lines of the following is good:

    TEST_REMOTE=1 t/TEST -run

I suppose the other option would be to write another test that would
only run if the above environment parameter is passed. Is there a better
way that folks here are using? I suspect this is rather non-standard
usage of test suites so it may be better to put my remote tests into
another location.

Many thanks!
William

-- 
Knowmad Services Inc.
http://www.knowmad.com

Re: Response Handlers (was Re: Perl test framework, TestConfig, and debugging A::T)

[Geoffrey Young <ge...@modperlcookbook.org>]

> 
> There's one last piece that I'm still unclear on which your article did
> not mention--reponse handlers. Is this where the two paths diverge? I'm
> not sure that I see much difference between the way you do it and the
> highly magical stuff.

well, that's the next step.  instead of calling plan() from within the test
script, you can call

  plan $r, tests => 5;

from within a handler() subroutine in a PerlHandler.  but again, you don't
need to use the magical parts of Apache-Test to autogenerated all the foo.
that is, you can still write your own foo.t and extra.conf.in or whatever
else you need.

if you understand that no matter what wrapper you use - Apache-Test,
Test::More, etc - in the end they are all just feeding "1..5" and "ok" to
Test::Harness, then it all kinda makes sense - of course you can plan() and
ok() from either a PerlHandler or *.t, so long as the proper sequence of
data is printed to STDOUT.

HTH

--Geoff