You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Alexey Melezhik <me...@gmail.com> on 2016/02/01 11:23:26 UTC
Re: collaboration request - apache server automation testing with
swat tool
Here the list of existed issues I was able to automate tests for:
vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
44221
46751
58789
58828
58854
I have informed developers at https://bz.apache.org/bugzilla/ about
these, so they rely upon them. Or you have some pre-release testing,
so run all of these in a whole chunk?
Regards
Alexey
PS chaining issues / tests listing as always -
https://github.com/melezhik/apache-swat
PS2 it'd be could to verify tests as well against "bleeding edge"
apache version gets installed from SCM, but I don't know how to do
this.
Currently I use http://httpd.apache.org/download.cgi ( last stable release )
2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
> Hi Bill!
>
> I have started to assemble swat tests for apache2 issues coming from
> https://bz.apache.org/bugzilla
>
> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>
> Test suite is failed for the moment :
>
>
> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
> ok 2 - output match '200 OK'
> 1..2
> ok
> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
> ok 2 - output match '200 OK'
> ok 3 - output match /Location: \S+/
> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> 1..4
> ok
> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
> ok 2 - output match '200 OK'
> ok 3 - output match /Location: \S+/
> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> 1..4
> ok
> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
> ok 2 - output match '200 OK'
> 1..2
> ok
> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>
> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> not ok 4 - output match 'Not Found'
>
> # Failed test 'output match 'Not Found''
> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> 1..4
> # Looks like you failed 2 tests of 4.
> Dubious, test returned 2 (wstat 512, 0x200)
> Failed 2/4 subtests
>
> Test Summary Report
> -------------------
> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
> 512 Tests: 4 Failed: 2)
> Failed tests: 3-4
> Non-zero exit status: 2
> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
> 0.00 csys = 0.30 CPU)
> Result: FAIL
>
> ...
>
> Ok, this is just the beginning ;-))) , watch soon updates at -
> https://github.com/melezhik/apache-swat
>
> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>
>>> Hi Bill!
>>>
>>> Any news? ( Please see my previous reply ...)
>>> Intr
>>
>>
>> Intrigued :) But my responses will be delayed, I personally
>> won't have time to look further myself until other backlogged
>> commitments to httpd are caught up some more.
>>
>> Be aware many of us have only a few hours to participate in the
>> httpd project itself, so it will take several days or more to gather
>> feedback on your interest and perhaps a champion to drive such
>> an effort if it is actively driven from this side. Of course that
>> should not stop you from assembling some ideas at the vcs of
>> your choice, and sharing a link for those who want to look at an
>> early draft of such an effort!
>>
>> Cheers,
>>
>> Bill
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
Hi ! I have got a few comments , questions concerning this topic, thanks
for everybody! though still have idea if Apache devs are interested in new
testing framework ? It is hard to keep moving doubting if such tool could
be of interest for someone else beside me ) , so give me a sign please :)
If I don't waist my time.
Thanks.
пятница, 5 февраля 2016 г. пользователь Jim Jagielski написал:
> Personally, I like the idea of having another framework;
> the current one is OK but somewhat "painful" to update.
>
> I wonder how possible it would be to transcode the old tests
> to Swat? We could then provide for 2 testing frameworks, one
> developed by the ASF and the other external and 3rd party.
>
>
> > On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com
> <javascript:;>> wrote:
> >
> > Here the list of existed issues I was able to automate tests for:
> >
> > vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P
> '\d'
> > 44221
> > 46751
> > 58789
> > 58828
> > 58854
> >
> >
> > I have informed developers at https://bz.apache.org/bugzilla/ about
> > these, so they rely upon them. Or you have some pre-release testing,
> > so run all of these in a whole chunk?
> >
> > Regards
> >
> > Alexey
> >
> > PS chaining issues / tests listing as always -
> > https://github.com/melezhik/apache-swat
> >
> > PS2 it'd be could to verify tests as well against "bleeding edge"
> > apache version gets installed from SCM, but I don't know how to do
> > this.
> > Currently I use http://httpd.apache.org/download.cgi ( last stable
> release )
> >
> >
> >
> > 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com
> <javascript:;>>:
> >> Hi Bill!
> >>
> >> I have started to assemble swat tests for apache2 issues coming from
> >> https://bz.apache.org/bugzilla
> >>
> >> And this the first one -
> https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
> >>
> >> Test suite is failed for the moment :
> >>
> >>
> >> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
> >> ok 2 - output match '200 OK'
> >> 1..2
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
> >> ok 2 - output match '200 OK'
> >> ok 3 - output match /Location: \S+/
> >> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> >> 1..4
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
> >> ok 2 - output match '200 OK'
> >> ok 3 - output match /Location: \S+/
> >> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> >> 1..4
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
> >> ok 2 - output match '200 OK'
> >> 1..2
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
> >> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
> >> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
> >> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
> >>
> >> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
> >> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> >> not ok 4 - output match 'Not Found'
> >>
> >> # Failed test 'output match 'Not Found''
> >> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> >> 1..4
> >> # Looks like you failed 2 tests of 4.
> >> Dubious, test returned 2 (wstat 512, 0x200)
> >> Failed 2/4 subtests
> >>
> >> Test Summary Report
> >> -------------------
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
> >> 512 Tests: 4 Failed: 2)
> >> Failed tests: 3-4
> >> Non-zero exit status: 2
> >> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
> >> 0.00 csys = 0.30 CPU)
> >> Result: FAIL
> >>
> >> ...
> >>
> >> Ok, this is just the beginning ;-))) , watch soon updates at -
> >> https://github.com/melezhik/apache-swat
> >>
> >> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net
> <javascript:;>>:
> >>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com
> <javascript:;>> wrote:
> >>>>
> >>>> Hi Bill!
> >>>>
> >>>> Any news? ( Please see my previous reply ...)
> >>>> Intr
> >>>
> >>>
> >>> Intrigued :) But my responses will be delayed, I personally
> >>> won't have time to look further myself until other backlogged
> >>> commitments to httpd are caught up some more.
> >>>
> >>> Be aware many of us have only a few hours to participate in the
> >>> httpd project itself, so it will take several days or more to gather
> >>> feedback on your interest and perhaps a champion to drive such
> >>> an effort if it is actively driven from this side. Of course that
> >>> should not stop you from assembling some ideas at the vcs of
> >>> your choice, and sharing a link for those who want to look at an
> >>> early draft of such an effort!
> >>>
> >>> Cheers,
> >>>
> >>> Bill
>
>
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
Hi Yann!
* per issues apache configurations *
https://github.com/entimelezhik/apache-swat/tree/master/htdocs is
document root , every issue thus is represented by directory inside
htdocs , named by issue number f.e.
- htdocs/44221
- htdocs/58777
If it's necessary per issue configuration defined by .htaccess files
placed at issues directories for example:
- htdocs/58777/.htaccess
All other static files are thus kept under issue directory , for example:
- htdocs/58777/data/foo.data
* global apache configuration ( httpd.conf )*
httpd.conf is generated by template file -
https://github.com/melezhik/apache-swat/blob/master/httpd.conf.tmpl -
this is vanilla apcahe2 config file taken from standard apach2e
installation ( https://httpd.apache.org/docs/2.2/en/install.html )
with some `LoadModule' lines uncommented to make it possible to run
tests for some none default enabled apache modules ( mod_rewrite ,
mod_include , ... )
* Initial test configuration *
Initial test configuration is trivial :
* install apache2 to /some/prefix/directory
$ git clone https://github.com/melezhik/apache-swat.git
$ cd apache-swat
# generate httpd.conf for your apache installation prefix
$ ./configure.bash /some/prefix/directory
# run apache with generated httpd.conf at CWD
$ sudo /some/prefix/directory/bin/apachectl -f `pwd`/httpd.conf
then you may choose to:
# run all tests
$ swat
# or run per-issue test
$ swat -t 58777
To summarize:
* per issues apache configurations based on related directories/htcaccess files
* one httpd.conf for all issues
* technically speaking it is possible to have multiple issues related
httpd.conf files ( one per issue ) , but this means:
- necessity of apache2 server restart at the beginning of every
issue test which has time impact ( or have many apache2 instances with
different httpd.conf which is hard to maintain )
- in many cases per issues apache2 configuration could be expressed
via directories/htaccess files which is much easier to maintain
Hope, this helps.
Regards
Alexey
2016-02-06 19:33 GMT+03:00 Yann Ylavic <yl...@gmail.com>:
> Hi Alexey,
>
> On Sat, Feb 6, 2016 at 5:03 PM, Alexey Melezhik <me...@gmail.com> wrote:
>> Sure, then it's probably won't be big deal to port existed test code base.
>>
>> Here I want to highlight some swat features which make it more then
>> just engine to make http requests and verify an output.
>
> How could we manage the httpd onfiguration files associated with each test?
>
> Regards,
> Yann.
Re: collaboration request - apache server automation testing with
swat tool
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Alexey,
On Sat, Feb 6, 2016 at 5:03 PM, Alexey Melezhik <me...@gmail.com> wrote:
> Sure, then it's probably won't be big deal to port existed test code base.
>
> Here I want to highlight some swat features which make it more then
> just engine to make http requests and verify an output.
How could we manage the httpd onfiguration files associated with each test?
Regards,
Yann.
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
Sure, then it's probably won't be big deal to port existed test code base.
Here I want to highlight some swat features which make it more then
just engine to make http requests and verify an output.
Swat adds some features could be of interest in httpd2 testing, to
describe briefly a few:
- *Curl centric design *
Swat relies on curl heavily, constructing all http requests in a terms
of curl command line utility and then passing them to curl to make
http request. This a single swat test unit is based on more or many
http request made by curl. IMHO using curl has some essential benefits
for httpd2 testing, to list a few:
- often apache server testing implies low level http context
which is out of the box when using cur. Consider a few cases:
- making http2 protocol requests ( curl does! )
- setting explicit timeouts ( per initial connection and per
whole requests )
- issuing "raw" http requests with chunked/broken data to
tests how apache reacts on invalid requests
- construction of various html related entities ( forms ,
cookies , named parameters )
- various types of authentication works out of the box -
--basic, --digest, --ntlm, and --negotiate
- ssl related testing ( not relates to http but ... )
Also please take into account:
- curl is stable and well supported
- no need to install to many perl dependencies for http related stuff
- all the features granted by latest bleeding edge versions of curl
are here to use
- *Simple debugging*.
As swat uses curl to make http request. _basically_ a failing test
mean you may reproduce the same case manually using curl as you have
all you need here ( http request parameters get dumped into console )
In other words swat is much closer to what people usually do when
carry out manual testing - get a grip with curl, send command and
then see the output. Such an approach makes it swat's usage extremely
simple and understandable.
- *Sequential curl requests*.
As curl a request oriented ( only one single request could be made at
time ) it has some limits. As swat just uses curl it makes it possible
to create so called sequential/compound requests. When before one
request other request is issued. As trivial example of such a case is
user authentication. When one needs access for restricted http
resources and a preliminary authentication is required. So first we
need get logged in and only then request a restricted resource. Such a
compound requests on other hand are just a single test story for end
user and are hidden under the hood wheh using swat - see
https://github.com/melezhik/swat#upstream-and-downstream-stories for
details.
- * Text validation DSL *
And finally swat provides rich and simple DSL for text validation.
This simplifies server output verification and _often_ even does not
require a single line of perl code ( as it none perlish DSL ). In
other hand swat is extendable by perl to add desired complexity to
your tests. Take a look at examples - https://github.com/melezhik/swat
and https://github.com/melezhik/swat/tree/master/examples
Regards
Alexey
2016-02-06 17:34 GMT+03:00 Jim Jagielski <ji...@jagunet.com>:
>
>> On Feb 5, 2016, at 9:26 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>
>>> I wonder how possible it would be to transcode the old tests to Swat?
>>
>> Swat has some architectural specifics. As it simply makes http
>> requests ( by curl ) and validate server response ( by swat DSL ) it
>> is considered as black box testing tools , there is no way to interact
>> with httpd internal structure. So probably most of old tests could be
>> easily ported but some probably not ( which implies something more
>> complicated ).
>>
>
> Most of the other tests w/ the old framework is the exact same...
> requests sent and responses verified against what is expected.
>
>> Basically swat could cover simple and average use cases, everything
>> that could expressed in terms of make http requests and then
>> analyzing an output , see examples for some issues I have expressed in
>> swat scenarios at https://github.com/melezhik/apache-swat
>>
>>
>> Also alternatively we may run 2 frameworks in parallel. Apache-swat is
>> intended for developers usage first and is able to run per-issues
>> test.
>>
>> This how I see possible workflow:
>>
>> * developer accept a bug request
>> * developer ( or some else ) create swat test for given bug/feature
>> * so we have `test first` approach - everyone involved in bug/feature
>> resolution has possibility to verify changes by simply running :
>>
>> swat -t $issue-id
>>
>>
>>
>>
>>
>>
>> 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>>> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <ji...@jagunet.com>:
>>>> Personally, I like the idea of having another framework;
>>>> the current one is OK but somewhat "painful" to update.
>>>
>>> Hi! What kind of possible issues you here ? when saying "painful" to
>>> update, please explain.
>>> Thanks.
>>>
>>> Also , please consider this link - https://github.com/melezhik/apache-swat and
>>> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
>>> as latest version of this idea ....
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> I wonder how possible it would be to transcode the old tests
>>>> to Swat? We could then provide for 2 testing frameworks, one
>>>> developed by the ASF and the other external and 3rd party.
>>>>
>>>>
>>>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>>
>>>>> Here the list of existed issues I was able to automate tests for:
>>>>>
>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>>>> 44221
>>>>> 46751
>>>>> 58789
>>>>> 58828
>>>>> 58854
>>>>>
>>>>>
>>>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>>>> these, so they rely upon them. Or you have some pre-release testing,
>>>>> so run all of these in a whole chunk?
>>>>>
>>>>> Regards
>>>>>
>>>>> Alexey
>>>>>
>>>>> PS chaining issues / tests listing as always -
>>>>> https://github.com/melezhik/apache-swat
>>>>>
>>>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>>>> apache version gets installed from SCM, but I don't know how to do
>>>>> this.
>>>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>>>>
>>>>>
>>>>>
>>>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>>>>>> Hi Bill!
>>>>>>
>>>>>> I have started to assemble swat tests for apache2 issues coming from
>>>>>> https://bz.apache.org/bugzilla
>>>>>>
>>>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>>>>>
>>>>>> Test suite is failed for the moment :
>>>>>>
>>>>>>
>>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>>>> ok 2 - output match '200 OK'
>>>>>> 1..2
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>>>> ok 2 - output match '200 OK'
>>>>>> ok 3 - output match /Location: \S+/
>>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>>> 1..4
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>>>> ok 2 - output match '200 OK'
>>>>>> ok 3 - output match /Location: \S+/
>>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>>> 1..4
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>>>> ok 2 - output match '200 OK'
>>>>>> 1..2
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>>>
>>>>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>>> not ok 4 - output match 'Not Found'
>>>>>>
>>>>>> # Failed test 'output match 'Not Found''
>>>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>>> 1..4
>>>>>> # Looks like you failed 2 tests of 4.
>>>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>>>> Failed 2/4 subtests
>>>>>>
>>>>>> Test Summary Report
>>>>>> -------------------
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>>>> 512 Tests: 4 Failed: 2)
>>>>>> Failed tests: 3-4
>>>>>> Non-zero exit status: 2
>>>>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
>>>>>> 0.00 csys = 0.30 CPU)
>>>>>> Result: FAIL
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>>>> https://github.com/melezhik/apache-swat
>>>>>>
>>>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>>>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Bill!
>>>>>>>>
>>>>>>>> Any news? ( Please see my previous reply ...)
>>>>>>>> Intr
>>>>>>>
>>>>>>>
>>>>>>> Intrigued :) But my responses will be delayed, I personally
>>>>>>> won't have time to look further myself until other backlogged
>>>>>>> commitments to httpd are caught up some more.
>>>>>>>
>>>>>>> Be aware many of us have only a few hours to participate in the
>>>>>>> httpd project itself, so it will take several days or more to gather
>>>>>>> feedback on your interest and perhaps a champion to drive such
>>>>>>> an effort if it is actively driven from this side. Of course that
>>>>>>> should not stop you from assembling some ideas at the vcs of
>>>>>>> your choice, and sharing a link for those who want to look at an
>>>>>>> early draft of such an effort!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Bill
>>>>
>
Re: collaboration request - apache server automation testing with swat tool
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Feb 5, 2016, at 9:26 AM, Alexey Melezhik <me...@gmail.com> wrote:
>
>> I wonder how possible it would be to transcode the old tests to Swat?
>
> Swat has some architectural specifics. As it simply makes http
> requests ( by curl ) and validate server response ( by swat DSL ) it
> is considered as black box testing tools , there is no way to interact
> with httpd internal structure. So probably most of old tests could be
> easily ported but some probably not ( which implies something more
> complicated ).
>
Most of the other tests w/ the old framework is the exact same...
requests sent and responses verified against what is expected.
> Basically swat could cover simple and average use cases, everything
> that could expressed in terms of make http requests and then
> analyzing an output , see examples for some issues I have expressed in
> swat scenarios at https://github.com/melezhik/apache-swat
>
>
> Also alternatively we may run 2 frameworks in parallel. Apache-swat is
> intended for developers usage first and is able to run per-issues
> test.
>
> This how I see possible workflow:
>
> * developer accept a bug request
> * developer ( or some else ) create swat test for given bug/feature
> * so we have `test first` approach - everyone involved in bug/feature
> resolution has possibility to verify changes by simply running :
>
> swat -t $issue-id
>
>
>
>
>
>
> 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <ji...@jagunet.com>:
>>> Personally, I like the idea of having another framework;
>>> the current one is OK but somewhat "painful" to update.
>>
>> Hi! What kind of possible issues you here ? when saying "painful" to
>> update, please explain.
>> Thanks.
>>
>> Also , please consider this link - https://github.com/melezhik/apache-swat and
>> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
>> as latest version of this idea ....
>>
>>
>>
>>
>>
>>>
>>> I wonder how possible it would be to transcode the old tests
>>> to Swat? We could then provide for 2 testing frameworks, one
>>> developed by the ASF and the other external and 3rd party.
>>>
>>>
>>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>
>>>> Here the list of existed issues I was able to automate tests for:
>>>>
>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>>> 44221
>>>> 46751
>>>> 58789
>>>> 58828
>>>> 58854
>>>>
>>>>
>>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>>> these, so they rely upon them. Or you have some pre-release testing,
>>>> so run all of these in a whole chunk?
>>>>
>>>> Regards
>>>>
>>>> Alexey
>>>>
>>>> PS chaining issues / tests listing as always -
>>>> https://github.com/melezhik/apache-swat
>>>>
>>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>>> apache version gets installed from SCM, but I don't know how to do
>>>> this.
>>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>>>
>>>>
>>>>
>>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>>>>> Hi Bill!
>>>>>
>>>>> I have started to assemble swat tests for apache2 issues coming from
>>>>> https://bz.apache.org/bugzilla
>>>>>
>>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>>>>
>>>>> Test suite is failed for the moment :
>>>>>
>>>>>
>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>>> ok 2 - output match '200 OK'
>>>>> 1..2
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>>> ok 2 - output match '200 OK'
>>>>> ok 3 - output match /Location: \S+/
>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>> 1..4
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>>> ok 2 - output match '200 OK'
>>>>> ok 3 - output match /Location: \S+/
>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>> 1..4
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>>> ok 2 - output match '200 OK'
>>>>> 1..2
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>>
>>>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>> not ok 4 - output match 'Not Found'
>>>>>
>>>>> # Failed test 'output match 'Not Found''
>>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>> 1..4
>>>>> # Looks like you failed 2 tests of 4.
>>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>>> Failed 2/4 subtests
>>>>>
>>>>> Test Summary Report
>>>>> -------------------
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>>> 512 Tests: 4 Failed: 2)
>>>>> Failed tests: 3-4
>>>>> Non-zero exit status: 2
>>>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
>>>>> 0.00 csys = 0.30 CPU)
>>>>> Result: FAIL
>>>>>
>>>>> ...
>>>>>
>>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>>> https://github.com/melezhik/apache-swat
>>>>>
>>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Bill!
>>>>>>>
>>>>>>> Any news? ( Please see my previous reply ...)
>>>>>>> Intr
>>>>>>
>>>>>>
>>>>>> Intrigued :) But my responses will be delayed, I personally
>>>>>> won't have time to look further myself until other backlogged
>>>>>> commitments to httpd are caught up some more.
>>>>>>
>>>>>> Be aware many of us have only a few hours to participate in the
>>>>>> httpd project itself, so it will take several days or more to gather
>>>>>> feedback on your interest and perhaps a champion to drive such
>>>>>> an effort if it is actively driven from this side. Of course that
>>>>>> should not stop you from assembling some ideas at the vcs of
>>>>>> your choice, and sharing a link for those who want to look at an
>>>>>> early draft of such an effort!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Bill
>>>
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
> I wonder how possible it would be to transcode the old tests to Swat?
Swat has some architectural specifics. As it simply makes http
requests ( by curl ) and validate server response ( by swat DSL ) it
is considered as black box testing tools , there is no way to interact
with httpd internal structure. So probably most of old tests could be
easily ported but some probably not ( which implies something more
complicated ).
Basically swat could cover simple and average use cases, everything
that could expressed in terms of make http requests and then
analyzing an output , see examples for some issues I have expressed in
swat scenarios at https://github.com/melezhik/apache-swat
Also alternatively we may run 2 frameworks in parallel. Apache-swat is
intended for developers usage first and is able to run per-issues
test.
This how I see possible workflow:
* developer accept a bug request
* developer ( or some else ) create swat test for given bug/feature
* so we have `test first` approach - everyone involved in bug/feature
resolution has possibility to verify changes by simply running :
swat -t $issue-id
2016-02-05 17:08 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <ji...@jagunet.com>:
>> Personally, I like the idea of having another framework;
>> the current one is OK but somewhat "painful" to update.
>
> Hi! What kind of possible issues you here ? when saying "painful" to
> update, please explain.
> Thanks.
>
> Also , please consider this link - https://github.com/melezhik/apache-swat and
> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
> as latest version of this idea ....
>
>
>
>
>
>>
>> I wonder how possible it would be to transcode the old tests
>> to Swat? We could then provide for 2 testing frameworks, one
>> developed by the ASF and the other external and 3rd party.
>>
>>
>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>
>>> Here the list of existed issues I was able to automate tests for:
>>>
>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>> 44221
>>> 46751
>>> 58789
>>> 58828
>>> 58854
>>>
>>>
>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>> these, so they rely upon them. Or you have some pre-release testing,
>>> so run all of these in a whole chunk?
>>>
>>> Regards
>>>
>>> Alexey
>>>
>>> PS chaining issues / tests listing as always -
>>> https://github.com/melezhik/apache-swat
>>>
>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>> apache version gets installed from SCM, but I don't know how to do
>>> this.
>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>>
>>>
>>>
>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>>>> Hi Bill!
>>>>
>>>> I have started to assemble swat tests for apache2 issues coming from
>>>> https://bz.apache.org/bugzilla
>>>>
>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>>>
>>>> Test suite is failed for the moment :
>>>>
>>>>
>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>
>>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> not ok 4 - output match 'Not Found'
>>>>
>>>> # Failed test 'output match 'Not Found''
>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> 1..4
>>>> # Looks like you failed 2 tests of 4.
>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>> Failed 2/4 subtests
>>>>
>>>> Test Summary Report
>>>> -------------------
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>> 512 Tests: 4 Failed: 2)
>>>> Failed tests: 3-4
>>>> Non-zero exit status: 2
>>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
>>>> 0.00 csys = 0.30 CPU)
>>>> Result: FAIL
>>>>
>>>> ...
>>>>
>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>> https://github.com/melezhik/apache-swat
>>>>
>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Bill!
>>>>>>
>>>>>> Any news? ( Please see my previous reply ...)
>>>>>> Intr
>>>>>
>>>>>
>>>>> Intrigued :) But my responses will be delayed, I personally
>>>>> won't have time to look further myself until other backlogged
>>>>> commitments to httpd are caught up some more.
>>>>>
>>>>> Be aware many of us have only a few hours to participate in the
>>>>> httpd project itself, so it will take several days or more to gather
>>>>> feedback on your interest and perhaps a champion to drive such
>>>>> an effort if it is actively driven from this side. Of course that
>>>>> should not stop you from assembling some ideas at the vcs of
>>>>> your choice, and sharing a link for those who want to look at an
>>>>> early draft of such an effort!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bill
>>
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
2016-02-05 17:01 GMT+03:00 Jim Jagielski <ji...@jagunet.com>:
> Personally, I like the idea of having another framework;
> the current one is OK but somewhat "painful" to update.
Hi! What kind of possible issues you here ? when saying "painful" to
update, please explain.
Thanks.
Also , please consider this link - https://github.com/melezhik/apache-swat and
this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
as latest version of this idea ....
>
> I wonder how possible it would be to transcode the old tests
> to Swat? We could then provide for 2 testing frameworks, one
> developed by the ASF and the other external and 3rd party.
>
>
>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>
>> Here the list of existed issues I was able to automate tests for:
>>
>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>> 44221
>> 46751
>> 58789
>> 58828
>> 58854
>>
>>
>> I have informed developers at https://bz.apache.org/bugzilla/ about
>> these, so they rely upon them. Or you have some pre-release testing,
>> so run all of these in a whole chunk?
>>
>> Regards
>>
>> Alexey
>>
>> PS chaining issues / tests listing as always -
>> https://github.com/melezhik/apache-swat
>>
>> PS2 it'd be could to verify tests as well against "bleeding edge"
>> apache version gets installed from SCM, but I don't know how to do
>> this.
>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>
>>
>>
>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>>> Hi Bill!
>>>
>>> I have started to assemble swat tests for apache2 issues coming from
>>> https://bz.apache.org/bugzilla
>>>
>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>>
>>> Test suite is failed for the moment :
>>>
>>>
>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>> ok 2 - output match '200 OK'
>>> 1..2
>>> ok
>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>> ok 2 - output match '200 OK'
>>> ok 3 - output match /Location: \S+/
>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>> 1..4
>>> ok
>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>> ok 2 - output match '200 OK'
>>> ok 3 - output match /Location: \S+/
>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>> 1..4
>>> ok
>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>> ok 2 - output match '200 OK'
>>> 1..2
>>> ok
>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>
>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>> not ok 4 - output match 'Not Found'
>>>
>>> # Failed test 'output match 'Not Found''
>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>> 1..4
>>> # Looks like you failed 2 tests of 4.
>>> Dubious, test returned 2 (wstat 512, 0x200)
>>> Failed 2/4 subtests
>>>
>>> Test Summary Report
>>> -------------------
>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>> 512 Tests: 4 Failed: 2)
>>> Failed tests: 3-4
>>> Non-zero exit status: 2
>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
>>> 0.00 csys = 0.30 CPU)
>>> Result: FAIL
>>>
>>> ...
>>>
>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>> https://github.com/melezhik/apache-swat
>>>
>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>>
>>>>> Hi Bill!
>>>>>
>>>>> Any news? ( Please see my previous reply ...)
>>>>> Intr
>>>>
>>>>
>>>> Intrigued :) But my responses will be delayed, I personally
>>>> won't have time to look further myself until other backlogged
>>>> commitments to httpd are caught up some more.
>>>>
>>>> Be aware many of us have only a few hours to participate in the
>>>> httpd project itself, so it will take several days or more to gather
>>>> feedback on your interest and perhaps a champion to drive such
>>>> an effort if it is actively driven from this side. Of course that
>>>> should not stop you from assembling some ideas at the vcs of
>>>> your choice, and sharing a link for those who want to look at an
>>>> early draft of such an effort!
>>>>
>>>> Cheers,
>>>>
>>>> Bill
>
Re: collaboration request - apache server automation testing with
swat tool
Posted by Alexey Melezhik <me...@gmail.com>.
Personally, I like the idea of having another framework;
the current one is OK but somewhat "painful" to update.
Ah, now I understand you, ))), you are talking about current framework.
пятница, 5 февраля 2016 г. пользователь Jim Jagielski написал:
> Personally, I like the idea of having another framework;
> the current one is OK but somewhat "painful" to update.
>
> I wonder how possible it would be to transcode the old tests
> to Swat? We could then provide for 2 testing frameworks, one
> developed by the ASF and the other external and 3rd party.
>
>
> > On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com
> <javascript:;>> wrote:
> >
> > Here the list of existed issues I was able to automate tests for:
> >
> > vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P
> '\d'
> > 44221
> > 46751
> > 58789
> > 58828
> > 58854
> >
> >
> > I have informed developers at https://bz.apache.org/bugzilla/ about
> > these, so they rely upon them. Or you have some pre-release testing,
> > so run all of these in a whole chunk?
> >
> > Regards
> >
> > Alexey
> >
> > PS chaining issues / tests listing as always -
> > https://github.com/melezhik/apache-swat
> >
> > PS2 it'd be could to verify tests as well against "bleeding edge"
> > apache version gets installed from SCM, but I don't know how to do
> > this.
> > Currently I use http://httpd.apache.org/download.cgi ( last stable
> release )
> >
> >
> >
> > 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com
> <javascript:;>>:
> >> Hi Bill!
> >>
> >> I have started to assemble swat tests for apache2 issues coming from
> >> https://bz.apache.org/bugzilla
> >>
> >> And this the first one -
> https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
> >>
> >> Test suite is failed for the moment :
> >>
> >>
> >> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
> >> ok 2 - output match '200 OK'
> >> 1..2
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
> >> ok 2 - output match '200 OK'
> >> ok 3 - output match /Location: \S+/
> >> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> >> 1..4
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
> >> ok 2 - output match '200 OK'
> >> ok 3 - output match /Location: \S+/
> >> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
> >> 1..4
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
> >> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
> >> ok 2 - output match '200 OK'
> >> 1..2
> >> ok
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
> >> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
> >> # http headers saved to
> /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
> >> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
> >> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
> >> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
> >>
> >> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
> >> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> >> not ok 4 - output match 'Not Found'
> >>
> >> # Failed test 'output match 'Not Found''
> >> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
> >> 1..4
> >> # Looks like you failed 2 tests of 4.
> >> Dubious, test returned 2 (wstat 512, 0x200)
> >> Failed 2/4 subtests
> >>
> >> Test Summary Report
> >> -------------------
> >> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
> >> 512 Tests: 4 Failed: 2)
> >> Failed tests: 3-4
> >> Non-zero exit status: 2
> >> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
> >> 0.00 csys = 0.30 CPU)
> >> Result: FAIL
> >>
> >> ...
> >>
> >> Ok, this is just the beginning ;-))) , watch soon updates at -
> >> https://github.com/melezhik/apache-swat
> >>
> >> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net
> <javascript:;>>:
> >>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com
> <javascript:;>> wrote:
> >>>>
> >>>> Hi Bill!
> >>>>
> >>>> Any news? ( Please see my previous reply ...)
> >>>> Intr
> >>>
> >>>
> >>> Intrigued :) But my responses will be delayed, I personally
> >>> won't have time to look further myself until other backlogged
> >>> commitments to httpd are caught up some more.
> >>>
> >>> Be aware many of us have only a few hours to participate in the
> >>> httpd project itself, so it will take several days or more to gather
> >>> feedback on your interest and perhaps a champion to drive such
> >>> an effort if it is actively driven from this side. Of course that
> >>> should not stop you from assembling some ideas at the vcs of
> >>> your choice, and sharing a link for those who want to look at an
> >>> early draft of such an effort!
> >>>
> >>> Cheers,
> >>>
> >>> Bill
>
>
Re: collaboration request - apache server automation testing with swat tool
Posted by Jim Jagielski <ji...@jaguNET.com>.
Personally, I like the idea of having another framework;
the current one is OK but somewhat "painful" to update.
I wonder how possible it would be to transcode the old tests
to Swat? We could then provide for 2 testing frameworks, one
developed by the ASF and the other external and 3rd party.
> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <me...@gmail.com> wrote:
>
> Here the list of existed issues I was able to automate tests for:
>
> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
> 44221
> 46751
> 58789
> 58828
> 58854
>
>
> I have informed developers at https://bz.apache.org/bugzilla/ about
> these, so they rely upon them. Or you have some pre-release testing,
> so run all of these in a whole chunk?
>
> Regards
>
> Alexey
>
> PS chaining issues / tests listing as always -
> https://github.com/melezhik/apache-swat
>
> PS2 it'd be could to verify tests as well against "bleeding edge"
> apache version gets installed from SCM, but I don't know how to do
> this.
> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>
>
>
> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <me...@gmail.com>:
>> Hi Bill!
>>
>> I have started to assemble swat tests for apache2 issues coming from
>> https://bz.apache.org/bugzilla
>>
>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>
>> Test suite is failed for the moment :
>>
>>
>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>> ok 2 - output match '200 OK'
>> 1..2
>> ok
>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>> ok 2 - output match '200 OK'
>> ok 3 - output match /Location: \S+/
>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>> 1..4
>> ok
>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>> ok 2 - output match '200 OK'
>> ok 3 - output match /Location: \S+/
>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>> 1..4
>> ok
>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>> ok 2 - output match '200 OK'
>> 1..2
>> ok
>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>
>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>> not ok 4 - output match 'Not Found'
>>
>> # Failed test 'output match 'Not Found''
>> # at /usr/local/share/perl/5.20.2/swat.pm line 218.
>> 1..4
>> # Looks like you failed 2 tests of 4.
>> Dubious, test returned 2 (wstat 512, 0x200)
>> Failed 2/4 subtests
>>
>> Test Summary Report
>> -------------------
>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>> 512 Tests: 4 Failed: 2)
>> Failed tests: 3-4
>> Non-zero exit status: 2
>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr
>> 0.00 csys = 0.30 CPU)
>> Result: FAIL
>>
>> ...
>>
>> Ok, this is just the beginning ;-))) , watch soon updates at -
>> https://github.com/melezhik/apache-swat
>>
>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>:
>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <me...@gmail.com> wrote:
>>>>
>>>> Hi Bill!
>>>>
>>>> Any news? ( Please see my previous reply ...)
>>>> Intr
>>>
>>>
>>> Intrigued :) But my responses will be delayed, I personally
>>> won't have time to look further myself until other backlogged
>>> commitments to httpd are caught up some more.
>>>
>>> Be aware many of us have only a few hours to participate in the
>>> httpd project itself, so it will take several days or more to gather
>>> feedback on your interest and perhaps a champion to drive such
>>> an effort if it is actively driven from this side. Of course that
>>> should not stop you from assembling some ideas at the vcs of
>>> your choice, and sharing a link for those who want to look at an
>>> early draft of such an effort!
>>>
>>> Cheers,
>>>
>>> Bill