You are viewing a plain text version of this content. The canonical link for it is here.
Posted to test-dev@httpd.apache.org by Norman Tuttle <nt...@photon.poly.edu> on 2003/08/05 17:22:24 UTC
Flood: Bug fixes and questions for developers
Dear fellow Flood developers:
It is beneficial to us as members of the Apache Flood community and as
developers of application testing software, using Flood as the basis for
testing web sites within an application we have developed to test
web-based software, within the context of the imminent rollout of the next
revision of Flood to the public by Apache, to try to make available as
many of the bug fixes which respond to the the bugs we find to the Apache
group, as well as provide direction towards a reworking of the Flood
architecture, which we have done in order to integrate with our entire
application as well as to provide greater functionality & efficiency. In
this context, people who have asked whether Flood can be faster may also
find a suitable answer in the solutions we provide..
Therefore, in the upcoming days, as time permits, I will try to provide
these bug fixes and potential feature improvements and efficiency-building
modifications, in exchange asking for guidance on other features which we
are trying to implement (or implement better).
-Norman Tuttle, developer, OpenDemand Systems, ntuttle@opendemand.com
Below, I have begun with some comments on various topics:
1) Bugs related to keepalives (sockets):
1a) The keepalive code in Flood is error-prone, especially when dealing
with sockets which can represent SSL or non-SSL protocols. We have made
some bug fixes AND added a patch for check_sockets() to work with SSL
sockets. Flood code is especially suspect at this point when attempting to
do keepalive where there are both SSL and non-SSL pages. I believe that I
have fixed this in our code. It may take some time to convert this to the
way the Apache team wants the code (in diff format), so I am only
providing a related fix now:
1b) Keepalive chunking issue
In April, I reported a problem with deploying Flood on Sun Solaris using
keepalive sockets. It appeared that Flood was failing on an assert in the
keepalive code, related to a chunking error. Both Justin Erenkrantz and
Aaron Bannert provided me with some ideas for resolving the issue; I
had to abandon it at the time in favor of other development but have
recently revisited it and found out what was causing the problem. Thank
you both for your assistance. It happens that the chunk length in hex
coming back is padded with extra spaces at the end, and the code currently
in Flood does not allow for this, causing a later error of failing the
assert in the keepalive code (basically saying that a CRLF has not been
found at that point). My solution has been to rewrite the
keepalive_read_chunk_size() function, basically a hex evaluator function,
from scratch as below, to be more flexibile than the one currently found
in flood_socket_keepalive.c:
/* This is really a hex evaluator function. */
static long keepalive_read_chunk_size(char *begin_chunk)
{
long OutputHex=0;
unsigned char Addend;
while (isspace(*begin_chunk)) begin_chunk++;
while (isxdigit(Addend=(*(begin_chunk++))))
{
OutputHex=(OutputHex<<4)+((Addend & 0xF)+((Addend > '9')?9:0));
}
return OutputHex;
}
1c) In "real life", we find the socket structure (using higher levels
accessing pointers to structures at intermediate and lower levels) to be
obfuscating and functionally difficult to deal with (even if it has a
somewhat object-oriented feel), and my next phase of development is to
make a single layer of socket flood_socket_t directly over the apr level.
But for the Flood project I intend to first publish my earlier levels of
changes from our CVS before this modification so that you can benefit from
our bug fixes without necessarily making this modification.
2) The following questions, regarding regex and responsetemplate, are
addressed to the Flood developers on the list, and those familiar with
regex (PCRE) library and regular expressions:
We're trying to use the responses coming back from the websites to fill
variables which can be used in future expressions sent back to the
website. While I understand that regular expressions should be able to
represent an expression to match including parts which represent a
variable or optional piece of information, I am not so familiar with how
Flood is using this information to feed the responsename value. For one, I
am not sure why this information is picked up by match[1], why there is an
nmatch value of 10 passed to the regexec function when picking up the
single matched value (when we are only matching for one responsename), why
a value of 2 for the initial variable name-pattern pattern match, and what
the other match[] members might represent. This will be helpful for our
application, if anyone is knowledgeable about these features.
Thanks in advance.
-Norman Tuttle, developer, OpenDemand Systems, ntuttle@opendemand.com
Re: Flood: Bug fixes and questions for developers
Posted by Jacek Prucia <ja...@acn.waw.pl>.
On Fri, 08 Aug 2003 19:06:01 -0700
Justin Erenkrantz <je...@ics.uci.edu> wrote:
[...]
> My real life is causing me not to be able to devote time to flood right
> now, but I would love to see a new release of flood out that incorporates
> the bugfixes since 0.4.
Yeah... The docs are quite okay right now, even though there are big dark
holes. I'll just have to post a simple patch here that introduces
CONFIG_VERSION define, that would allow us to see if user tries to run
obsolete config against flood version that uses slightly different
config file (something like MMN but for config files). I have reasons for
this little puppy to get checked in before release. If that's get accepted,
then I think we're ready to roll flood 1.1.
We just have to find RM. I can't be RM since as a commiter rookie, I don't
have required access nor skills to do all the job. However I'm willing to help
as much as I can. So... somebody with enough access right and skills -- please
help with flood 1.1 release.
regards,
--
Jacek Prucia
Re: Flood: Bug fixes and questions for developers
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Saturday, August 09, 2003 10:52:03 -0700 Aaron Bannert
<aa...@clove.org> wrote:
>> I heard a report that flood 0.4 won't even compile with the latest
>> httpd/APR, so we really need to do this. (The pollset API changed.)
>
> There were recent (as in the last couple months) apr_pollset changes?
> I thought we took care of those already once.
No, AIUI, the code in CVS will compile against the latest releases just
fine, but the flood-0.4 tarball won't compile against the current releases
as it uses an APR API that has since been removed.
IIRC, flood-0.4 was from last July or August. I think pollset API changed
last September or something. -- justin
Re: Flood: Bug fixes and questions for developers
Posted by Aaron Bannert <aa...@clove.org>.
On Friday, August 8, 2003, at 07:06 PM, Justin Erenkrantz wrote:
> I heard a report that flood 0.4 won't even compile with the latest
> httpd/APR, so we really need to do this. (The pollset API changed.)
There were recent (as in the last couple months) apr_pollset changes?
I thought we took care of those already once.
-aaron
Re: Flood: Bug fixes and questions for developers
Posted by Justin Erenkrantz <je...@ics.uci.edu>.
--On Tuesday, August 05, 2003 11:22:24 -0400 Norman Tuttle
<nt...@photon.poly.edu> wrote:
> Dear fellow Flood developers:
>
> It is beneficial to us as members of the Apache Flood community and as
> developers of application testing software, using Flood as the basis for
> testing web sites within an application we have developed to test
> web-based software, within the context of the imminent rollout of the next
> revision of Flood to the public by Apache, to try to make available as
> many of the bug fixes which respond to the the bugs we find to the Apache
> group, as well as provide direction towards a reworking of the Flood
*Please* go for it!
My real life is causing me not to be able to devote time to flood right
now, but I would love to see a new release of flood out that incorporates
the bugfixes since 0.4. I heard a report that flood 0.4 won't even compile
with the latest httpd/APR, so we really need to do this. (The pollset API
changed.)
If I get a chance, I will try to look at the rest of your questions later,
but that's probably a bit on the unrealistic side. I'd suggest just
posting patches of what you think is best. -- justin
Re: Flood: Bug fixes and questions for developers
Posted by Jacek Prucia <j....@defbank.com.pl>.
[Note to moderator: I'm not able to use my normal e-mail address (jacek.prucia@acn.waw.pl) right now, so please pass me through just once -- Thanks :)]
On Tue, 5 Aug 2003 11:22:24 -0400 (EDT)
Norman Tuttle <nt...@photon.poly.edu> wrote:
> Dear fellow Flood developers:
[...]
> 2) The following questions, regarding regex and responsetemplate, are
> addressed to the Flood developers on the list, and those familiar with
> regex (PCRE) library and regular expressions:
>
> We're trying to use the responses coming back from the websites to fill
> variables which can be used in future expressions sent back to the
> website. While I understand that regular expressions should be able to
> represent an expression to match including parts which represent a
> variable or optional piece of information, I am not so familiar with how
> Flood is using this information to feed the responsename value. For one, I
> am not sure why this information is picked up by match[1],
Most regexp packages works that way. The first item in matches table (that is: match[0]) is the portion of the input string that was used for matching. You can assume that everytime it is exact replica of input string. Personally I have no idea what this is good for.
> why there is an nmatch value of 10 passed to the regexec function when picking up the
> single matched value (when we are only matching for one responsename),
I have absolutelly no idea. The result is that if there are more matches, they end up in match[2], match[3] and so on, but as you can tell from the following code -- only the first match is used. Maybe Justin or Aaron can shed some light on this...
> why a value of 2 for the initial variable name-pattern pattern match,
it just makes sure, that regexec returns array with 2 elements:
match[0] -- entire string
match[1] -- first match
So even when your regexp matches twice or more -- regexec returns only first match. Because of that users are required to finetune their regexp's, so that they are unique across whole response.
> and what the other match[] members might represent.
simply, next matches. If your regexp is quite generic it might match more than one time. Look at regexp from round-robin-dynamic.xml (XML specific encoding stripped)
/<a href="([^"]*)">/
given some HTML you can have:
match[0] = ... // whole HTML
match[1] = http://www.apache.org/
match[2] = http://cvs.apache.org/
match[2] = http://perl.apache.org/
and so on, you get the idea...
Personally I think that regexp matching code schould be rewritten at some point, so that you can have any number of regexps against response, and every match turned into flood variable. However, this is more work with config file than with the code. Because of XML restrictions (attribute names must be unique) we have to change url element to something like this:
<url>
<address>http://www.example.com/</address>
<postprocess>
<regexp>
<pattern><a href="([^"]*);"></pattern>
<matches>
<var>first_match</var>
<var>second_match</var>
<!-- any number of variables -->
</matches>
</regexp>
<regexp>
<!-- another regexp -->
</regexp>
</postprocess>
</url>
This is however serious change (breaks existing configs), so it schould be scheduled for major flood rewrite (like flood 2.0 with apr-serf on board :)
regards,
Jacek Prucia