You are viewing a plain text version of this content. The canonical link for it is here.
Posted to test-dev@httpd.apache.org by Justin Erenkrantz <je...@ebuilt.com> on 2001/08/24 07:35:50 UTC
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
On Fri, Aug 24, 2001 at 04:26:39AM -0000, jerenkrantz@apache.org wrote:
> jerenkrantz 01/08/23 21:26:39
>
> Modified: flood config.h.in configure.in flood.c
> flood_easy_reports.c flood_farm.c flood_net.c
> flood_net_ssl.c flood_report_relative_times.c
> Log:
> Per Sander, Linux 2.4 returns EAGAIN when we run out of ports.
This was a royal screw-up on my part. I only meant to commit
flood_net.c, but all of the fork stuff slipped in. It wasn't
really ready for commit. But, uh, yeah, it's there now. =)
I'll test it some more (it works) and then I'll commit lines to CHANGES
with what I really changed on this commit. *sigh* -- justin
Future of flood
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Aug 24, 2001 at 12:05:59AM -0700, Aaron Bannert wrote:
> Unfortunately that is not the case. A urllist is simply a list of
> urls, it has no implicit ideas of being serial or parallel. We have
> only implemented a roundrobin-style profile, which happens to use
> a urllist in a serial manner, but there is no reason the same
> urllist couldn't be used by a hundred farmers all in parallel
> and in some bizarre non-linear manner (like random-weighted, for
> example).
Okay, point taken. The code and typical usage intends for it to be
serial. Think of all the response/request regex stuff. That
implication is fairly strong. Yes, you can replace the round robin
profile with something completely separate. However, I don't see
anyone rushing to do that.
This may be where we differ from httperf the most - they are solely a
framework for a test tool - their tool, as is, can't do lots of stuff.
Flood, while a framework itself, is also a test tool that stands by
itself. There's a difference between the framework section and the
actual implementation of flood. Consider it design vs.
implementation.
I consider flood's design period to be over and it's implementation
drawing to a close. I don't expect/want to do much more coding with
it. Minor features and such. Possibly new protocols (waka?), but
nothing earth-shattering. Yes, bucket brigades would be nice. But,
I'm ready to move on. I don't want to live with this code for the
rest of my life...
It's done what I/eBuilt wanted and it's done it fairly well. The
framework is laid down for others to come through and continue our
goals. This is sort of why I want to switch to evangelical mode
and get people *using* flood. If people use it, they will see what
they like about it and what they dislike about it. They can
then continue on with the development of this tool. I can't take
it much farther than I already have...
This is also an ideal time for us to stop and try and document it so
that other people can pick up on flood. Why is it a good time?
I'm going on vacation for three weeks next week (as is Roy).
You're going to start a new job soon. We're not going to be able
to babysit this project much longer.
My goal with the user docs is to document what we HAVE right now.
The design docs can indicate what we intend. But, as far as the
current implementation goes, no one (other than you or I) care
about the design too much - they want to use it. (And, if you
think about how httpd started, the people who *used* NCSA took
over *development* because they had changes they wished to see
in the product that weren't there in the first place.)
How to use flood intelligently isn't something we have any docs
on right now. I hope I can come up with stuff so that Apache
Random Hacker can figure it out. If they want to change the
code or the design after finding flaws, it's open source - sit
down, read the code, and have lots of fun. I'll even make sure
anyone has commit access to this repository if they want it.
-- justin
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Aug 23, 2001 at 11:26:39PM -0700, Justin Erenkrantz wrote:
> On Thu, Aug 23, 2001 at 11:13:16PM -0700, Aaron Bannert wrote:
> > I thought mpmt was what we were striving for all along for platforms that
> > could support it. *BSD is the only platform we're really worried about
> > WRT threads, no?
>
> It's the most glaring omission. =)
>
> > as much as possible". If you think we can emulate the typical 4-threaded
> > GUI browser by treating threads and processes the same, then I am totally
> > cool with that.
>
> I think that is our best option at the farm level. On platforms without
> threads, this is as close as we can get it.
>
> I think the key thing with the farm is that the count variable is
> intended to be run in parallel not serial (unlike the count for a
> urllist which is implicitly serial).
Unfortunately that is not the case. A urllist is simply a list of
urls, it has no implicit ideas of being serial or parallel. We have
only implemented a roundrobin-style profile, which happens to use
a urllist in a serial manner, but there is no reason the same
urllist couldn't be used by a hundred farmers all in parallel
and in some bizarre non-linear manner (like random-weighted, for
example).
> If we allow a farm to also
> be run in serial, then a test that can simulate 10 users by creating
> 10 threads on platform X, but simulates 1 user doing the same
> thing 10x as long on platform Y isn't good. So, yes, I think a farm
> must be run in parallel. =-) If that's not in the docs, then too
> bad.
You're missing my point. Although it matters to me that we have the
ability to run multiple farmers in parallel, it also matters to me that
we define these things beforehand. We discussed all of this before we
started coding and we decided that it would be 2-level, to basicly do an
MPMT model. The patch you commited implicitly changes our definitions of
"collective" and "farm", but you seem reluctant to keep the design docs
in sync, or to acknowledge that this may limit our ability to accurately
represent some real-world scenarios. Like I said, I'd rather have the
flexibility to mix and match threads and processes, instead of making
it transparent to the user how the test was made parallel.
Maybe a better model would be simple 2-layer where each layer can
be implemented using threads or processes to achieve parallel processing
of the next level, only that the top level prefers forks and the bottom
layer prefers threads. Some creative coding could save us some code
duplication and it would satisfy both of the goals we are trying to
achieve here.
> FWIW, I'm going to be spending some time on the flood site/docs
> tomorrow and over the weekend. Roy's added the subproject members
> (you and jsachs) to httpd-site, so you should be able to commit
> there now too. -- justin
Cool, that will be useful.
-aaron
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Thu, Aug 23, 2001 at 11:13:16PM -0700, Aaron Bannert wrote:
> I thought mpmt was what we were striving for all along for platforms that
> could support it. *BSD is the only platform we're really worried about
> WRT threads, no?
It's the most glaring omission. =)
> as much as possible". If you think we can emulate the typical 4-threaded
> GUI browser by treating threads and processes the same, then I am totally
> cool with that.
I think that is our best option at the farm level. On platforms without
threads, this is as close as we can get it.
I think the key thing with the farm is that the count variable is
intended to be run in parallel not serial (unlike the count for a
urllist which is implicitly serial). If we allow a farm to also
be run in serial, then a test that can simulate 10 users by creating
10 threads on platform X, but simulates 1 user doing the same
thing 10x as long on platform Y isn't good. So, yes, I think a farm
must be run in parallel. =-) If that's not in the docs, then too
bad.
FWIW, I'm going to be spending some time on the flood site/docs
tomorrow and over the weekend. Roy's added the subproject members
(you and jsachs) to httpd-site, so you should be able to commit
there now too. -- justin
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Aug 23, 2001 at 10:54:35PM -0700, Justin Erenkrantz wrote:
> On Thu, Aug 23, 2001 at 10:47:57PM -0700, Aaron Bannert wrote:
> > On Thu, Aug 23, 2001 at 10:35:50PM -0700, Justin Erenkrantz wrote:
> > > On Fri, Aug 24, 2001 at 04:26:39AM -0000, jerenkrantz@apache.org wrote:
> > > > jerenkrantz 01/08/23 21:26:39
> > > >
> > > > Modified: flood config.h.in configure.in flood.c
> > > > flood_easy_reports.c flood_farm.c flood_net.c
> > > > flood_net_ssl.c flood_report_relative_times.c
> > > > Log:
> > > > Per Sander, Linux 2.4 returns EAGAIN when we run out of ports.
> > >
> > > This was a royal screw-up on my part. I only meant to commit
> > > flood_net.c, but all of the fork stuff slipped in. It wasn't
> > > really ready for commit. But, uh, yeah, it's there now. =)
> > >
> > > I'll test it some more (it works) and then I'll commit lines to CHANGES
> > > with what I really changed on this commit. *sigh* -- justin
> >
> > After looking through the changes I'm not so sure this is what we want.
> > I'd rather not treat threads and forked processes the same, since we may
> > want to mix the two. The fork()ed scenario was what we originally defined
> > as a "collective", which is basicly a bunch of "farms" with each "farm"
> > running in a single process. I'm worried that we're losing flexibility
> > here, at least in terms of properly emulating some real-world clients
> > (like a browser that truely uses threads). This may just be invalid
> > paranoia on my part, but at this time I see no good reason to restrict
> > ourselves.
>
> But, what do we do about the farm semantics when running on a
> non-threaded system? Do nothing? Ignore their parameters? That's
> not good (I'd say it violates principle of least surprise - we shouldn't
> change how we interpret a test based on the OS we are running on).
I agree, and that's probably why I put off doing it in the first place :)
I think my solution was to just drop down and do the farm tests in serial.
I've never heard of the principle of least surprise, what do you mean? We
document how it's supposed to work (*before we write it, hopefully*) then
if it doesn't work the way we said it would you are allowed to be surprised.
> A collective on a non-threaded system would do forks upon forks. On a
> threaded system, it'd be mpmt (what we'd expect).
I thought mpmt was what we were striving for all along for platforms that
could support it. *BSD is the only platform we're really worried about
WRT threads, no?
Let me put it a different way. I'm fine with the idea of interchanging
threads with processes for running "farmers" as long as that satisfies
all of our requirements. I don't think our requirements are limited to
"make the damn fastest load-testing harness you can possibly make". I
want it to also include "and make sure it can match real-world clients
as much as possible". If you think we can emulate the typical 4-threaded
GUI browser by treating threads and processes the same, then I am totally
cool with that.
> > I like using conditional compilation for things like optimizing OpenSSL
> > usage, but I don't like it for deciding what a "farmer" does. Anyway,
> > I realize that this was a premature commit, but I thought I'd voice my
> > objections anyway ;)
> >
> > BTW, I'm glad to see we're going to support fork()ing soon :)
>
> Yeah, I actually wrote that code before I even compiled. There was
> only one error in logic. Impressive. =) -- justin
not bad :)
-aaron
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Thu, Aug 23, 2001 at 10:47:57PM -0700, Aaron Bannert wrote:
> On Thu, Aug 23, 2001 at 10:35:50PM -0700, Justin Erenkrantz wrote:
> > On Fri, Aug 24, 2001 at 04:26:39AM -0000, jerenkrantz@apache.org wrote:
> > > jerenkrantz 01/08/23 21:26:39
> > >
> > > Modified: flood config.h.in configure.in flood.c
> > > flood_easy_reports.c flood_farm.c flood_net.c
> > > flood_net_ssl.c flood_report_relative_times.c
> > > Log:
> > > Per Sander, Linux 2.4 returns EAGAIN when we run out of ports.
> >
> > This was a royal screw-up on my part. I only meant to commit
> > flood_net.c, but all of the fork stuff slipped in. It wasn't
> > really ready for commit. But, uh, yeah, it's there now. =)
> >
> > I'll test it some more (it works) and then I'll commit lines to CHANGES
> > with what I really changed on this commit. *sigh* -- justin
>
> After looking through the changes I'm not so sure this is what we want.
> I'd rather not treat threads and forked processes the same, since we may
> want to mix the two. The fork()ed scenario was what we originally defined
> as a "collective", which is basicly a bunch of "farms" with each "farm"
> running in a single process. I'm worried that we're losing flexibility
> here, at least in terms of properly emulating some real-world clients
> (like a browser that truely uses threads). This may just be invalid
> paranoia on my part, but at this time I see no good reason to restrict
> ourselves.
But, what do we do about the farm semantics when running on a
non-threaded system? Do nothing? Ignore their parameters? That's
not good (I'd say it violates principle of least surprise - we shouldn't
change how we interpret a test based on the OS we are running on).
A collective on a non-threaded system would do forks upon forks. On a
threaded system, it'd be mpmt (what we'd expect).
> I like using conditional compilation for things like optimizing OpenSSL
> usage, but I don't like it for deciding what a "farmer" does. Anyway,
> I realize that this was a premature commit, but I thought I'd voice my
> objections anyway ;)
>
> BTW, I'm glad to see we're going to support fork()ing soon :)
Yeah, I actually wrote that code before I even compiled. There was
only one error in logic. Impressive. =) -- justin
Re: cvs commit: httpd-test/flood config.h.in configure.in flood.c flood_easy_reports.c flood_farm.c flood_net.c flood_net_ssl.c flood_report_relative_times.c
Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Aug 23, 2001 at 10:35:50PM -0700, Justin Erenkrantz wrote:
> On Fri, Aug 24, 2001 at 04:26:39AM -0000, jerenkrantz@apache.org wrote:
> > jerenkrantz 01/08/23 21:26:39
> >
> > Modified: flood config.h.in configure.in flood.c
> > flood_easy_reports.c flood_farm.c flood_net.c
> > flood_net_ssl.c flood_report_relative_times.c
> > Log:
> > Per Sander, Linux 2.4 returns EAGAIN when we run out of ports.
>
> This was a royal screw-up on my part. I only meant to commit
> flood_net.c, but all of the fork stuff slipped in. It wasn't
> really ready for commit. But, uh, yeah, it's there now. =)
>
> I'll test it some more (it works) and then I'll commit lines to CHANGES
> with what I really changed on this commit. *sigh* -- justin
After looking through the changes I'm not so sure this is what we want.
I'd rather not treat threads and forked processes the same, since we may
want to mix the two. The fork()ed scenario was what we originally defined
as a "collective", which is basicly a bunch of "farms" with each "farm"
running in a single process. I'm worried that we're losing flexibility
here, at least in terms of properly emulating some real-world clients
(like a browser that truely uses threads). This may just be invalid
paranoia on my part, but at this time I see no good reason to restrict
ourselves.
I like using conditional compilation for things like optimizing OpenSSL
usage, but I don't like it for deciding what a "farmer" does. Anyway,
I realize that this was a premature commit, but I thought I'd voice my
objections anyway ;)
BTW, I'm glad to see we're going to support fork()ing soon :)
-aaron