On what HelloWorldExample performance predicts

[Roy Wilson <de...@bellatlantic.net>]

Craig,
Craig: I agree with this, and would word it slightly differently:  the 
only thing a "Hello, World" benchmark predicts is how fast your 
application can say "Hello, World".  :-)
Roy: Well, not quite: it also predicts an upper bound on the avg thruput, 
and a lower bound bound on the average response time, of any more 
resource-intensive request on the machine where the resource measurements 
were taken.
Bell Labs uses this kind of bounding approach to performance assessment, 
especially in the design phase.
Roy

The Ping servlet

[Roy Wilson <de...@bellatlantic.net>]

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 11/8/00, 10:48:07 PM, "Remy Maucherat" <re...@apache.org> wrote regarding 
Re: Thruput, response, and perf servlets:


> Thruput, response, and perf servlets>Remy, I should have said 
"throughput,
> as in number of requests
> >processed per second". My point is that if  some servlet S requires TC
> >to use more disk+cpu than the HelloWorldExample on a particular
> >machine, then TC's processing rate for S will be lower than it is for
> >the HelloWorldExample.

> It does way more than nothing, as Costin (I think) pointed out :
>         ResourceBundle rb =
>             ResourceBundle.getBundle("LocalStrings",request.getLocale());
>         response.setContentType("text/html");
> So there's disk access (which is very bad).

All software does something, yes? (even if it's only a no-op).

> For maximum number of requests / s, I need a servlet whose service 
function
> is empty.

I'm confused: I don't see how what I've said  and the above statement are 
in disagreement.

> >Yes, but the response time for processing HelloWorldExample servlets
> >should be less than the avg response time for processing a servlet S
> >which "does more" than HelloWorldExample.

> With the IO caused by the resource bundle loading, I'm not sure.

I was trying to make a general point, which is also (I think) what Craig 
was up to, not a statement about a particular servlet. Unfortunately, he 
and I both used the HelloWorldServlet to illustrate that point.

> >I think we agree, but stated things differently. Or, maybe not :).

> We do, but the HelloWorldServlet does more than :
>         PrintWriter out = response.getWriter();
>         out.println("Hello");

> >A kind of synthetic workload? There are commercial products that do
> >that sort of thing, I think.

> Great. It's not that I don't like commercial software, but I think we 
could
> use a free test suite.

> >So, I propose a workload model based on servlets we already have (or
> >could develop for). Starements about TC performance are relative to a
> >transportable set of requests and there is no need to get into wars
> >about which version of TC is better. Too simple, I know. This is a bit
> >like what you propose about a "perf" webapp maybe?

> Np. I still want the maximum performance case, though ;) Real people will
> probably not be interested by it, but developers who may want to help
> optimizing Tomcat would find them helpful.
> Could we do both so that everyone's happy ?

Yes (but since when can everyone be happy? :-))

In "Java Performance and Scalability, Vol 1", Dov Bulka takes a similar 
approach to what you've described, I think. He defines a PingServlet, 
which he describes as "the world's fastest servlet" and uses it as a 
baseline for evaluating the perf of more "real" servlets. He also 
describes it as a "performance upper bound": it provides a way to 
"evaluate the upper limit on the throughput of the web application server 
under  test." 

I can send the PingServlet code if you're interested.

Roy 

> Remy


> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org

Re: Thruput, response, and perf servlets

[Remy Maucherat <re...@apache.org>]

Thruput, response, and perf servlets>Remy, I should have said "throughput,
as in number of requests
>processed per second". My point is that if  some servlet S requires TC
>to use more disk+cpu than the HelloWorldExample on a particular
>machine, then TC's processing rate for S will be lower than it is for
>the HelloWorldExample.

It does way more than nothing, as Costin (I think) pointed out :
        ResourceBundle rb =
            ResourceBundle.getBundle("LocalStrings",request.getLocale());
        response.setContentType("text/html");
So there's disk access (which is very bad).
For maximum number of requests / s, I need a servlet whose service function
is empty.

>Yes, but the response time for processing HelloWorldExample servlets
>should be less than the avg response time for processing a servlet S
>which "does more" than HelloWorldExample.

With the IO caused by the resource bundle loading, I'm not sure.

>I think we agree, but stated things differently. Or, maybe not :).

We do, but the HelloWorldServlet does more than :
        PrintWriter out = response.getWriter();
        out.println("Hello");

>A kind of synthetic workload? There are commercial products that do
>that sort of thing, I think.

Great. It's not that I don't like commercial software, but I think we could
use a free test suite.

>So, I propose a workload model based on servlets we already have (or
>could develop for). Starements about TC performance are relative to a
>transportable set of requests and there is no need to get into wars
>about which version of TC is better. Too simple, I know. This is a bit
>like what you propose about a "perf" webapp maybe?

Np. I still want the maximum performance case, though ;) Real people will
probably not be interested by it, but developers who may want to help
optimizing Tomcat would find them helpful.
Could we do both so that everyone's happy ?

Remy

Thruput, response, and perf servlets

[Roy Wilson <de...@bellatlantic.net>]

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 11/8/00, 7:15:09 PM, "Remy Maucherat" <re...@apache.org> wrote regarding 
Re: On what HelloWorldExample performance predicts:


> On what HelloWorldExample performance predicts>Roy: Well, not quite: it 
also
> predicts an upper bound on the avg thruput, and a lower >bound bound on 
the
> average response time, of any more resource-intensive request on >the
> machine where the resource measurements were taken.

> I don't agree with that.
> The upper bound to thoughtput would be measured on a servlet which would
> output a significant chunk of data (more than 50K, I would say), while 
not
> doing anything else. 

Remy, I should have said "throughput, as in number of requests processed 
per second". My point is that if  some servlet S requires TC to use more 
disk+cpu than the HelloWorldExample on a particular machine, then TC's 
processing rate for S will be lower than it is for the HelloWorldExample.

> Hello world outputs a small amount of data (the time
> spent parsing the request becomes more significant), and does other stuff 
in
> the process (IO, ...).
> The lower bound to response time is a servlet which does nothing.

Yes, but the response time for processing HelloWorldExample servlets 
should be less than the avg response time for processing a servlet S 
which "does more" than HelloWorldExample.

I think we agree, but stated things differently. Or, maybe not :). 

> How about creating a new "perf" webapp which would contain a selection of
> tests ?

A kind of synthetic workload? There are commercial products that do that 
sort of thing, I think. My guess is that, for OS work, it's too difficult 
to arrange. I think the ab approach is a positive because it allows us to 
capture information about how TC behaves across a variety of hw/sw 
environments. It's rough info, but it's potentially consistent, and 
potentially extensible. 

I think it's important to design a test to give reasonable information 
about server response as a function of workload characteristics (resource 
usage), load, and environment (hw/sw). By this I mean, suppose a 
potential user HOMER (as in Simpson?)says:

I'm thinking of running TC on dual Celeron 400MHz, 256MB SD100 RAM, WD 
Caviar 10.2MB/5400RPM system. There might be (say) 1500 users on a 
company Intranet who each make a single request, wait 5 seconds, then 
make another. 60% of the requests are very simple, kind of like 
HelloWorldExample, 30% are kind of like ReqInfoExample, and 10% are IO 
intensive, shoving out 50K each. [Obviously, a very simple scenario: but 
remember, this is HOMER :-)).

HOMER may be interested in some estimate of what the server throughput is 
likely to be for each type of request, what the server response time is 
likely to be, and what the CPU and disk utilizations are like (since they 
might want to put TC on a system with some room for growth).

So, I propose a workload model based on servlets we already have (or 
could develop for). Starements about TC performance are relative to a 
transportable set of requests and there is no need to get into wars about 
which version of TC is better. Too simple, I know. This is a bit like 
what you propose about a "perf" webapp maybe?

Roy

Re: On what HelloWorldExample performance predicts

[Remy Maucherat <re...@apache.org>]

On what HelloWorldExample performance predicts>Roy: Well, not quite: it also
predicts an upper bound on the avg thruput, and a lower >bound bound on the
average response time, of any more resource-intensive request on >the
machine where the resource measurements were taken.

I don't agree with that.
The upper bound to thoughtput would be measured on a servlet which would
output a significant chunk of data (more than 50K, I would say), while not
doing anything else. Hello world outputs a small amount of data (the time
spent parsing the request becomes more significant), and does other stuff in
the process (IO, ...).
The lower bound to response time is a servlet which does nothing.

How about creating a new "perf" webapp which would contain a selection of
tests ?

Remy

TC overhead, etc.

[Roy Wilson <de...@bellatlantic.net>]

Costin,

How about a simple example to illustrate the calculation?

Roy

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 11/9/00, 11:40:06 AM, cmanolache@yahoo.com wrote regarding Re: On what 
HelloWorldExample performance predicts:


> > Roy: Well, not quite: it also predicts an upper bound on the avg thruput,
> > and a lower bound bound on the average response time, of any more
> > resource-intensive request on the machine where the resource measurements
> > were taken.

> And even more - it shows the tomcat overhead ( how much time is tomcat
> adding to each request ) - and it can be used to find other "timings" -
> for example a servlet that does getParameter() or getHeader() or a
> dispatch.

> The dispatch time is very important ( and it allows you to make decisions
> about the way you design critical pieces of your software ) - and to find
> it you need the time of a servlet doing a number of dispatches minus the
> tomcat overhead ( that you got from HelloWorld ).

> It's not an exact science, but it's not just a random thing, without any
> value. Your app response time can be estimated - at least the lower
> bounds, based on what your app is doing.

> Costin


> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org

Re: On what HelloWorldExample performance predicts

[cm...@yahoo.com]

> Roy: Well, not quite: it also predicts an upper bound on the avg thruput, 
> and a lower bound bound on the average response time, of any more 
> resource-intensive request on the machine where the resource measurements 
> were taken.

And even more - it shows the tomcat overhead ( how much time is tomcat
adding to each request ) - and it can be used to find other "timings" -
for example a servlet that does getParameter() or getHeader() or a
dispatch.

The dispatch time is very important ( and it allows you to make decisions
about the way you design critical pieces of your software ) - and to find
it you need the time of a servlet doing a number of dispatches minus the
tomcat overhead ( that you got from HelloWorld ).

It's not an exact science, but it's not just a random thing, without any
value. Your app response time can be estimated - at least the lower
bounds, based on what your app is doing.

Costin