Benchmark oddities

[sebb <se...@gmail.com>]

I've been trying out the Benchmark tests on a WinXP box.

There is some strange behaviour I cannot explain.

HC3 and HClient4 both tend to generate a single error in the GET test.
I added debug to the test, and it turns out that it is a read timeout.

In the case of HC4, I tried increasing the timeout, and it still
occurs even at 85 seconds, which is longer than the test normally
takes.
So I assume this request will never complete.

Increasing the timeout in HC3.1 to 85 secs does not result in a
timeout any more.

This is a bit odd; it suggests there may be an issue in HC4.1.

The Ning test was generating lots of errors. These turned out to be
"already bound" errors, which (on Windows XP at least) can be
generated when the OS runs out of sockets.
So I tried a run of just the Ning test, and it seems to generate
hundreds of sockets.
It does not seem to re-use connections, unlike HC3.1 and HC4.
Not sure if this is a Ning bug or a bug in the test harness.
The Ning test always fails when I run it as part of the full benchmark
suite; I think that is partly because it runs last, which means there
may be fewer sockets available.

I need to do some more tests, in particular to check connection re-use
more carefully.
Suggestions welcome on what might cause the timeout (and how to debug further).

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org

Re: Benchmark oddities

[Oleg Kalnichevski <ol...@apache.org>]

On Mon, 2011-11-28 at 02:15 +0000, sebb wrote:
> I've been trying out the Benchmark tests on a WinXP box.
> 
> There is some strange behaviour I cannot explain.
> 
> HC3 and HClient4 both tend to generate a single error in the GET test.
> I added debug to the test, and it turns out that it is a read timeout.
> 

I observed a somewhat similar phenomenon and always suspected that
something was going wrong on the Jetty side when it was hit with the
burst of simultaneous connect requests. I never saw request timeouts
when running the benchmark against a standalone Tomcat instance on the
same platform. My theory may well be wrong, though.


> In the case of HC4, I tried increasing the timeout, and it still
> occurs even at 85 seconds, which is longer than the test normally
> takes.
> So I assume this request will never complete.
> 
> Increasing the timeout in HC3.1 to 85 secs does not result in a
> timeout any more.
> 
> This is a bit odd; it suggests there may be an issue in HC4.1.
> 
> The Ning test was generating lots of errors. These turned out to be
> "already bound" errors, which (on Windows XP at least) can be
> generated when the OS runs out of sockets.
> So I tried a run of just the Ning test, and it seems to generate
> hundreds of sockets.
> It does not seem to re-use connections, unlike HC3.1 and HC4.
> Not sure if this is a Ning bug or a bug in the test harness.
> The Ning test always fails when I run it as part of the full benchmark
> suite; I think that is partly because it runs last, which means there
> may be fewer sockets available.
> 

Ning async HTTP client never performed well for me and was annoyingly
finicky on Windows. It tends to do better on Linux, though.

> I need to do some more tests, in particular to check connection re-use
> more carefully.
> Suggestions welcome on what might cause the timeout (and how to debug further).
> 

I suggest that you drop those benchmark samples that are irrelevant for
JMeter (Jetty and Ning) and concentrate on benchmarking HC3.1, HC4.1,
and HTTP JRE only. Try running benchmark samples separately against a
local server (HTTPD or Tomcat) instead of using embedded Jetty and see
if that makes any difference. 

Hope this helps

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org