DefaultHttpAsyncClient performance problems when high-loaded

[Іван Лаппо <va...@gmail.com>]

I tested DefaultHttpAsyncClient on performing instantly growing number of
requests per minute. It works pretty well before it reaches some point,
after which client changes behavior. Number of requests executed per minute
dramatically decrease, and average response time grow up. In Statistics.txt
- number of requests executed and average response time with discretization
15 sec. It seems that after some point (after 19:38:30 in Statistics.txt)
new requests are not submitted, and only old request continue execution.
(AsyncClientTest/out/artifacts/AsyncClient_jar/AsyncClient.jar 12000 32000
80000 150000 180000 300000 400000 ) - jar with command line params - number
of requests per minute to execute.

 Also after reaching peek state I start getting
LoggingClientProtocolHandlerException - see stackTrace in
AsyncClientErrorLog.txt

 I have written own AsyncHttp client based on http.nio package using
DefaultConnectingIOReactor, decorators to NHttpClientHandler,
NHttpRequestExecutionHandler - for managing and reusing opened connections
- but it behaved the same way. (Results on puncherChart.png) on the chart
there are few params shown with discretization 15 seconds - green - number
of requests executed, red - average response time (between
connection.requestOutput() method and received response, sky-blue - number
of requests that failed to execute because of connection being
closed).Horizontal axis represents timeline, Left vertical axis - values of
these charts. I monitored state of my machine using vmstat and it shows for
first 10 minutes 4-10 % CPU, 35-40 after. This was tested on m1.small
amazon instance. With c1.medium (which has 4 times more computation CPU
power) it works the same, but with upper bound of 16000 requests per 15
seconds (against 5000 on weaker machine) - therefore it is not networking
problem.

I noticed , that asking DefaultConnectiongIOReactor for big amount of
connection can cause instant reduce of its performance ( time for opening
new connections increase very much) - therefore in my http client I limited
the number of concurrent SessionRequest for connections. Maybe for this
reason  my client acts little more adequately in this case.

Please help to figure out, how to improve AsyncClients performance.

https://sourceforge.net/projects/asyncclienttest/files/ - here there are
project and files i have mentioned

Re: DefaultHttpAsyncClient performance problems when high-loaded

[Oleg Kalnichevski <ol...@apache.org>]

On Thu, 2012-01-05 at 15:23 +0200, Іван Лаппо wrote:
> I tested DefaultHttpAsyncClient on performing instantly growing number of
> requests per minute. It works pretty well before it reaches some point,
> after which client changes behavior. Number of requests executed per minute
> dramatically decrease, and average response time grow up. In Statistics.txt
> - number of requests executed and average response time with discretization
> 15 sec. It seems that after some point (after 19:38:30 in Statistics.txt)
> new requests are not submitted, and only old request continue execution.
> (AsyncClientTest/out/artifacts/AsyncClient_jar/AsyncClient.jar 12000 32000
> 80000 150000 180000 300000 400000 ) - jar with command line params - number
> of requests per minute to execute.
> 
>  Also after reaching peek state I start getting
> LoggingClientProtocolHandlerException - see stackTrace in
> AsyncClientErrorLog.txt
> 
>  I have written own AsyncHttp client based on http.nio package using
> DefaultConnectingIOReactor, decorators to NHttpClientHandler,
> NHttpRequestExecutionHandler - for managing and reusing opened connections
> - but it behaved the same way. (Results on puncherChart.png) on the chart
> there are few params shown with discretization 15 seconds - green - number
> of requests executed, red - average response time (between
> connection.requestOutput() method and received response, sky-blue - number
> of requests that failed to execute because of connection being
> closed).Horizontal axis represents timeline, Left vertical axis - values of
> these charts. I monitored state of my machine using vmstat and it shows for
> first 10 minutes 4-10 % CPU, 35-40 after. This was tested on m1.small
> amazon instance. With c1.medium (which has 4 times more computation CPU
> power) it works the same, but with upper bound of 16000 requests per 15
> seconds (against 5000 on weaker machine) - therefore it is not networking
> problem.
> 
> I noticed , that asking DefaultConnectiongIOReactor for big amount of
> connection can cause instant reduce of its performance ( time for opening
> new connections increase very much) - therefore in my http client I limited
> the number of concurrent SessionRequest for connections. Maybe for this
> reason  my client acts little more adequately in this case.
> 
> Please help to figure out, how to improve AsyncClients performance.
> 
> https://sourceforge.net/projects/asyncclienttest/files/ - here there are
> project and files i have mentioned

I took a cursory look at the benchmark and even though I am not entirely
sure I understand what it is exactly you are trying to measure, I can
basically tell the following

(1) Benchmarks that involve remote hosts are largely meaningless. There
are simply too many factors that can distort the results

(2) 'java.io.IOException: Connection reset by peer' means that the
connection was terminated abnormally by the opposite endpoint and is
very unlikely to be HttpAsyncClient's fault (and may well be related to
point 1)

I'll try to adapt your benchmark to run against localhost or a host of
the same network and see if I can reproduce the same result pattern.

Oleg 



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org