You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Oleg Kalnichevski <ol...@apache.org> on 2011/03/11 09:15:31 UTC
HttpClient benchmark; was RE: [Httpcomponents Wiki] Update of
"HttpClient3vsHttpClient4vsHttpCore" by OlegKalnichevski
On Fri, 2011-03-11 at 07:19 +0100, Hubert, Eric wrote:
> Hi
>
> > The first section under 500'000 requests / up to 250 concurrent
> > connections does not specify the client used:
> > I assume this is probably HTTP agent: Apache HttpClient 3.1 but it
> > would be good to add it to the page.
>
> I noticed the same, sharing this assumption. ;-)
>
> Additionally I would be interested in some background information
> helping to interpret the results.
> For easier readability I put all results concentrating on just a
> Single metric in one table (truncated req/second - hope I did
> not messed some numbers).
>
> Conc 20 (get/post) Con 250 (get/post)
> Client 3.1 16170 / 16788 8188 / 9792
> JRE 6u18 21705 / 16882 14446 / 14358
> Core 4.1 31438 / 24236 19705 / 17815
> Client 4.1 25154 / 22520 21360 / 21762
> Client 4.2 24069 / 19929 21675 / 18270
> Jetty 7.2.0 7734 / 8140 19948 / 20016
> Jetty 7.3.1 17727 / 17828 20903 / 18250
>
> The following questions came into my mind
> (please excuse if answers are obvious!)
> a) Why performs 3.1 better for POSTs than for GET?
I do not have a good answer to this question, just a guess. I suspect
that it simply takes longer to generate random content on the server
side when responding to GET requests and to do a simple echo when
responding to POST requests.
> b) Why is 4.2 Http Client faster than plain Http Core 4.1 for concurrency
This one I know for sure. This is the effect of connection pooling.
HttpCore does not support connection pooling and therefore has to
maintain 250 physical connections. Apparently, for a large number of
threads fewer shared connections tend to perform better than a large
number of non-shared connections.
> level of up to 250 (for up to 20 conc. Connections it is the opposite, which
> seems to be obvious).
> c) What is the reason for the performance degradation for POST between
> Http Client 4.1 and 4.2?
Benchmark numbers do tend to fluctuate somewhat. I see no reason why HC
4.2 should be slower than 4.1. They share exactly the same core as the
moment.
> (The test runs have to be performed on the same hardware, or?
Yes. They would be meaningless otherwise.
> Only expected volatility between test runs (more than 10%)?
I can't really say. The only way to get better data / less volatility in
the benchmark in my opinion is to execute test runs longer (have more
requests to execute)
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpClient benchmark; was RE: [Httpcomponents Wiki] Update of
"HttpClient3vsHttpClient4vsHttpCore" by OlegKalnichevski
Posted by sebb <se...@gmail.com>.
On 11 March 2011 17:28, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2011-03-11 at 16:53 +0000, sebb wrote:
>> On 11 March 2011 08:15, Oleg Kalnichevski <ol...@apache.org> wrote:
>> > On Fri, 2011-03-11 at 07:19 +0100, Hubert, Eric wrote:
>> >> Hi
>> >>
>> >> > The first section under 500'000 requests / up to 250 concurrent
>> >> > connections does not specify the client used:
>> >> > I assume this is probably HTTP agent: Apache HttpClient 3.1 but it
>> >> > would be good to add it to the page.
>>
>> @Oleg: Ping?
>>
>
> I corrected it this morning. Was not there a notification mail?
Yes, I see it has been fixed - sorry, only just got around to looking
at the Wiki messages (different GMail folder)
>
>> >>
>> >> I noticed the same, sharing this assumption. ;-)
>> >>
>> >> Additionally I would be interested in some background information
>> >> helping to interpret the results.
>> >> For easier readability I put all results concentrating on just a
>> >> Single metric in one table (truncated req/second - hope I did
>> >> not messed some numbers).
>> >>
>> >> Conc 20 (get/post) Con 250 (get/post)
>> >> Client 3.1 16170 / 16788 8188 / 9792
>> >> JRE 6u18 21705 / 16882 14446 / 14358
>> >> Core 4.1 31438 / 24236 19705 / 17815
>> >> Client 4.1 25154 / 22520 21360 / 21762
>> >> Client 4.2 24069 / 19929 21675 / 18270
>> >> Jetty 7.2.0 7734 / 8140 19948 / 20016
>> >> Jetty 7.3.1 17727 / 17828 20903 / 18250
>> >>
>> >> The following questions came into my mind
>> >> (please excuse if answers are obvious!)
>> >> a) Why performs 3.1 better for POSTs than for GET?
>> >
>> > I do not have a good answer to this question, just a guess. I suspect
>> > that it simply takes longer to generate random content on the server
>> > side when responding to GET requests and to do a simple echo when
>> > responding to POST requests.
>>
>> Perhaps the echo code can be replaced with something that does a
>> similar amount of work on the server?
>>
>
> The 'echo' servlet can be replaced with something that sends back a
> random generated response of the same length as the request.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpClient benchmark; was RE: [Httpcomponents Wiki] Update of
"HttpClient3vsHttpClient4vsHttpCore" by OlegKalnichevski
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2011-03-11 at 16:53 +0000, sebb wrote:
> On 11 March 2011 08:15, Oleg Kalnichevski <ol...@apache.org> wrote:
> > On Fri, 2011-03-11 at 07:19 +0100, Hubert, Eric wrote:
> >> Hi
> >>
> >> > The first section under 500'000 requests / up to 250 concurrent
> >> > connections does not specify the client used:
> >> > I assume this is probably HTTP agent: Apache HttpClient 3.1 but it
> >> > would be good to add it to the page.
>
> @Oleg: Ping?
>
I corrected it this morning. Was not there a notification mail?
> >>
> >> I noticed the same, sharing this assumption. ;-)
> >>
> >> Additionally I would be interested in some background information
> >> helping to interpret the results.
> >> For easier readability I put all results concentrating on just a
> >> Single metric in one table (truncated req/second - hope I did
> >> not messed some numbers).
> >>
> >> Conc 20 (get/post) Con 250 (get/post)
> >> Client 3.1 16170 / 16788 8188 / 9792
> >> JRE 6u18 21705 / 16882 14446 / 14358
> >> Core 4.1 31438 / 24236 19705 / 17815
> >> Client 4.1 25154 / 22520 21360 / 21762
> >> Client 4.2 24069 / 19929 21675 / 18270
> >> Jetty 7.2.0 7734 / 8140 19948 / 20016
> >> Jetty 7.3.1 17727 / 17828 20903 / 18250
> >>
> >> The following questions came into my mind
> >> (please excuse if answers are obvious!)
> >> a) Why performs 3.1 better for POSTs than for GET?
> >
> > I do not have a good answer to this question, just a guess. I suspect
> > that it simply takes longer to generate random content on the server
> > side when responding to GET requests and to do a simple echo when
> > responding to POST requests.
>
> Perhaps the echo code can be replaced with something that does a
> similar amount of work on the server?
>
The 'echo' servlet can be replaced with something that sends back a
random generated response of the same length as the request.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
RE: HttpClient benchmark; was RE: [Httpcomponents Wiki] Update of
"HttpClient3vsHttpClient4vsHttpCore" by OlegKalnichevski
Posted by "Hubert, Eric" <Er...@jestadigital.com>.
> >> > I assume this is probably HTTP agent: Apache HttpClient 3.1 but it
> >> > would be good to add it to the page.
>
> @Oleg: Ping?
Oleg immediately did the change this morning. I got a notification mail!
Many thanks also for the helpful explanations.
> Also, repeating the tests multiple times should show if there is much
> variation.
I agree.
Also it might be interesting to keep results of runs against previous versions of the libraries (at least in a kind of summary table). Looking at the diffs of the wikipage is not that convenient - though helpful.
Thanks,
Eric
Re: HttpClient benchmark; was RE: [Httpcomponents Wiki] Update of
"HttpClient3vsHttpClient4vsHttpCore" by OlegKalnichevski
Posted by sebb <se...@gmail.com>.
On 11 March 2011 08:15, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2011-03-11 at 07:19 +0100, Hubert, Eric wrote:
>> Hi
>>
>> > The first section under 500'000 requests / up to 250 concurrent
>> > connections does not specify the client used:
>> > I assume this is probably HTTP agent: Apache HttpClient 3.1 but it
>> > would be good to add it to the page.
@Oleg: Ping?
>>
>> I noticed the same, sharing this assumption. ;-)
>>
>> Additionally I would be interested in some background information
>> helping to interpret the results.
>> For easier readability I put all results concentrating on just a
>> Single metric in one table (truncated req/second - hope I did
>> not messed some numbers).
>>
>> Conc 20 (get/post) Con 250 (get/post)
>> Client 3.1 16170 / 16788 8188 / 9792
>> JRE 6u18 21705 / 16882 14446 / 14358
>> Core 4.1 31438 / 24236 19705 / 17815
>> Client 4.1 25154 / 22520 21360 / 21762
>> Client 4.2 24069 / 19929 21675 / 18270
>> Jetty 7.2.0 7734 / 8140 19948 / 20016
>> Jetty 7.3.1 17727 / 17828 20903 / 18250
>>
>> The following questions came into my mind
>> (please excuse if answers are obvious!)
>> a) Why performs 3.1 better for POSTs than for GET?
>
> I do not have a good answer to this question, just a guess. I suspect
> that it simply takes longer to generate random content on the server
> side when responding to GET requests and to do a simple echo when
> responding to POST requests.
Perhaps the echo code can be replaced with something that does a
similar amount of work on the server?
>> b) Why is 4.2 Http Client faster than plain Http Core 4.1 for concurrency
>
> This one I know for sure. This is the effect of connection pooling.
> HttpCore does not support connection pooling and therefore has to
> maintain 250 physical connections. Apparently, for a large number of
> threads fewer shared connections tend to perform better than a large
> number of non-shared connections.
>
>
>> level of up to 250 (for up to 20 conc. Connections it is the opposite, which
>> seems to be obvious).
>> c) What is the reason for the performance degradation for POST between
>> Http Client 4.1 and 4.2?
>
> Benchmark numbers do tend to fluctuate somewhat. I see no reason why HC
> 4.2 should be slower than 4.1. They share exactly the same core as the
> moment.
>
>
>> (The test runs have to be performed on the same hardware, or?
>
> Yes. They would be meaningless otherwise.
>
>> Only expected volatility between test runs (more than 10%)?
>
> I can't really say. The only way to get better data / less volatility in
> the benchmark in my opinion is to execute test runs longer (have more
> requests to execute)
Also, repeating the tests multiple times should show if there is much variation.
> Hope this helps
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org