You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2018/07/31 16:35:00 UTC
[Bug 62590] New: performance drop after moving from apache 2.2 to
apache 2.4
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
Bug ID: 62590
Summary: performance drop after moving from apache 2.2 to
apache 2.4
Product: Apache httpd-2
Version: 2.4.34
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: bugs@httpd.apache.org
Reporter: paolo@adnovum.ch
Target Milestone: ---
Hi,
I have got an important performance drop after moving from apache httpd/2.2 to
apache httpd/2.4.
In order to tail down the problem, I did the following.
I downloaded, built and installed Apache/2.2.34. I did add ssl-support and the
following lines to httpd.conv:
StartServers 1
ServerLimit 1
ThreadLimit 2500
ThreadsPerChild 2500
ThreadStackSize 1048576
MinSpareThreads 1
MaxSpareThreads 500
MaxClients 2500
MaxRequestsPerChild 0
Here the output using ApacheBench:
Apache 2.2:
ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking adnvl005 (be patient)
Completed 200000 requests
Completed 400000 requests
Completed 600000 requests
Completed 800000 requests
Completed 1000000 requests
Completed 1200000 requests
Completed 1400000 requests
Completed 1600000 requests
Completed 1800000 requests
Completed 2000000 requests
Finished 2000000 requests
Server Software: Apache/2.2.34
Server Hostname: adnvl005
Server Port: 44300
SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
Document Path: /
Document Length: 44 bytes
Concurrency Level: 60
Time taken for tests: 192.131 seconds
Complete requests: 2000000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1980237
Total transferred: 731130414 bytes
HTML transferred: 88000000 bytes
Requests per second: 10409.59 [#/sec] (mean)
Time per request: 5.764 [ms] (mean)
Time per request: 0.096 [ms] (mean, across all concurrent requests)
Transfer rate: 3716.20 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 87
Processing: 0 6 1.2 6 71
Waiting: 0 6 1.2 5 70
Total: 0 6 1.3 6 98
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 8
98% 9
99% 10
100% 98 (longest request)
----
I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with
the following changes in the httpd.conf (including ssl-support):
StartServers 1
ServerLimit 1
ThreadLimit 2500
ThreadsPerChild 2500
ThreadStackSize 1048576
MinSpareThreads 1
MaxSpareThreads 500
MaxRequestWorkers 2500
MaxConnectionsPerChild 0
and here the output of ApacheBench:
ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking adnvl005 (be patient)
Completed 200000 requests
Completed 400000 requests
Completed 600000 requests
Completed 800000 requests
Completed 1000000 requests
Completed 1200000 requests
Completed 1400000 requests
Completed 1600000 requests
Completed 1800000 requests
Completed 2000000 requests
Finished 2000000 requests
Apache 2.4:
Server Software: Apache/2.4.34
Server Hostname: adnvl005
Server Port: 44300
SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
Document Path: /
Document Length: 45 bytes
Concurrency Level: 60
Time taken for tests: 571.058 seconds
Complete requests: 2000000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1980229
Total transferred: 689130083 bytes
HTML transferred: 90000000 bytes
Requests per second: 3502.27 [#/sec] (mean)
Time per request: 17.132 [ms] (mean)
Time per request: 0.286 [ms] (mean, across all concurrent requests)
Transfer rate: 1178.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.1 0 208
Processing: 0 17 20.3 10 285
Waiting: 0 17 20.3 10 285
Total: 0 17 20.4 10 285
Percentage of the requests served within a certain time (ms)
50% 10
66% 16
75% 23
80% 28
90% 44
95% 59
98% 79
99% 94
100% 285 (longest request)
----
As you can see, with apache2.4 it took more as twice as long. I think this is a
bug.
This performance drop occurs only if ssl is involved. For non-ssl request there
is no performance impact.
There are no error in neither logs.
For technical reasons I can't increase the ServerLimit.
Please let me know if you need any other information.
Many thanks for your help.
Regards
Paolo
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
veit <ve...@adnovum.ch> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |veit@adnovum.ch
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #2 from Eric Covener <co...@gmail.com> ---
(In reply to paolo from comment #1)
> After some debug-session I found out that the problem are the
> ERR_clear_error calls in ssl_filter_write and ssl_io_input_read. If I remove
> those calls the performance is the same like with httpd/2.2.
>
> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
> function?
> Isn't it enough to have it only in the ssl_io_filter_handshake function.
>
> Or what about to call this function only if an error occurred:
>
> else /* (rc < 0) */ {
> int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
> conn_rec *c =
> (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
>
> + ERR_clear_error();
>
> if (ssl_err == SSL_ERROR_WANT_READ) {
>
>
> Many thanks for any answer.
ref: https://bz.apache.org/bugzilla/show_bug.cgi?id=60223
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #4 from paolo <pa...@adnovum.ch> ---
I used for both tests the ab installed on my host:
paolo@adnvl005:~ ab -V
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/\
Apart that I get to the same performance issue if using other software, like
gatling etc.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #6 from William A. Rowe Jr. <wr...@apache.org> ---
> Apart that I get to the same performance issue if using other software, like gatling etc.
Thanks Paolo!
Presuming the other software is on the same machine, relying on the same
openssl client library, what is your system's response to;
$ openssl version
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #5 from William A. Rowe Jr. <wr...@apache.org> ---
Confirmed in dev@ discussion, this flavor of ab dates to the 2.2.x epoch and is
consistent for both (observed with 'This is ApacheBench, Version 2.3
<$Revision: 655654 $>'.)
Because of the transition of TLS session cache to TLS session tokens, it may be
worthwhile to see if this is only true of very ancient clients, and whether
this combination of an older ab with an older openssl is in part to blame for
the failure to speak efficiently to modern httpd linked to modern openssl. This
can be tested using the most recent ab utility against both servers.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #7 from paolo <pa...@adnovum.ch> ---
Hi William,
here the openssl-version which is installed on my host:
paolo@adnvl005:~/ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #3 from William A. Rowe Jr. <wr...@apache.org> ---
For 100% clarity, this was observed with the version of ab shipped with 2.2.34,
or the version shipped with 2.4.24? It should be obvious that ab has also
undergone some enhancements and changes for support of TLS, and the two
different versions are expected to produce two different results.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
Hello,
On 2018-08-28 10:54, William A Rowe Jr wrote:
> As we unwind various regressions and breakage, one non-lethal but
> somewhat horrid report stands out. Eric correctly tied it to the patch
> applied for https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the
> 2.4.24 timeframe.
I'd like to comment on the last entry in the ticket:
---
For 100% clarity, this was observed with the version of ab shipped with
2.2.34, or the version shipped with 2.4.24? It should be obvious that ab
has also undergone some enhancements and changes for support of TLS, and
the two different versions are expected to produce two different results.
---
It shouldn't matter which version was used as long as the same one was
used for both tests.
According to the ab output, Paolo used the same version for both tests:
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Unfortunately this output is useless unless you know the revision
numbers by hart. It looks like it is a 2.2 version, but I could be
wrong. (A 2.4 version has a revision around 1826891.)
Note to devs: it would be great, if ab could use the same version
numbers as the server. ;-)
Cheers,
K. C.
--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Re: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by Stefan Eissing <st...@greenbytes.de>.
Just for interest ran my h2 load tests on trunk/2.4.x unpatched and with commenting ERR_clear_error(); in ssl_engine_io.c
On my Mac I see no differences, really. trunk used openssl 1.0.2o and 2.4.x had 1.0.2l:
> h2load -i gen/load-urls-1.txt -n 200000 -t 4 -m 100 -c <clients>
trunk
raw, 128 clients: 33940.36 req/s
patched, 128 clients: 33699.89 req/s
2.4.x
raw, 128 clients: 33417.12 req/s
patched, 128 clients: 34071.83 req/s
(just some runs, nothing with any statistical relevance)
Caveat: this does not necessarily say anything about the http/1.1 performance. h2 buffers its writes into he ssl output filters more than h1 does, I think.
Cheers, Stefan
> Am 29.08.2018 um 09:55 schrieb Ruediger Pluem <rp...@apache.org>:
>
>
>
> On 08/29/2018 09:37 AM, Christophe Jaillet wrote:
>> Pure speculation:
>>
>> Actually we ERR_clear_error unconditionally so that in case of error, we can safely call SSL_get_error.
>>
>> Doc says:
>> <<<<<<<<<< >>>>>>>>>>
>> ERR_get_error() returns the earliest error code from the thread's error queue and removes the entry. This function can be called repeatedly until there are no more error codes to return.
>>
>> ERR_peek_error() returns the earliest error code from the thread's error queue without modifying it.
>>
>> ERR_peek_last_error() returns the latest error code from the thread's error queue without modifying it.
>> <<<<<<<<<< >>>>>>>>>>
>>
>> Couldn't we avoid the ERR_clear_error call (which looks expensive according to the thread), and:
>> - loop on SSL_get_error to empty the error queue, in case an error occurred and we want la latest one?
>> or
>> - do a ERR_peek_last_error() / ERR_clear_error() in the error handling path (if we really care about clearing the error queue)
>>
>> IMHO, these 2 solutions would do the same as the current code, without requiring ERR_clear_error in the normal case.
>
> Unfortunately it isn't that easy, because SSL_get_error does not only look at the error stack but also at the return
> code from a read / write call passed to it. It quickly breaks out with two possible error return values if something
> is on the error stack and this what caused the addition of the ERR_clear_error call. We had the situation that another
> consumer of openssl in the code base left an error unhandled on the stack and thus caused SSL_get_error to draw the
> wrong conclusions. While we could argue the error is in the other part of the code and the other consumer of openssl
> should clear up the error stack correctly I don't think this is the complete solution, because
>
> 1. We should not fail if a another consumer is bogus. We should be prepared for others to have bugs regarding this.
> 2. As the error handling is a stack I think it is a valid use case for another consumer of openssl to check for the
> error later and not directly like e.g. with errno.
>
> I am not sure yet if it is not SSL_get_error that makes wrong assumptions about the error stack, but that would be
> something hard to solve for us.
>
> Regards
>
> Rüdiger
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by Ruediger Pluem <rp...@apache.org>.
On 08/29/2018 10:49 AM, Christophe Jaillet wrote:
> Ok, it is trickier than I thought.
>
> On 2018/08/29 07:55:04, Ruediger Pluem <rp...@apache.org> wrote:
>>
>> 2. As the error handling is a stack I think it is a valid use case for another consumer of openssl to check for the
>> error later and not directly like e.g. with errno.
>>
>
> but in this case, haven't we also broken these behaviors (i.e. leave error on stack for later processing) when we have added some more ERR_clear_error?
Agreed, but obviously that wasn't noticed yet :-).
Regards
Rüdiger
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
Posted by Christophe Jaillet <ja...@apache.org>.
Ok, it is trickier than I thought.
On 2018/08/29 07:55:04, Ruediger Pluem <rp...@apache.org> wrote:
>
> 2. As the error handling is a stack I think it is a valid use case for another consumer of openssl to check for the
> error later and not directly like e.g. with errno.
>
but in this case, haven't we also broken these behaviors (i.e. leave error on stack for later processing) when we have added some more ERR_clear_error?
CJ
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by Ruediger Pluem <rp...@apache.org>.
On 08/29/2018 09:37 AM, Christophe Jaillet wrote:
> Pure speculation:
>
> Actually we ERR_clear_error unconditionally so that in case of error, we can safely call SSL_get_error.
>
> Doc says:
> <<<<<<<<<< >>>>>>>>>>
> ERR_get_error() returns the earliest error code from the thread's error queue and removes the entry. This function can be called repeatedly until there are no more error codes to return.
>
> ERR_peek_error() returns the earliest error code from the thread's error queue without modifying it.
>
> ERR_peek_last_error() returns the latest error code from the thread's error queue without modifying it.
> <<<<<<<<<< >>>>>>>>>>
>
> Couldn't we avoid the ERR_clear_error call (which looks expensive according to the thread), and:
> - loop on SSL_get_error to empty the error queue, in case an error occurred and we want la latest one?
> or
> - do a ERR_peek_last_error() / ERR_clear_error() in the error handling path (if we really care about clearing the error queue)
>
> IMHO, these 2 solutions would do the same as the current code, without requiring ERR_clear_error in the normal case.
Unfortunately it isn't that easy, because SSL_get_error does not only look at the error stack but also at the return
code from a read / write call passed to it. It quickly breaks out with two possible error return values if something
is on the error stack and this what caused the addition of the ERR_clear_error call. We had the situation that another
consumer of openssl in the code base left an error unhandled on the stack and thus caused SSL_get_error to draw the
wrong conclusions. While we could argue the error is in the other part of the code and the other consumer of openssl
should clear up the error stack correctly I don't think this is the complete solution, because
1. We should not fail if a another consumer is bogus. We should be prepared for others to have bugs regarding this.
2. As the error handling is a stack I think it is a valid use case for another consumer of openssl to check for the
error later and not directly like e.g. with errno.
I am not sure yet if it is not SSL_get_error that makes wrong assumptions about the error stack, but that would be
something hard to solve for us.
Regards
Rüdiger
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
Posted by Christophe Jaillet <ja...@apache.org>.
Pure speculation:
Actually we ERR_clear_error unconditionally so that in case of error, we can safely call SSL_get_error.
Doc says:
<<<<<<<<<< >>>>>>>>>>
ERR_get_error() returns the earliest error code from the thread's error queue and removes the entry. This function can be called repeatedly until there are no more error codes to return.
ERR_peek_error() returns the earliest error code from the thread's error queue without modifying it.
ERR_peek_last_error() returns the latest error code from the thread's error queue without modifying it.
<<<<<<<<<< >>>>>>>>>>
Couldn't we avoid the ERR_clear_error call (which looks expensive according to the thread), and:
- loop on SSL_get_error to empty the error queue, in case an error occurred and we want la latest one?
or
- do a ERR_peek_last_error() / ERR_clear_error() in the error handling path (if we really care about clearing the error queue)
IMHO, these 2 solutions would do the same as the current code, without requiring ERR_clear_error in the normal case.
Just my 2c. I'm not an ssl/tls man.
CJ
On 2018/08/29 07:17:04, Ruediger Pluem <rp...@apache.org> wrote:
> Having a brief look at ERR_clear_error in openssl it looks like that various locking operations are performed to get the
> error state. I assume that this causes a lot of contention in the load case.
> The question is whether there is another solution to BZ62590 but to call ERR_clear_error before every read and write. So
> far, I have no idea for an efficient check of not processed errors in the stack and only clear if they are present /
> consider them correctly for further error processing.
>
> Regards
>
> Rüdiger
>
>
> On 08/29/2018 09:16 AM, Ruediger Pluem wrote:
> > I guess it would be helpful to know which openssl version was used for the test. There was a valid use case for BZ62590
> > and the question is why ERR_clear_error is that expensive. But the expensiveness might depend on the used openssl version.
> >
> > Regards
> >
> > Rüdiger
> >
> >
> > On 08/28/2018 04:54 PM, William A Rowe Jr wrote:
> >> As we unwind various regressions and breakage, one non-lethal but somewhat horrid report stands out. Eric correctly tied
> >> it to the patch applied for https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the 2.4.24 timeframe.
> >>
> >> Server Software: Apache/2.2.34
> >> SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
> >>
> >> vs
> >>
> >> Server Software: Apache/2.4.34SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
> >>
> >> Measures out with
> >>
> >> Time taken for tests: 192.131 seconds
> >> Total transferred: 731130414 bytes
> >> HTML transferred: 88000000 bytes
> >> Requests per second: 10409.59 [#/sec] (mean)
> >> Time per request: 5.764 [ms] (mean)
> >> Time per request: 0.096 [ms] (mean, across all concurrent requests)
> >> Transfer rate: 3716.20 [Kbytes/sec] received
> >>
> >> vs
> >>
> >> Time taken for tests: 571.058 seconds
> >> Total transferred: 689130083 bytes
> >> HTML transferred: 90000000 bytes
> >> Requests per second: 3502.27 [#/sec] (mean)
> >> Time per request: 17.132 [ms] (mean)
> >> Time per request: 0.286 [ms] (mean, across all concurrent requests)
> >> Transfer rate: 1178.48 [Kbytes/sec] received
> >>
> >>
> >>
> >>
> >> Connection Times (ms)
> >> min mean[+/-sd] median max
> >> Connect: 0 0 0.4 0 87
> >> Processing: 0 6 1.2 6 71
> >> Waiting: 0 6 1.2 5 70
> >> Total: 0 6 1.3 6 98
> >>
> >> Percentage of the requests served within a certain time (ms)
> >> 50% 6
> >> 66% 6
> >> 75% 6
> >> 80% 6
> >> 90% 7
> >> 95% 8
> >> 98% 9
> >> 99% 10
> >> 100% 98 (longest request)
> >>
> >> ----
> >>
> >> I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support):
> >> StartServers 1
> >> ServerLimit 1
> >> ThreadLimit 2500
> >> ThreadsPerChild 2500
> >> ThreadStackSize 1048576
> >> MinSpareThreads 1
> >> MaxSpareThreads 500
> >> MaxRequestWorkers 2500
> >> MaxConnectionsPerChild 0
> >>
> >>
> >> and here the output of ApacheBench:
> >>
> >> ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
> >> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> >> Licensed to The Apache Software Foundation, http://www.apache.org/
> >>
> >> Benchmarking adnvl005 (be patient)
> >> Completed 200000 requests
> >> Completed 400000 requests
> >> Completed 600000 requests
> >> Completed 800000 requests
> >> Completed 1000000 requests
> >> Completed 1200000 requests
> >> Completed 1400000 requests
> >> Completed 1600000 requests
> >> Completed 1800000 requests
> >> Completed 2000000 requests
> >> Finished 2000000 requests
> >>
> >>
> >> Connection Times (ms)
> >> min mean[+/-sd] median max
> >> Connect: 0 0 2.1 0 208
> >> Processing: 0 17 20.3 10 285
> >> Waiting: 0 17 20.3 10 285
> >> Total: 0 17 20.4 10 285
> >>
> >> Percentage of the requests served within a certain time (ms)
> >> 50% 10
> >> 66% 16
> >> 75% 23
> >> 80% 28
> >> 90% 44
> >> 95% 59
> >> 98% 79
> >> 99% 94
> >> 100% 285 (longest request)
> >>
> >>
> >>
> >>
> >> ---------- Forwarded message ----------
> >> From: ** <bugzilla@apache.org <ma...@apache.org>>
> >> Date: Mon, Aug 27, 2018 at 9:11 AM
> >> Subject: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
> >> To: bugs@httpd.apache.org <ma...@httpd.apache.org>
> >>
> >>
> >> https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 <https://bz.apache.org/bugzilla/show_bug.cgi?id=62590>
> >>
> >> --- Comment #1 from paolo <paolo@adnovum.ch <ma...@adnovum.ch>> ---
> >> After some debug-session I found out that the problem are the ERR_clear_error
> >> calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
> >> performance is the same like with httpd/2.2.
> >>
> >> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
> >> function?
> >> Isn't it enough to have it only in the ssl_io_filter_handshake function.
> >>
> >> Or what about to call this function only if an error occurred:
> >>
> >> else /* (rc < 0) */ {
> >> int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
> >> conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
> >>
> >> + ERR_clear_error();
> >>
> >> if (ssl_err == SSL_ERROR_WANT_READ) {
> >>
> >>
> >> Many thanks for any answer.
> >>
> >> --
> >> You are receiving this mail because:
> >> You are the assignee for the bug.
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org <ma...@httpd.apache.org>
> >> For additional commands, e-mail: bugs-help@httpd.apache.org <ma...@httpd.apache.org>
> >>
> >>
> >
>
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by Ruediger Pluem <rp...@apache.org>.
Having a brief look at ERR_clear_error in openssl it looks like that various locking operations are performed to get the
error state. I assume that this causes a lot of contention in the load case.
The question is whether there is another solution to BZ62590 but to call ERR_clear_error before every read and write. So
far, I have no idea for an efficient check of not processed errors in the stack and only clear if they are present /
consider them correctly for further error processing.
Regards
Rüdiger
On 08/29/2018 09:16 AM, Ruediger Pluem wrote:
> I guess it would be helpful to know which openssl version was used for the test. There was a valid use case for BZ62590
> and the question is why ERR_clear_error is that expensive. But the expensiveness might depend on the used openssl version.
>
> Regards
>
> Rüdiger
>
>
> On 08/28/2018 04:54 PM, William A Rowe Jr wrote:
>> As we unwind various regressions and breakage, one non-lethal but somewhat horrid report stands out. Eric correctly tied
>> it to the patch applied for https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the 2.4.24 timeframe.
>>
>> Server Software: Apache/2.2.34
>> SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>>
>> vs
>>
>> Server Software: Apache/2.4.34SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>>
>> Measures out with
>>
>> Time taken for tests: 192.131 seconds
>> Total transferred: 731130414 bytes
>> HTML transferred: 88000000 bytes
>> Requests per second: 10409.59 [#/sec] (mean)
>> Time per request: 5.764 [ms] (mean)
>> Time per request: 0.096 [ms] (mean, across all concurrent requests)
>> Transfer rate: 3716.20 [Kbytes/sec] received
>>
>> vs
>>
>> Time taken for tests: 571.058 seconds
>> Total transferred: 689130083 bytes
>> HTML transferred: 90000000 bytes
>> Requests per second: 3502.27 [#/sec] (mean)
>> Time per request: 17.132 [ms] (mean)
>> Time per request: 0.286 [ms] (mean, across all concurrent requests)
>> Transfer rate: 1178.48 [Kbytes/sec] received
>>
>>
>>
>>
>> Connection Times (ms)
>> min mean[+/-sd] median max
>> Connect: 0 0 0.4 0 87
>> Processing: 0 6 1.2 6 71
>> Waiting: 0 6 1.2 5 70
>> Total: 0 6 1.3 6 98
>>
>> Percentage of the requests served within a certain time (ms)
>> 50% 6
>> 66% 6
>> 75% 6
>> 80% 6
>> 90% 7
>> 95% 8
>> 98% 9
>> 99% 10
>> 100% 98 (longest request)
>>
>> ----
>>
>> I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support):
>> StartServers 1
>> ServerLimit 1
>> ThreadLimit 2500
>> ThreadsPerChild 2500
>> ThreadStackSize 1048576
>> MinSpareThreads 1
>> MaxSpareThreads 500
>> MaxRequestWorkers 2500
>> MaxConnectionsPerChild 0
>>
>>
>> and here the output of ApacheBench:
>>
>> ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
>> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> Licensed to The Apache Software Foundation, http://www.apache.org/
>>
>> Benchmarking adnvl005 (be patient)
>> Completed 200000 requests
>> Completed 400000 requests
>> Completed 600000 requests
>> Completed 800000 requests
>> Completed 1000000 requests
>> Completed 1200000 requests
>> Completed 1400000 requests
>> Completed 1600000 requests
>> Completed 1800000 requests
>> Completed 2000000 requests
>> Finished 2000000 requests
>>
>>
>> Connection Times (ms)
>> min mean[+/-sd] median max
>> Connect: 0 0 2.1 0 208
>> Processing: 0 17 20.3 10 285
>> Waiting: 0 17 20.3 10 285
>> Total: 0 17 20.4 10 285
>>
>> Percentage of the requests served within a certain time (ms)
>> 50% 10
>> 66% 16
>> 75% 23
>> 80% 28
>> 90% 44
>> 95% 59
>> 98% 79
>> 99% 94
>> 100% 285 (longest request)
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: ** <bugzilla@apache.org <ma...@apache.org>>
>> Date: Mon, Aug 27, 2018 at 9:11 AM
>> Subject: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
>> To: bugs@httpd.apache.org <ma...@httpd.apache.org>
>>
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 <https://bz.apache.org/bugzilla/show_bug.cgi?id=62590>
>>
>> --- Comment #1 from paolo <paolo@adnovum.ch <ma...@adnovum.ch>> ---
>> After some debug-session I found out that the problem are the ERR_clear_error
>> calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
>> performance is the same like with httpd/2.2.
>>
>> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
>> function?
>> Isn't it enough to have it only in the ssl_io_filter_handshake function.
>>
>> Or what about to call this function only if an error occurred:
>>
>> else /* (rc < 0) */ {
>> int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
>> conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
>>
>> + ERR_clear_error();
>>
>> if (ssl_err == SSL_ERROR_WANT_READ) {
>>
>>
>> Many thanks for any answer.
>>
>> --
>> You are receiving this mail because:
>> You are the assignee for the bug.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org <ma...@httpd.apache.org>
>> For additional commands, e-mail: bugs-help@httpd.apache.org <ma...@httpd.apache.org>
>>
>>
>
Re: Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by Ruediger Pluem <rp...@apache.org>.
I guess it would be helpful to know which openssl version was used for the test. There was a valid use case for BZ62590
and the question is why ERR_clear_error is that expensive. But the expensiveness might depend on the used openssl version.
Regards
Rüdiger
On 08/28/2018 04:54 PM, William A Rowe Jr wrote:
> As we unwind various regressions and breakage, one non-lethal but somewhat horrid report stands out. Eric correctly tied
> it to the patch applied for https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the 2.4.24 timeframe.
>
> Server Software: Apache/2.2.34
> SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>
> vs
>
> Server Software: Apache/2.4.34SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
>
> Measures out with
>
> Time taken for tests: 192.131 seconds
> Total transferred: 731130414 bytes
> HTML transferred: 88000000 bytes
> Requests per second: 10409.59 [#/sec] (mean)
> Time per request: 5.764 [ms] (mean)
> Time per request: 0.096 [ms] (mean, across all concurrent requests)
> Transfer rate: 3716.20 [Kbytes/sec] received
>
> vs
>
> Time taken for tests: 571.058 seconds
> Total transferred: 689130083 bytes
> HTML transferred: 90000000 bytes
> Requests per second: 3502.27 [#/sec] (mean)
> Time per request: 17.132 [ms] (mean)
> Time per request: 0.286 [ms] (mean, across all concurrent requests)
> Transfer rate: 1178.48 [Kbytes/sec] received
>
>
>
>
> Connection Times (ms)
> min mean[+/-sd] median max
> Connect: 0 0 0.4 0 87
> Processing: 0 6 1.2 6 71
> Waiting: 0 6 1.2 5 70
> Total: 0 6 1.3 6 98
>
> Percentage of the requests served within a certain time (ms)
> 50% 6
> 66% 6
> 75% 6
> 80% 6
> 90% 7
> 95% 8
> 98% 9
> 99% 10
> 100% 98 (longest request)
>
> ----
>
> I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support):
> StartServers 1
> ServerLimit 1
> ThreadLimit 2500
> ThreadsPerChild 2500
> ThreadStackSize 1048576
> MinSpareThreads 1
> MaxSpareThreads 500
> MaxRequestWorkers 2500
> MaxConnectionsPerChild 0
>
>
> and here the output of ApacheBench:
>
> ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
> This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Licensed to The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking adnvl005 (be patient)
> Completed 200000 requests
> Completed 400000 requests
> Completed 600000 requests
> Completed 800000 requests
> Completed 1000000 requests
> Completed 1200000 requests
> Completed 1400000 requests
> Completed 1600000 requests
> Completed 1800000 requests
> Completed 2000000 requests
> Finished 2000000 requests
>
>
> Connection Times (ms)
> min mean[+/-sd] median max
> Connect: 0 0 2.1 0 208
> Processing: 0 17 20.3 10 285
> Waiting: 0 17 20.3 10 285
> Total: 0 17 20.4 10 285
>
> Percentage of the requests served within a certain time (ms)
> 50% 10
> 66% 16
> 75% 23
> 80% 28
> 90% 44
> 95% 59
> 98% 79
> 99% 94
> 100% 285 (longest request)
>
>
>
>
> ---------- Forwarded message ----------
> From: ** <bugzilla@apache.org <ma...@apache.org>>
> Date: Mon, Aug 27, 2018 at 9:11 AM
> Subject: [Bug 62590] performance drop after moving from apache 2.2 to apache 2.4
> To: bugs@httpd.apache.org <ma...@httpd.apache.org>
>
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 <https://bz.apache.org/bugzilla/show_bug.cgi?id=62590>
>
> --- Comment #1 from paolo <paolo@adnovum.ch <ma...@adnovum.ch>> ---
> After some debug-session I found out that the problem are the ERR_clear_error
> calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
> performance is the same like with httpd/2.2.
>
> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
> function?
> Isn't it enough to have it only in the ssl_io_filter_handshake function.
>
> Or what about to call this function only if an error occurred:
>
> else /* (rc < 0) */ {
> int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
> conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
>
> + ERR_clear_error();
>
> if (ssl_err == SSL_ERROR_WANT_READ) {
>
>
> Many thanks for any answer.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org <ma...@httpd.apache.org>
> For additional commands, e-mail: bugs-help@httpd.apache.org <ma...@httpd.apache.org>
>
>
Fwd: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
As we unwind various regressions and breakage, one non-lethal but somewhat
horrid report stands out. Eric correctly tied it to the patch applied for
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 in the 2.4.24
timeframe.
Server Software: Apache/2.2.34
SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
vs
Server Software: Apache/2.4.34SSL/TLS Protocol:
TLSv1/SSLv3,AES256-GCM-SHA384,1024,256
Measures out with
Time taken for tests: 192.131 seconds Total transferred: 731130414 bytes
HTML transferred: 88000000 bytes Requests per second: 10409.59 [#/sec]
(mean) Time per request: 5.764 [ms] (mean) Time per request: 0.096 [ms]
(mean, across all concurrent requests) Transfer rate: 3716.20 [Kbytes/sec]
received
vs
Time taken for tests: 571.058 seconds
Total transferred: 689130083 bytes
HTML transferred: 90000000 bytes
Requests per second: 3502.27 [#/sec] (mean)
Time per request: 17.132 [ms] (mean)
Time per request: 0.286 [ms] (mean, across all concurrent requests)
Transfer rate: 1178.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 87
Processing: 0 6 1.2 6 71
Waiting: 0 6 1.2 5 70
Total: 0 6 1.3 6 98
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 8
98% 9
99% 10
100% 98 (longest request)
----
I did then the same for Apache/2.4.34 (with apr-1.6.3 and
apr-util-1.6.1), with the following changes in the httpd.conf
(including ssl-support):
StartServers 1
ServerLimit 1
ThreadLimit 2500
ThreadsPerChild 2500
ThreadStackSize 1048576
MinSpareThreads 1
MaxSpareThreads 500
MaxRequestWorkers 2500
MaxConnectionsPerChild 0
and here the output of ApacheBench:
ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking adnvl005 (be patient)
Completed 200000 requests
Completed 400000 requests
Completed 600000 requests
Completed 800000 requests
Completed 1000000 requests
Completed 1200000 requests
Completed 1400000 requests
Completed 1600000 requests
Completed 1800000 requests
Completed 2000000 requests
Finished 2000000 requests
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.1 0 208
Processing: 0 17 20.3 10 285
Waiting: 0 17 20.3 10 285
Total: 0 17 20.4 10 285
Percentage of the requests served within a certain time (ms)
50% 10
66% 16
75% 23
80% 28
90% 44
95% 59
98% 79
99% 94
100% 285 (longest request)
---------- Forwarded message ----------
From: <bu...@apache.org>
Date: Mon, Aug 27, 2018 at 9:11 AM
Subject: [Bug 62590] performance drop after moving from apache 2.2 to
apache 2.4
To: bugs@httpd.apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #1 from paolo <pa...@adnovum.ch> ---
After some debug-session I found out that the problem are the
ERR_clear_error
calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
performance is the same like with httpd/2.2.
Are those calls really needed in the ssl_io_input_read/ssl_filter_write
function?
Isn't it enough to have it only in the ssl_io_filter_handshake function.
Or what about to call this function only if an error occurred:
else /* (rc < 0) */ {
int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
conn_rec *c = (conn_rec*)SSL_get_app_data(
inctx->filter_ctx->pssl);
+ ERR_clear_error();
if (ssl_err == SSL_ERROR_WANT_READ) {
Many thanks for any answer.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62590] performance drop after moving from apache 2.2 to apache
2.4
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
--- Comment #1 from paolo <pa...@adnovum.ch> ---
After some debug-session I found out that the problem are the ERR_clear_error
calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the
performance is the same like with httpd/2.2.
Are those calls really needed in the ssl_io_input_read/ssl_filter_write
function?
Isn't it enough to have it only in the ssl_io_filter_handshake function.
Or what about to call this function only if an error occurred:
else /* (rc < 0) */ {
int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
+ ERR_clear_error();
if (ssl_err == SSL_ERROR_WANT_READ) {
Many thanks for any answer.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org