You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2017/10/30 07:56:03 UTC
[Bug 61695] New: Load Avarage problem with 2.4.29
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
Bug ID: 61695
Summary: Load Avarage problem with 2.4.29
Product: Apache httpd-2
Version: 2.4.29
Hardware: All
OS: Linux
Status: NEW
Severity: major
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: bernhard.pfeiffer@brz.gv.at
Target Milestone: ---
top - 08:53:28 up 37 days, 2:13, 2 users, load average: 273.69, 269.66,
286.06
Tasks: 333 total, 13 running, 320 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.6%us, 68.3%sy, 0.0%ni, 28.1%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 64521M total, 34465M used, 30056M free, 1749M buffers
Swap: 32767M total, 38M used, 32729M free, 21997M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
48695 nobody 20 0 538m 45m 4456 S 187 0.1 75:36.02 httpd
60804 nobody 20 0 602m 44m 4472 S 184 0.1 68:50.45 httpd
46940 nobody 20 0 538m 42m 4472 S 163 0.1 88:37.97 httpd
47659 nobody 20 0 538m 45m 4460 S 157 0.1 89:51.49 httpd
47093 nobody 20 0 538m 44m 4432 S 152 0.1 84:53.72 httpd
52089 nobody 20 0 475m 42m 4460 S 151 0.1 74:03.62 httpd
46944 nobody 20 0 538m 44m 4456 S 149 0.1 84:18.83 httpd
64521 nobody 20 0 602m 43m 4468 S 130 0.1 68:19.66 httpd
52117 nobody 20 0 539m 44m 4476 S 129 0.1 70:20.57 httpd
52611 nobody 20 0 475m 42m 4480 S 128 0.1 69:36.91 httpd
47198 nobody 20 0 538m 44m 4444 S 123 0.1 77:32.66 httpd
48596 nobody 20 0 539m 43m 4484 S 120 0.1 74:33.53 httpd
47130 nobody 20 0 539m 45m 4476 S 116 0.1 80:53.40 httpd
48632 nobody 20 0 539m 43m 4476 S 115 0.1 75:57.15 httpd
46941 nobody 20 0 539m 43m 4460 S 102 0.1 78:12.21 httpd
47515 nobody 20 0 602m 42m 4496 S 82 0.1 73:44.02 httpd
Since the new version of httpd-2.4.29, there are some problems with the load
avarage on our linux system. It is the same config, like 2.4.28 before. No
changes...
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #16 from Jacob Lundberg <ja...@collegenet.com> ---
(In reply to Stefan Eissing from comment #15)
> Jacob and Bernd: are you both talking about the same system?
No, and I apologize if it turns out I hijacked this bug for a different
problem. It sounded very similar (high load started at 2.4.29 and only with
HTTP/2 enabled).
> Both manipulate list data structure and - unless memory is corrupted here -
> they cannot go into an infinite loop there.
A week later (and all points in between I've tried) the same thread will still
be at 100% CPU and will show the same stack trace, down to the URL (which I've
left out of the stack traces I've provided). This continues until we restart
(not graceful) the httpd.
> I have no good idea how to move forward on this, I must say.
Unfortunately I am not familiar with the httpd code, but if you do think of
anything else for me to try I will attempt it if I can do so without disrupting
our production services.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #7 from Bernhard Pfeiffer <be...@brz.gv.at> ---
top - 14:26:20 up 37 days, 7:46, 3 users, load average: 24.07, 7.53, 3.78
Tasks: 336 total, 4 running, 332 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.2%us, 65.7%sy, 0.0%ni, 28.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 64521M total, 50441M used, 14079M free, 1741M buffers
Swap: 32767M total, 38M used, 32729M free, 37658M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20576 nobody 20 0 473m 32m 3932 S 289 0.1 2:37.45 httpd
20574 nobody 20 0 472m 32m 3980 S 280 0.1 3:45.05 httpd
20773 nobody 20 0 472m 32m 3932 S 257 0.0 2:14.78 httpd
20774 nobody 20 0 472m 32m 3944 S 244 0.0 1:25.65 httpd
20728 nobody 20 0 472m 31m 3656 S 220 0.0 1:31.67 httpd
20853 nobody 20 0 537m 31m 3924 S 207 0.0 1:11.62 httpd
20696 nobody 20 0 473m 31m 4016 S 184 0.0 1:49.45 httpd
21058 nobody 20 0 408m 28m 3484 S 131 0.0 0:15.62 httpd
20937 nobody 20 0 536m 30m 3644 S 98 0.0 0:45.32 httpd
42627 tomcat 20 0 10.3g 4.1g 17m S 89 6.6 740:01.42 java
20660 nobody 20 0 472m 31m 3940 S 87 0.0 3:11.22 httpd
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #4 from Bernhard Pfeiffer <be...@brz.gv.at> ---
httpd-ssl
SetEnvIf User-Agent ".*MSIE [2-6]\..*" nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
SetEnvIf User-Agent ".*MSIE [17-9].*" ssl-unclean-shutdown
Header always set Strict-Transport-Security "max-age=31536000;
includeSubDomains"
Header always edit Set-Cookie (.*) "$1;HttpOnly;Secure"
#SSL Stappling
SSLUseStapling off
SSLOptions +ExportCertData +StrictRequire +OptRenegotiate
<Location /secure-services>
SSLVerifyClient require
SSLRenegBufferSize 100000000
</Location>
SSLVerifyDepth 2
<FilesMatch "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/data0/apache/stp/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
CustomLog /logs/apache_xxx/xxx.cal.local_ssl-request_log "%t %h
%{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%{User-agent}i\" \"%r\" %b"
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #5 from Bernhard Pfeiffer <be...@brz.gv.at> ---
The new version will come into problems, if there is heavy load!
Older Version (2.4.28) had no problems with the configuration.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
Jacob Lundberg <ja...@collegenet.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jacob@collegenet.com
--- Comment #12 from Jacob Lundberg <ja...@collegenet.com> ---
We are using Debian's packaging (basically). They do not appear to be using
-fno-strict-aliasing so I am rebuilding with it added to the C and C++ flags
and will test. It will take a few days to be certain of the result if it
appears to be successful. The problem only reproduces after some hours in our
fairly high-concurrency production environment.
Sample build command from the build output (this is after I added
-fno-strict-aliasing):
/usr/share/apr-1.0/build/libtool --no-silent --mode=compile
x86_64-linux-gnu-gcc -pthread -pipe -fno-strict-aliasing -g -O2
-fdebug-prefix-map=/home/jacob/temp/apache/build/apache2-2.4.32=.
-fstack-protector-strong -Wformat -Werror=format-security -DLINUX
-D_REENTRANT -D_GNU_SOURCE -DPLATFORM='"Debian"'
-DBUILD_DATETIME='"2018-03-15T23:26:04"' -fno-strict-aliasing -Wdate-time
-D_FORTIFY_SOURCE=2 -I.
-I/home/jacob/temp/apache/build/apache2-2.4.32/os/unix
-I/home/jacob/temp/apache/build/apache2-2.4.32/include -I/usr/include/apr-1.0
-I/usr/include -I/home/jacob/temp/apache/build/apache2-2.4.32/modules/aaa
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/cache
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/core
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/database
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/filters
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/ldap
-I/home/jacob/temp/apache/build/apache2-2.4.32/server
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/loggers
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/lua
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/proxy
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/session
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/ssl
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/test
-I/home/jacob/temp/apache/build/apache2-2.4.32/server
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/md
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/arch/unix
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/dav/main
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/generators
-I/home/jacob/temp/apache/build/apache2-2.4.32/modules/mappers -fPIE
-prefer-non-pic -static -c config.c && touch config.lo
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #14 from Jacob Lundberg <ja...@collegenet.com> ---
Created attachment 35816
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35816&action=edit
A pair of stuck processes in 2.4.33
I built and deployed 2.4.33 and after some time we've got another pair of stuck
processes. Stack traces are in the attachment.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #2 from Eric Covener <co...@gmail.com> ---
Also backtraces of the high cpu processes.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #3 from Bernhard Pfeiffer <be...@brz.gv.at> ---
httpd-mpm:
<IfModule mpm_worker_module>
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
MaxRequestWorkers 400
MaxConnectionsPerChild 0
</IfModule>
httpd-default:
Timeout 10
KeepAlive On
KeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
LimitRequestFieldSize 16380
<IfModule mod_reqtimeout.c>
RequestReadTimeout header=20-40,MinRate=500 body=20-40,MinRate=500
</IfModule>
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #6 from Bernhard Pfeiffer <be...@brz.gv.at> ---
After 20 sec it start:
top - 14:25:42 up 37 days, 7:45, 3 users, load average: 4.79, 2.43, 2.05
Tasks: 334 total, 1 running, 333 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8%us, 71.8%sy, 0.0%ni, 21.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 64521M total, 50364M used, 14156M free, 1741M buffers
Swap: 32767M total, 38M used, 32729M free, 37608M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20574 nobody 20 0 472m 31m 3952 S 495 0.0 1:29.03 httpd
20660 nobody 20 0 472m 31m 3912 S 492 0.0 1:28.48 httpd
20576 nobody 20 0 472m 30m 3592 S 392 0.0 0:46.92 httpd
20773 nobody 20 0 472m 30m 3896 S 298 0.0 0:17.65 httpd
20696 nobody 20 0 408m 30m 3948 S 198 0.0 0:19.84 httpd
20774 nobody 20 0 472m 30m 3872 S 109 0.0 0:14.11 httpd
20728 nobody 20 0 408m 29m 3572 S 108 0.0 0:13.18 httpd
20853 nobody 20 0 471m 28m 3536 S 107 0.0 0:02.99 httpd
20937 nobody 20 0 406m 26m 3324 S 103 0.0 0:01.83 httpd
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #15 from Stefan Eissing <st...@eissing.org> ---
Jacob and Bernd: are you both talking about the same system? Otherwise I'd like
to hear from Bern about the original issue and version 2.4.33.
If yes: I see not how the process is stuck in these strack traces. It might be
busy and there might be a bug causing this, but see no evidence of deadlocks or
infinite loops.
For example: the latest stacktraces show 2 threads:
1. a HTTP/1.1 request using mod_proxy
2. a HTTP/2 main connection that does a non-blocking read on a stream response
Both manipulate list data structure and - unless memory is corrupted here -
they cannot go into an infinite loop there. These locations might be visible in
strack traces often, since both locations are I/O related and would be
expected to show up on random traces.
I have no good idea how to move forward on this, I must say.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #10 from Jacob Lundberg <ja...@collegenet.com> ---
Created attachment 35773
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35773&action=edit
Three examples of stuck processes
We have been seeing the same thing on 2.4.29 with HTTP/2. We built 2.4.32
(from https://dist.apache.org/repos/dist/dev/httpd/) and we are seeing the same
result as 2.4.29. I am attaching stack traces from three threads which are
each consuming 100% of one CPU core. The stack traces do not appear to change
over time.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #13 from Jacob Lundberg <ja...@collegenet.com> ---
Created attachment 35779
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35779&action=edit
Four more examples of stuck processes
It looks like there is no change with -fno-strict-aliasing; I am attaching some
more stack traces in case they are useful.
I forgot to mention a few things I had in IRC; namely, the problem only
reproduces with HTTP/2 active, and the stuck processes almost always seem to
appear in pairs (load average goes up in steps of 2 and two more threads are
stuck).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #11 from Yann Ylavic <yl...@gmail.com> ---
How do you build httpd (which cflags)?
For instance, do you use "-fno-strict-aliasing" (if no, does it help)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
--- Comment #9 from Bernhard Pfeiffer <be...@brz.gv.at> ---
Thanks for the information. We will wait for version 2.4.30.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
Stefan Eissing <st...@eissing.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #8 from Stefan Eissing <st...@eissing.org> ---
There was a fix in trunk (r1816619) that eliminated unfair scheduling of http2
requests. That mistake was introduced in 2.4.29 and could also impact cpu load
due to repeated attempts to - unsuccessfully - schedule h2 requests.
http://svn.apache.org/viewvc?view=revision&revision=1816619
is the relevant change. If you want, you may also build from
https://github.com/icing/mod_h2/releases/tag/v1.10.16 which includes this fix
and others.
All this will also be in the upcoming 2.4.30 release.
Hope this addresses the issue for you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61695] Load Avarage problem with 2.4.29
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61695
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #1 from Yann Ylavic <yl...@gmail.com> ---
We'll need more informations to determine what's happening, like configured
MPM, http2 (or not), other modules, minimal configuration...
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org