You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Douglas Duckworth <do...@med.cornell.edu> on 2017/11/10 17:41:40 UTC
[users@httpd] Apache 2.4 DoS?
Hi
I am running old PHP under Apache httpd-2.4.
During a typical day:
Server load: 0.03 0.03 0.05
Total accesses: 16028 - Total Traffic: 1.4 GB
CPU Usage: u20.92 s1.24 cu.01 cs.23 - .00163% CPU load
.0116 requests/sec - 1104 B/second - 92.7 kB/request
2 requests currently being processed, 8 idle workers
Though, ever few weeks, we see sudden increase in workers who never seem to
retire:
[Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584] AH00161:
server reached MaxRequestWorkers setting, consider raising the
MaxRequestWorkers setting
user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
257
It's my belief that this occurs due to malicious activity involving our old
PHP sites, given this version has multiple known denial of service
vulnerabilities, however the only thing I see in logs, during the time when
workers were spawned, are light spider and bot activity.
We are running mod_security, mod_evasive, and mod_reqtimeout.
apachectl -t -D DUMP_MODULES | grep -e timeout -e security -e evasive
reqtimeout_module (shared)
security2_module (shared)
evasive20_module (shared)
httpd.conf:
MaxKeepAliveRequests 50
KeepAlive On
Timeout 30
KeepAliveTimeout 10
<IfModule reqtimeout_module>
RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 2
MaxSpareServers 10
MaxRequestWorkers 128
MaxRequestsPerChild 50
MaxRequestWorkers 100
</IfModule>
modsecurity.conf:
SecRuleEngine on
mod_evasive.conf:
DOSPageCount 50
DOSSiteCount 100
DOSPageInterval 1
DOSSiteInterval 1
php.ini:
max_execution_time = 10
max_input_time = 10
memory_limit = 32M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
log_errors = On
I set MaxRequestWorkers to 100 though it seems that threshold was passed
meanwhile the server's no longer serving data, as the failover's now
active, but these httpd workers *refuse to die*!
If my VirtualHosts were under DoS, in a manner that exploits PHP, then
would I even be able to detect them in the logs?
Based upon my limited experience, I should be protected against both "slow"
and "fast" DoS though of course not DDoS. Greatly appreciate the insight
and assistance. We plan on replacing our old PHP sites but until then I
want to do what I can to ensure this stops happening other than bringing up
the failover.
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: doug@med.cornell.edu
O: 212-746-6305 <(212)%20746-6305>
F: 212-746-8690 <(212)%20746-8690>
Re: [users@httpd] Apache 2.4 DoS?
Posted by Ruben Safir <mr...@panix.com>.
On 11/10/2017 12:41 PM, Douglas Duckworth wrote:
> Hi
>
> I am running old PHP under Apache httpd-2.4.
>
> During a typical day:
>
> Server load: 0.03 0.03 0.05
> Total accesses: 16028 - Total Traffic: 1.4 GB
> CPU Usage: u20.92 s1.24 cu.01 cs.23 - .00163% CPU load
> .0116 requests/sec - 1104 B/second - 92.7 kB/request
> 2 requests currently being processed, 8 idle workers
>
> Though, ever few weeks, we see sudden increase in workers who never seem to
> retire:
>
> [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584] AH00161:
> server reached MaxRequestWorkers setting, consider raising the
> MaxRequestWorkers setting
>
> user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> 257
>
> It's my belief that this occurs due to malicious activity involving our old
> PHP sites, given this version has multiple known denial of service
> vulnerabilities, however the only thing I see in logs, during the time when
> workers were spawned, are light spider and bot activity.
>
> We are running mod_security, mod_evasive, and mod_reqtimeout.
>
> apachectl -t -D DUMP_MODULES | grep -e timeout -e security -e evasive
>
> reqtimeout_module (shared)
> security2_module (shared)
> evasive20_module (shared)
>
> httpd.conf:
>
> MaxKeepAliveRequests 50
> KeepAlive On
> Timeout 30
> KeepAliveTimeout 10
>
> <IfModule reqtimeout_module>
> RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
> </IfModule>
>
> <IfModule mpm_prefork_module>
> StartServers 5
> MinSpareServers 2
> MaxSpareServers 10
> MaxRequestWorkers 128
> MaxRequestsPerChild 50
> MaxRequestWorkers 100
> </IfModule>
>
> modsecurity.conf:
>
> SecRuleEngine on
>
> mod_evasive.conf:
>
> DOSPageCount 50
> DOSSiteCount 100
> DOSPageInterval 1
> DOSSiteInterval 1
>
> php.ini:
>
> max_execution_time = 10
> max_input_time = 10
> memory_limit = 32M
> error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
> log_errors = On
>
> I set MaxRequestWorkers to 100 though it seems that threshold was passed
> meanwhile the server's no longer serving data, as the failover's now
> active, but these httpd workers *refuse to die*!
>
> If my VirtualHosts were under DoS, in a manner that exploits PHP, then
> would I even be able to detect them in the logs?
>
> Based upon my limited experience, I should be protected against both "slow"
> and "fast" DoS though of course not DDoS. Greatly appreciate the insight
> and assistance. We plan on replacing our old PHP sites but until then I
> want to do what I can to ensure this stops happening other than bringing up
> the failover.
>
> Thanks,
>
> Douglas Duckworth, MSc, LFCS
> HPC System Administrator
> Scientific Computing Unit
> Physiology and Biophysics
> Weill Cornell Medicine
> E: doug@med.cornell.edu
> O: 212-746-6305 <(212)%20746-6305>
> F: 212-746-8690 <(212)%20746-8690>
>
Did you try the PHP mailing list?
That version of PHP is like Swiss cheese and you are not going to be
able to avoid a complete rebuild of the server which should be chrooted,
or in a readonly file system, and analysis is of minimal value, if not
counter productive.
Ruben Safir. MS Comp Sci (Computational Evolutionary Biology), BS Pharm RPh
NYLXS Chief Technologist
Free Software Education and Advocacy since 1997
E: ruben@mrbrklyn.com
O: 718-715-1771
--
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive
http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache 2.4 DoS?
Posted by Douglas Duckworth <do...@med.cornell.edu>.
Thank you Rainer Canavan!
Will try using gdb to get more information!
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: doug@med.cornell.edu
O: 212-746-6305
F: 212-746-8690
On Mon, Nov 13, 2017 at 7:04 AM, Rainer Canavan <rainer.canavan@sevenval.com
> wrote:
> On Fri, Nov 10, 2017 at 6:41 PM, Douglas Duckworth
> <do...@med.cornell.edu> wrote:
> > Hi
> >
> > I am running old PHP under Apache httpd-2.4.
>
> [...]
> > Though, ever few weeks, we see sudden increase in workers who never seem
> to
> > retire:
> >
> > [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584]
> AH00161:
> > server reached MaxRequestWorkers setting, consider raising the
> > MaxRequestWorkers setting
> >
> > user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> > 257
>
> If the php locks up while processing your request, no logs will be
> written. You may be running
> into a bug where circular, unresolvable dependencies for a lock
> prevent the processes from
> completing their requests. To check what's going on, install gdb, the
> debug info for your php
> and httpd and find the .gdbinfo that came with the httpd and php
> version you're using. Then
> attach gdb to any of the hanging processes (gdb `which httpd` PID),
> source both .gdbinit files,
> do a "zbacktrace" and a "bt full", and repeat for some other hanging
> processes. Depending
> on the type of lock, you may be able to identify the first process
> that has acquired that lock
> that all others are waiting for, and the php code and / or php module
> that causes it.
>
> rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] Apache 2.4 DoS?
Posted by Douglas Duckworth <do...@med.cornell.edu>.
Hi Ranier Caravan,
Looks like this problem didn't go away.
To get symbols I had to first:
sudo debuginfo-install httpd
Then
sudo gdb -p PID
Several hundred apache forks are present so I selected random one....
*(gdb) where*
#0 0x00007f247f302f0d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f247fa1e94b in poll (__timeout=30000, __nfds=1,
__fds=0x7ffd5cd92b00) at /usr/include/bits/poll2.h:46
#2 apr_wait_for_io_or_timeout (f=f@entry=0x0, s=s@entry=0x55a9964b7610,
for_read=for_read@entry=1) at support/unix/waitio.c:51
#3 0x00007f247fa1877a in apr_socket_recv (sock=sock@entry=0x55a9964b7610,
buf=buf@entry=0x55a996541b08
"2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\ny:
Express\r\nCache-Control: no-store, no-cache, no-transform,
must-revalidate, max-age=0\r\nAccess-Control-Allow-Origin: *\r\nVary:
Origin\r\nContent-Type: application/javascript;"...,
len=len@entry=0x7ffd5cd92be0)
at network_io/unix/sendrecv.c:87
#4 0x00007f2480457281 in socket_bucket_read (a=0x55a99653dc48,
str=0x7ffd5cd92bd8, len=0x7ffd5cd92be0, block=<optimized out>) at
buckets/apr_buckets_socket.c:36
#5 0x00007f2480455756 in apr_brigade_split_line
(bbOut=bbOut@entry=0x55a9964d8ed8,
bbIn=0x55a9964b7f28, block=block@entry=APR_BLOCK_READ,
maxbytes=maxbytes@entry=8192) at buckets/apr_brigade.c:319
#6 0x000055a993422d5c in ap_core_input_filter (f=0x55a9964b7e28,
b=0x55a9964d8ed8, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at core_filters.c:147
#7 0x00007f247a134c4e in logio_in_filter (f=<optimized out>,
bb=0x55a9964d8ed8, mode=<optimized out>, block=<optimized out>,
readbytes=<optimized out>) at mod_logio.c:140
#8 0x000055a99343e2bd in ap_http_filter (f=<optimized out>, b=<optimized
out>, mode=<optimized out>, block=<optimized out>, readbytes=8192) at
http_filters.c:515
#9 0x00007f2474eed058 in ap_proxy_http_process_response
(p=p@entry=0x55a9965018e8,
r=r@entry=0x55a996501960, backend_ptr=backend_ptr@entry=0x7ffd5cd97058,
server_portstr=server_portstr@entry=0x7ffd5cd97100 "", conf=0x55a994beaef0,
conf=0x55a994beaef0, worker=<optimized out>) at mod_proxy_http.c:1708
*#10 0x00007f2474eef7f9 in proxy_http_handler (r=<optimized out>,
worker=<optimized out>, conf=0x55a994beaef0, url=0x55a9964d819e
"http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
<http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming>",
*
* proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038*
*#11 0x00007f247673edc4 in proxy_run_scheme_handler
(r=r@entry=0x55a996501960, worker=0x55a994beb1b0,
conf=conf@entry=0x55a994beaef0, *
* url=0x55a9964d819e
"http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
<http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming>",
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at
mod_proxy.c:2746*
#12 0x00007f247673fcad in proxy_handler (r=0x55a996501960) at
mod_proxy.c:1125
#13 0x000055a993427990 in ap_run_handler (r=r@entry=0x55a996501960) at
config.c:169
#14 0x000055a993427ed9 in ap_invoke_handler (r=r@entry=0x55a996501960) at
config.c:439
#15 0x000055a99343ca8a in ap_process_async_request (r=r@entry=0x55a996501960)
at http_request.c:328
#16 0x000055a99343cd64 in ap_process_request (r=r@entry=0x55a996501960) at
http_request.c:363
#17 0x000055a993438f62 in ap_process_http_sync_connection
(c=0x55a9964a9090) at http_core.c:190
#18 ap_process_http_connection (c=0x55a9964a9090) at http_core.c:231
#19 0x000055a993430fc0 in ap_run_process_connection (c=c@entry=0x55a9964a9090)
at connection.c:41
#20 0x000055a9934313a8 in ap_process_connection (c=c@entry=0x55a9964a9090,
csd=<optimized out>) at connection.c:202
#21 0x00007f24769547af in child_main (child_num_arg=child_num_arg@entry=6)
at prefork.c:707
#22 0x00007f24769549f5 in make_child (s=0x55a994af9e30, slot=6) at
prefork.c:810
#23 0x00007f247695568e in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:912
#24 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized
out>) at prefork.c:1100
#25 0x000055a99340bffe in ap_run_mpm (pconf=pconf@entry=0x55a994ab5138,
plog=0x55a994ae2358, s=0x55a994af9e30) at mpm_common.c:96
#26 0x000055a993404d76 in main (argc=2, argv=0x7ffd5cd97878) at main.c:783
*bt full for 10 and 11:*
#10 0x00007f2474eef7f9 in proxy_http_handler (r=<optimized out>,
worker=<optimized out>, conf=0x55a994beaef0, url=0x55a9964d819e "
http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
",
proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038
locurl = 0x55a9964d8490
"/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming"
status = <optimized out>
server_portstr =
"\000vM\226\251U\000\000\350\030P\226\251U\000\000\000q\331\\\375\177\000\000\000\000\000\000\000\000\000"
scheme = <optimized out>
proxy_function = 0x7f2474ef0bfa "HTTP"
u = <optimized out>
backend = 0x55a9964b5600
is_ssl = 0
c = 0x55a9964a9090
retry = 0
p = <optimized out>
uri = 0x55a9964d83b0
#11 0x00007f247673edc4 in proxy_run_scheme_handler (r=r@entry=0x55a996501960,
worker=0x55a994beb1b0, conf=conf@entry=0x55a994beaef0,
url=0x55a9964d819e "
http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming",
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at
mod_proxy.c:2746
pHook = <optimized out>
n = 3
rv = -1
I am beyond my knowledge limit at this point.
The server nike.pbtech contains an R application running on internal
server. It's reached through this Apace proxy.
I have a few questions.
1) How do we tell the apache fork's hung hung?
2) How can I check all forks at once? Attaching to parent does not work.
3) When trying other Apache forks I get hung gdb at line "attaching to
process." Any way to avoid?
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: doug@med.cornell.edu
O: 212-746-6305
F: 212-746-8690
On Mon, Nov 13, 2017 at 7:04 AM, Rainer Canavan <rainer.canavan@sevenval.com
> wrote:
> On Fri, Nov 10, 2017 at 6:41 PM, Douglas Duckworth
> <do...@med.cornell.edu> wrote:
> > Hi
> >
> > I am running old PHP under Apache httpd-2.4.
>
> [...]
> > Though, ever few weeks, we see sudden increase in workers who never seem
> to
> > retire:
> >
> > [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584]
> AH00161:
> > server reached MaxRequestWorkers setting, consider raising the
> > MaxRequestWorkers setting
> >
> > user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> > 257
>
> If the php locks up while processing your request, no logs will be
> written. You may be running
> into a bug where circular, unresolvable dependencies for a lock
> prevent the processes from
> completing their requests. To check what's going on, install gdb, the
> debug info for your php
> and httpd and find the .gdbinfo that came with the httpd and php
> version you're using. Then
> attach gdb to any of the hanging processes (gdb `which httpd` PID),
> source both .gdbinit files,
> do a "zbacktrace" and a "bt full", and repeat for some other hanging
> processes. Depending
> on the type of lock, you may be able to identify the first process
> that has acquired that lock
> that all others are waiting for, and the php code and / or php module
> that causes it.
>
> rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] Apache 2.4 DoS?
Posted by Rainer Canavan <ra...@sevenval.com>.
On Fri, Nov 10, 2017 at 6:41 PM, Douglas Duckworth
<do...@med.cornell.edu> wrote:
> Hi
>
> I am running old PHP under Apache httpd-2.4.
[...]
> Though, ever few weeks, we see sudden increase in workers who never seem to
> retire:
>
> [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584] AH00161:
> server reached MaxRequestWorkers setting, consider raising the
> MaxRequestWorkers setting
>
> user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> 257
If the php locks up while processing your request, no logs will be
written. You may be running
into a bug where circular, unresolvable dependencies for a lock
prevent the processes from
completing their requests. To check what's going on, install gdb, the
debug info for your php
and httpd and find the .gdbinfo that came with the httpd and php
version you're using. Then
attach gdb to any of the hanging processes (gdb `which httpd` PID),
source both .gdbinit files,
do a "zbacktrace" and a "bt full", and repeat for some other hanging
processes. Depending
on the type of lock, you may be able to identify the first process
that has acquired that lock
that all others are waiting for, and the php code and / or php module
that causes it.
rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org