You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Bill Unruh <un...@physics.ubc.ca> on 2012/05/22 19:02:03 UTC
[users@httpd] Denial of Service due to multiplication of httpd running
Madriva 2010.2 running httpd apache 2.2.22
I am having trouble with httpd requests staying active and multiplying.
I just came off having 160 versions of httpd running and completely slowing
down the system. I upgraded to 2.2.22 and it still happens (it went from the
normal 10 servers running to 15 in about a 1/2 hour.) According to the start
times, these seem to be associated with totally bizarre requests from google
(forged addresses?)
Eg, here is one entry from the ps auxww list
apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
At that time in the
access_log I have a whole bunch of entries like
::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
In the past I have also had connections like
66.249.68.198 - - [22/May/2012:09:35:25 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11 HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
associated with the times of the startup of those persistant connections. This
looks to be a totally bizzare GET. since that address certainly has nothing to
do with my site.
In the error log around that time I get nothing that looks suspicious
[Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist: /usr/local/http/htdocs/robots.txt
[Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist: /usr/local/http/htdocs/favicon.ico
[Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist: /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer: http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
On Thu, 24 May 2012, Pete Houston wrote:
> A dozen or so idle processes is perfectly normal for prefork (which
> you are clearly running, BTW). Only worry about this if there are a
> consistently high number of idle processes (say 30 or more for a lightly
> loaded server) in which case you can tune the value of MaxSpareServers
> to suit.
Thanks. As I said that problem raised its head because there were 160 of them
and the system was swapping itself to death. But that was on an earlier
version (2.2.0 I think it was). I will look at MaxSpareServers.
>
> Have a read about the prefork MPM in the documentation:
> http://httpd.apache.org/docs/2.2/mod/prefork.html
>
> If the number of idle processes is consistently higher than
> MaxSpareServers you have a bug.
OK, my MaxSpareServers is 20 and I had 21 (mind you that is counting the
master, so maybe it was actually 20) yesterday all day.
>
> HTH,
>
> Pete
>
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Pete Houston <ph...@openstrike.co.uk>.
A dozen or so idle processes is perfectly normal for prefork (which
you are clearly running, BTW). Only worry about this if there are a
consistently high number of idle processes (say 30 or more for a lightly
loaded server) in which case you can tune the value of MaxSpareServers
to suit.
Have a read about the prefork MPM in the documentation:
http://httpd.apache.org/docs/2.2/mod/prefork.html
If the number of idle processes is consistently higher than
MaxSpareServers you have a bug.
HTH,
Pete
--
Openstrike - improving business through open source
http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
I was too quick to cry ignorance. I have now gotten mod_status working and
here is the output. For some bizarre reason the number of servers is
multiplying
68.58.140.236 was downloading another page with lots of pictures on it.
(http://axion.physics.ubc.ca/titanic/-- axion is an ancient pseudonym for
www.theory.physics.ubc.ca)
Why are these idle servers multiplying? As I said this all started out by my
having 160 of them which was causing swapping death for the system.
How do I tell apache not to do this multiplication of servers?
Note that the aggregator requests keep coming even though the virtual host
which hosted them is disconnected (commented out in the httpd configuratio
nfiles)
-------------------------------------------------------------
Apache Server Status for www.theory.physics.ubc.ca
Server Version: Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2)
mod_ssl/2.2.22 OpenSSL/1.0.0a PHP/5.3.13 with Suhosin-Patch mod_perl/2.0.4
Perl/v5.10.1
Server Built: Feb 1 2012 12:26:04
Current Time: Thursday, 24-May-2012 09:18:07 PDT
Restart Time: Thursday, 24-May-2012 08:55:18 PDT
Parent Server Generation: 0
Server uptime: 22 minutes 48 seconds
Total accesses: 172 - Total Traffic: 3.9 MB
CPU Usage: u.01 s.02 cu0 cs0 - .00219% CPU load
.126 requests/sec - 2976 B/second - 23.1 kB/request
1 requests currently being processed, 14 idle workers
______________W.................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 11536 0/38/38 _ 0.01 117 1 0.0 0.55 0.55 68.58.140.236 www.theory.physics.ubc.ca NULL
1-0 11537 0/13/13 _ 0.00 117 332 0.0 0.51 0.51 68.58.140.236 www.theory.physics.ubc.ca NULL
2-0 11538 0/11/11 _ 0.00 117 5 0.0 0.47 0.47 68.58.140.236 www.theory.physics.ubc.ca NULL
3-0 11539 0/14/14 _ 0.00 117 1 0.0 0.61 0.61 68.58.140.236 www.theory.physics.ubc.ca NULL
4-0 11540 0/19/19 _ 0.00 117 1 0.0 0.67 0.67 68.58.140.236 www.theory.physics.ubc.ca NULL
5-0 11541 0/17/17 _ 0.00 117 0 0.0 0.30 0.30 66.249.68.41 www.theory.physics.ubc.ca GET /aggregator/www.guardian.co.uk/world/2012/jan/17/node/node/
6-0 11542 0/6/6 _ 0.00 1159 1 0.0 0.01 0.01 24.82.146.42 www.theory.physics.ubc.ca NULL
7-0 11544 0/49/49 _ 0.02 117 1 0.0 0.77 0.77 68.58.140.236 www.theory.physics.ubc.ca NULL
8-0 11737 0/4/4 _ 0.00 3 0 0.0 0.00 0.00 66.249.68.193 www.theory.physics.ubc.ca GET /aggregator/www.youtube.com/www.iaea.org/Publications/Docum
14-0 11743 0/1/1 W 0.00 0 0 0.0 0.00 0.00 24.82.146.42 www.theory.physics.ubc.ca GET /server-status HTTP/1.1
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
CPU CPU usage, number of seconds
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
On Thu, 24 May 2012, Pete Houston wrote:
> On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
>> But this is clearly a horrible kludge. Is there any way I can figure
>> out what is triggering these versions of httpd
>> to be piling up?
>
> Since you've looked at the logs and not found anything there (or at
> least eliminated what problems you have found), mod_status would be
> a good next step.
>
> If there is some reason why you cannot enable mod_status, then check who
> is holding the connections open with netstat.
>
> Pete
>
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
On Thu, 24 May 2012, Pete Houston wrote:
> On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
>> But this is clearly a horrible kludge. Is there any way I can figure
>> out what is triggering these versions of httpd
>> to be piling up?
>
> Since you've looked at the logs and not found anything there (or at
> least eliminated what problems you have found), mod_status would be
> a good next step.
>
> If there is some reason why you cannot enable mod_status, then check who
> is holding the connections open with netstat.
Unfortunately I am a bit out of my depth here. How do I enable mod_status and
then use it? And what exactly do I look for in netstat. I see nothing in
netstat -pa just shows one httpd listaning on port www and one on port https
which I presume is the master daemon listening for connections. (what do the
other 8 daemons which are started up do?)
tcp 0 0 *:www *:* LISTEN 6105/httpd
tcp 0 0 *:https *:* LISTEN 6105/httpd
It also shows
tcp 0 0 boson:www mail5.alphameric.com:31040 FIN_WAIT2 -
Why would it be waiting for a shutdown from the remote end?
It does seem to be stuck. (ie, I do netstat -pa ten minutes apart and that
line is still there)
This address occurs in the logs as downloading my "thirdman" page
(www.theory.physics.ubc.ca/thirdman/thirdman.html) which has a huge load of
pictures on it.
>
> Pete
>
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Pete Houston <ph...@openstrike.co.uk>.
On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
> But this is clearly a horrible kludge. Is there any way I can figure
> out what is triggering these versions of httpd
> to be piling up?
Since you've looked at the logs and not found anything there (or at
least eliminated what problems you have found), mod_status would be
a good next step.
If there is some reason why you cannot enable mod_status, then check who
is holding the connections open with netstat.
Pete
--
Openstrike - improving business through open source
http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
On Tue, 22 May 2012, Bill Unruh wrote:
>
> Madriva 2010.2 running httpd apache 2.2.22
>
> I am having trouble with httpd requests staying active and multiplying. I
> just came off having 160 versions of httpd running and completely slowing
> down the system. I upgraded to 2.2.22 and it still happens (it went from the
> normal 10 servers running to 15 in about a 1/2 hour.) According to the start
> times, these seem to be associated with totally bizarre requests from google
> (forged addresses?)
>
> Eg, here is one entry from the ps auxww list
>
> apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00
> /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL
> -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS -DHAVE_ASIS -DHAVE_AUTH_BASIC
> -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON -DHAVE_AUTHN_DBM
> -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM -DHAVE_AUTHZ_DEFAULT
> -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER -DHAVE_AUTHZ_USER
> -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER -DHAVE_CASE_FILTER_IN
> -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE -DHAVE_DIR
> -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
> -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP
> -DHAVE_INCLUDE -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO
> -DHAVE_MIME -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT
> -DHAVE_OPTIONAL_FN_IMPORT -DHAVE_OPTIONAL_HOOK_EXPORT
> -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF -DHAVE_SPELING
> -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
> -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
>
> At that time in the access_log I have a whole bunch of entries like
> : : 1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
> : : "Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy
> : : connection)"
>
>
> In the past I have also had connections like 66.249.68.198 - -
> [22/May/2012:09:35:25 -0700] "GET
> /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
> HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
> +http://www.google.com/bot.html)"
>
> associated with the times of the startup of those persistant connections.
> This
> looks to be a totally bizzare GET. since that address certainly has nothing
> to
> do with my site.
>
> In the error log around that time I get nothing that looks suspicious
>
> [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not
> exist: /usr/local/http/htdocs/robots.txt
> [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
> /usr/local/http/htdocs/favicon.ico
> [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not
> exist: /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
> http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
>
OK, I have closed down that virtual host I had set up ( which was the source
of those aggregator web page requests), but I am still getting the same
problems. It starts out with 9 copies of httpd daemon running. After a few
hours it is up to 15 or 20. I have no idea what is causing this.
I have now put in a cron job which checks every 10 min and if it finds more
than 24 instances of httpd running, it restarts httpd (service httpd restart)
But this is clearly a horrible kludge.
Is there any way I can figure out what is triggering these versions of httpd
to be piling up?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
Aha-- under "www.emergentgravity.org" I do have an aggregator apparently.
(Just tried it and I do get a whole bunch of news items.) so this must be what
is giving those strange "GET" entries.
(This site was set up by someone else and thus my knowledge of what it does is
pretty sketchy.)
On Tue, 22 May 2012, Bill Unruh wrote:
> On Tue, 22 May 2012, William A. Rowe Jr. wrote:
>
>> On 5/22/2012 6:00 PM, Bill Unruh wrote:
>> > On Tue, 22 May 2012, William A. Rowe Jr. wrote:
>> >
>> > > On 5/22/2012 12:02 PM, Bill Unruh wrote:
>> > >
>> > > > At that time in the access_log I have a whole bunch of entries like
>> > > > : : 1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
>> > > > : : "Apache/2.2.22
>> > > > (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
>> > >
>> > > That's your own local loopback from a process running on this same
>> > > box.
>> >
>> > There are no processes running on this same box. It is rarely used. and
>> > certainly did not have a browser running at that time.
>>
>> Then a server module is likely pinging itself. Any chance you set up an
>> infinite proxy
>> recursion here?
>>
>> > > > In the past I have also had connections like 66.249.68.198 - -
>> > > > [22/May/2012:09:35:25
>> > > > -0700] "GET
>> > > > /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
>> > > >
>> > > > HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
>> > > > +http://www.google.com/bot.html)"
>> > >
>> > > No clue. Maybe playing with open proxies? The document seems to be
>> > > 58k if that helps you
>> > > at all (maybe a local index page?)
>> >
>> > There is no such file or path on my system. If I try to use it, I get
>> > file not
>> > found. I have nothing called /aggregator/
>>
>> Looking more and more like a proxy recursion/infinite looping sort of
>> config error.
>
> OK, I do have a "virtual host", www.emergentgravity.org defined on this
> system.
>
> In /etc/httpd/conf/vhosts.d/00_default_vhosts.conf I have
> ---------------------------------------- NameVirtualHost *:80
> <VirtualHost *:80>
> ServerName www.theory.physics.ubc.ca
> ServerAlias theory.physics.ubc.ca
> DocumentRoot /usr/local/http/htdocs
> </VirtualHost>
>
> <VirtualHost *:80>
> ServerName www.emergentgravity.org
> ServerAlias emergentgravity.org
> DocumentRoot /usr/local/http/htdocs/emergent/drupal.new
> </VirtualHost>
> -------------------------------------------
>
> This is the whole of that file.
> This is supposed to be so that a www.emergentgravity.org address goes to its
> drupal document root, and such that anything coming in to
> www.theory.physics.ubc.ca comes in to /usr/local/http/htdocs.
>
> If you suspect I have no idea what I am doing in setting this up, you are
> correct. Have I managed to stick in a feedback loop in here?
>
>
>>
>> > > Can you interrupt one of the truly hosed processes using gdb and try
>> > > 'thread apply all bt'
>> >
>> > What does that do?
>>
>> Dumps all threads instead of just one of them.
>>
>> > Thread 1 (Thread 0xb760f700 (LWP 20861)):
>> > #0 0xffffe424 in __kernel_vsyscall ()
>> > #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
>> > #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
>> > #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
>> > #4 0x0809294c in ?? ()
>> > #5 0x08092e0b in ?? ()
>> > #6 0x08093be4 in ap_mpm_run ()
>> > #7 0x08064cd1 in main ()
>>
>> It might be helpful to first install the debuginfo for the apr/httpd
>> packages, but this
>
> Unfortunately Mandriva never bothered to supply the apache-debug package for
> this version of apache.
>
>
>> looks like it might be in the accept queue waiting on the MutexFile to
>> unblock this one
>> (and is probably a healthy process).
>>
>> If you are running prefork we would encourage you to try the worker mpm.
>
> Sorry, I have no idea what this sentence means. (prefork, worker mpm)
>
>
>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
>
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
On Tue, 22 May 2012, William A. Rowe Jr. wrote:
> On 5/22/2012 6:00 PM, Bill Unruh wrote:
>> On Tue, 22 May 2012, William A. Rowe Jr. wrote:
>>
>>> On 5/22/2012 12:02 PM, Bill Unruh wrote:
>>>
>>>> At that time in the access_log I have a whole bunch of entries like
>>>> ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
>>>> (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
>>>
>>> That's your own local loopback from a process running on this same box.
>>
>> There are no processes running on this same box. It is rarely used. and
>> certainly did not have a browser running at that time.
>
> Then a server module is likely pinging itself. Any chance you set up an infinite proxy
> recursion here?
>
>>>> In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
>>>> -0700] "GET
>>>> /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
>>>>
>>>> HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
>>>> +http://www.google.com/bot.html)"
>>>
>>> No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
>>> at all (maybe a local index page?)
>>
>> There is no such file or path on my system. If I try to use it, I get file not
>> found. I have nothing called /aggregator/
>
> Looking more and more like a proxy recursion/infinite looping sort of config error.
OK, I do have a "virtual host", www.emergentgravity.org defined on this
system.
In /etc/httpd/conf/vhosts.d/00_default_vhosts.conf I have
----------------------------------------
NameVirtualHost *:80
<VirtualHost *:80>
ServerName www.theory.physics.ubc.ca
ServerAlias theory.physics.ubc.ca
DocumentRoot /usr/local/http/htdocs
</VirtualHost>
<VirtualHost *:80>
ServerName www.emergentgravity.org
ServerAlias emergentgravity.org
DocumentRoot /usr/local/http/htdocs/emergent/drupal.new
</VirtualHost>
-------------------------------------------
This is the whole of that file.
This is supposed to be so that a www.emergentgravity.org address goes to its
drupal document root, and such that anything coming in to
www.theory.physics.ubc.ca comes in to /usr/local/http/htdocs.
If you suspect I have no idea what I am doing in setting this up, you are
correct. Have I managed to stick in a feedback loop in here?
>
>>> Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
>>
>> What does that do?
>
> Dumps all threads instead of just one of them.
>
>> Thread 1 (Thread 0xb760f700 (LWP 20861)):
>> #0 0xffffe424 in __kernel_vsyscall ()
>> #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
>> #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
>> #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
>> #4 0x0809294c in ?? ()
>> #5 0x08092e0b in ?? ()
>> #6 0x08093be4 in ap_mpm_run ()
>> #7 0x08064cd1 in main ()
>
> It might be helpful to first install the debuginfo for the apr/httpd packages, but this
Unfortunately Mandriva never bothered to supply the apache-debug package for
this version of apache.
> looks like it might be in the accept queue waiting on the MutexFile to unblock this one
> (and is probably a healthy process).
>
> If you are running prefork we would encourage you to try the worker mpm.
Sorry, I have no idea what this sentence means. (prefork, worker mpm)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 5/22/2012 6:00 PM, Bill Unruh wrote:
> On Tue, 22 May 2012, William A. Rowe Jr. wrote:
>
>> On 5/22/2012 12:02 PM, Bill Unruh wrote:
>>
>>> At that time in the access_log I have a whole bunch of entries like
>>> ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
>>> (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
>>
>> That's your own local loopback from a process running on this same box.
>
> There are no processes running on this same box. It is rarely used. and
> certainly did not have a browser running at that time.
Then a server module is likely pinging itself. Any chance you set up an infinite proxy
recursion here?
>>> In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
>>> -0700] "GET
>>> /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
>>>
>>> HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
>>> +http://www.google.com/bot.html)"
>>
>> No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
>> at all (maybe a local index page?)
>
> There is no such file or path on my system. If I try to use it, I get file not
> found. I have nothing called /aggregator/
Looking more and more like a proxy recursion/infinite looping sort of config error.
>> Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
>
> What does that do?
Dumps all threads instead of just one of them.
> Thread 1 (Thread 0xb760f700 (LWP 20861)):
> #0 0xffffe424 in __kernel_vsyscall ()
> #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
> #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
> #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
> #4 0x0809294c in ?? ()
> #5 0x08092e0b in ?? ()
> #6 0x08093be4 in ap_mpm_run ()
> #7 0x08064cd1 in main ()
It might be helpful to first install the debuginfo for the apr/httpd packages, but this
looks like it might be in the accept queue waiting on the MutexFile to unblock this one
(and is probably a healthy process).
If you are running prefork we would encourage you to try the worker mpm.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by Bill Unruh <un...@physics.ubc.ca>.
On Tue, 22 May 2012, William A. Rowe Jr. wrote:
> On 5/22/2012 12:02 PM, Bill Unruh wrote:
>>
>> Eg, here is one entry from the ps auxww list
>>
>> apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f
>> /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS
>> -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON
>> -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM
>> -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER
>> -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER
>> -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE
>> -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
>> -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE
>> -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME
>> -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT
>> -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF
>> -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
>> -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
>
> Never seen such a crazy startup line, I presume all of your modules have individual
> <IfDefine > blocks around each?
It is basically what Mandriva has as its default.
>
>> At that time in the access_log I have a whole bunch of entries like
>> ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
>> (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
>
> That's your own local loopback from a process running on this same box.
There are no processes running on this same box. It is rarely used. and
certainly did not have a browser running at that time.
\
>
>> In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
>> -0700] "GET
>> /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
>> HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
>> +http://www.google.com/bot.html)"
>
> No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
> at all (maybe a local index page?)
There is no such file or path on my system. If I try to use it, I get file not
found. I have nothing called /aggregator/
>
>> In the error log around that time I get nothing that looks suspicious
>>
>> [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist:
>> /usr/local/http/htdocs/robots.txt
>> [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
>> /usr/local/http/htdocs/favicon.ico
>
> Certainly harmless and most common from crawlers and web browsers respectively. You may
> want to add a simple one line robots.txt file, and a simple default icon.
That was what I thought.
>
>
>> [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist:
>> /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
>> http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
>
> Your own mistake in the html, it seems.
Yup. Just displying it on the off chance it could be problematic. Since I have
not the ghost of an idea what could be wrong, I also have not the ghost of an
idea what could be a symptom either.
>
> Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
>
What does that do?
Thread 1 (Thread 0xb760f700 (LWP 20861)):
#0 0xffffe424 in __kernel_vsyscall ()
#1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
#2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
#3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
#4 0x0809294c in ?? ()
#5 0x08092e0b in ?? ()
#6 0x08093be4 in ap_mpm_run ()
#7 0x08064cd1 in main ()
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Denial of Service due to multiplication of httpd
running
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 5/22/2012 12:02 PM, Bill Unruh wrote:
>
> Eg, here is one entry from the ps auxww list
>
> apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f
> /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS
> -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON
> -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM
> -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER
> -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER
> -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE
> -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
> -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE
> -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME
> -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT
> -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF
> -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
> -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
Never seen such a crazy startup line, I presume all of your modules have individual
<IfDefine > blocks around each?
> At that time in the access_log I have a whole bunch of entries like
> ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
> (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
That's your own local loopback from a process running on this same box.
> In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
> -0700] "GET
> /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
> HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
> +http://www.google.com/bot.html)"
No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
at all (maybe a local index page?)
> In the error log around that time I get nothing that looks suspicious
>
> [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist:
> /usr/local/http/htdocs/robots.txt
> [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
> /usr/local/http/htdocs/favicon.ico
Certainly harmless and most common from crawlers and web browsers respectively. You may
want to add a simple one line robots.txt file, and a simple default icon.
> [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist:
> /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
> http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
Your own mistake in the html, it seems.
Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org