You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Mike -- EMAIL IGNORED <m_...@yahoo.com> on 2009/08/04 17:05:25 UTC

[users@httpd] Re: excessive dns slows httpd

On Tue, 04 Aug 2009 08:22:14 +0200, André Warnier wrote:

[...]
>> 
> You could first check the obvious, such as whether in your Apache/Centos
> config, you are not *asking* the server to do these DNS lookups. Look
> here for instance :
> http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats (I mean
> that if your are asking Apache to log the *name* of the client in the
> access log for instance, then it has to do a lookup) Any time, anywhere,
> where Apache, for whatever reason, has to translate an IP address into a
> name or vice-versa, may result in this behaviour.
> 
[...]

In LogFormat, I changed %h to %a, which hat no effect.  I did "solve"
the problem by completely blocking DNS in iptables.  As it happens,
in my present configuration, this is not a problem, but I would hardly
call it a satisfactory solution.  Remember, the httpd.config works
without problem on a neighboring box (with a different OS).

Mike.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by André Warnier <aw...@ice-sa.com>.
André Warnier wrote:
> Mark H. Wood wrote:
>> Local DNS cache of some sort on the Fedora box?  Some of my hosts run
>> 'nscd' (which comes with glibc) and some run BIND in cache-only mode.
>>
> That's also, I believe, what a local "dnsmasq" would do.  But in order 
> for the local Apache to use it, you'd have to set 127.0.0.1 as a 
> resolver in /etc/resolv.conf, which apparently is not the case here.
> 
> To get back to the basic issue..
> We seem to have, on the same host
> - an "old" Apache configuration, which does not trigger client DNS names 
> lookups
> - a "new" Apache configuration, which does
> Yes ?
> 
> So I guess we can exclude for the time being the possibility that it 
> might be something else than Apache which does the lookups (I was 
> thinking of denyhosts or the like).
> 
> Next obvious question : what's the difference between the old and new 
> configurations ? (sorry if that was explained before, I did not get back 
> to the archives).
> Do I vaguely remember it had something to do with SSL, or was that 
> another thread ?
> 

Add-on : I guess something else that could trigger lookups, are 
Allow/Deny directives, if you have things like "Allow from .domain.com".
Or, let's be fancy : some nice little thingy that displays in shapes and 
colors the accesses by domains, or some sophisticated SSI include which 
somehow does a lookup of the client...
As someone else mentioned, without having a real peek at your full 
configuration, its a bit tedious to make guesses.

If it is about SSL - not my domain at all - then maybe someone else 
could tell us what kind of SSL configuration could trigger a client name 
lookup ? (I would imagine that the use of client certs does).


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by André Warnier <aw...@ice-sa.com>.
Mark H. Wood wrote:
> Local DNS cache of some sort on the Fedora box?  Some of my hosts run
> 'nscd' (which comes with glibc) and some run BIND in cache-only mode.
> 
That's also, I believe, what a local "dnsmasq" would do.  But in order 
for the local Apache to use it, you'd have to set 127.0.0.1 as a 
resolver in /etc/resolv.conf, which apparently is not the case here.

To get back to the basic issue..
We seem to have, on the same host
- an "old" Apache configuration, which does not trigger client DNS names 
lookups
- a "new" Apache configuration, which does
Yes ?

So I guess we can exclude for the time being the possibility that it 
might be something else than Apache which does the lookups (I was 
thinking of denyhosts or the like).

Next obvious question : what's the difference between the old and new 
configurations ? (sorry if that was explained before, I did not get back 
to the archives).
Do I vaguely remember it had something to do with SSL, or was that 
another thread ?



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by André Warnier <aw...@ice-sa.com>.
Mike -- EMAIL IGNORED wrote:
> On Tue, 04 Aug 2009 14:33:12 -0400, Mark H. Wood wrote:
> 
>> Local DNS cache of some sort on the Fedora box?  Some of my hosts run
>> 'nscd' (which comes with glibc) and some run BIND in cache-only mode.
> 
> Indeed there are two problems, and you nailed one of them:
> 
> 1. Both boxes have nscd, but it was not running on the CentOS
>    box.  Now that I fixed that, all but the first connection
>    are rapid, as you might expect.
> 
> 2. We still do not why the change in httpd.conf caused the
>    problem to appear.  However, my belief that there was a
>    difference between the two machines was probably incorrect.
>    When I tried it for the first time this morning, the box
>    that previously been fast was now slow.  No doubt, the
>    nscd storage had timed out.
> 
So, you still have DNS lookups of the clients happening.
If they are satisfied locally by some name server running on 127.0.0.1, 
you would not see them with wireshark, probably.
And when they are not in the cache, then it is probably nscd which does 
the DNS lookup elsewhere, not Apache.
Now the question remains of why these client lookups take place at all, 
even if they are often satisfied locally and fast.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: excessive dns slows httpd

Posted by Mike -- EMAIL IGNORED <m_...@yahoo.com>.
On Tue, 04 Aug 2009 14:33:12 -0400, Mark H. Wood wrote:

> Local DNS cache of some sort on the Fedora box?  Some of my hosts run
> 'nscd' (which comes with glibc) and some run BIND in cache-only mode.

Indeed there are two problems, and you nailed one of them:

1. Both boxes have nscd, but it was not running on the CentOS
   box.  Now that I fixed that, all but the first connection
   are rapid, as you might expect.

2. We still do not why the change in httpd.conf caused the
   problem to appear.  However, my belief that there was a
   difference between the two machines was probably incorrect.
   When I tried it for the first time this morning, the box
   that previously been fast was now slow.  No doubt, the
   nscd storage had timed out.

Thanks for your help.

Mike


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by "Mark H. Wood" <mw...@IUPUI.Edu>.
Local DNS cache of some sort on the Fedora box?  Some of my hosts run
'nscd' (which comes with glibc) and some run BIND in cache-only mode.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Friends don't let friends publish revisable-form documents.

[users@httpd] Re: excessive dns slows httpd

Posted by Mike -- EMAIL IGNORED <m_...@yahoo.com>.
On Tue, 04 Aug 2009 17:06:34 +0000, Mike -- EMAIL IGNORED wrote:

> Later examination showed that the Fedora box (the one working well)
> occasionally also does a DNS lookup, but it happens very rapidly. All
> the DNS activity is the same -- it is doing a reverse lookup on the
> client IP address.  Thanks for your suggestion regarding
> /etc/resolv.conf .  I'll do that now.
> 
> No matter what I find there, however, we would still lack an explanation
> regarding why the new configuration causes the problem and the old one
> does not.  This is reproducible -- I have both.
> 
> Mike.
> 
/etc/resolv.conf is identical on the two boxes.  All they have are the
addresses of my routers, which have the real DNS addresses.  I have
two nested routers; the outer one emulates the WAN for tests.  (Don't
worry, my test results were confirmed from a real external location.)

Mike.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: excessive dns slows httpd

Posted by Mike -- EMAIL IGNORED <m_...@yahoo.com>.
On Tue, 04 Aug 2009 18:43:14 +0200, André Warnier wrote:

[...]

> Yet one appears to do tons of DNS lookups, and the other one not. Logic
> would have it that there /is/ something different somewhere in either
> the configuration, or some file(s) you are not thinking about right now,
> that make the difference.
> You mentioned that you saw the DNS lookups in wireshark. Can you tell
> what these DNS lookups are for ? Have you compared the DNS setup of the
> servers (meaning their /etc/resolv.conf and e.g. whether one of them is
> running dnsmasq and the other one not, or something like that) ?
> 

Later examination showed that the Fedora box (the one working well)
occasionally also does a DNS lookup, but it happens very rapidly.
All the DNS activity is the same -- it is doing a reverse lookup on
the client IP address.  Thanks for your suggestion regarding
/etc/resolv.conf .  I'll do that now.

No matter what I find there, however, we would still lack an
explanation regarding why the new configuration causes the
problem and the old one does not.  This is reproducible --
I have both.

Mike.



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by André Warnier <aw...@ice-sa.com>.
Mike -- EMAIL IGNORED wrote:
> On Tue, 04 Aug 2009 16:00:54 +0000, Mike -- EMAIL IGNORED wrote:
> 
>> On Tue, 04 Aug 2009 16:24:25 +0100, Tom Evans wrote:
>>
>>> HostNameLookups
>> HostNameLookups Off
>> in the only place it appears.  Perhaps I should compare all files in
>> conf.d on my two machines.
>>
>> Mike.
>>
> [...]
> 
> I did the comparison and the contents of conf.d on my two
> boxes are identical except that the Fedora box has
> BackupPC.conf, which does not appear on the CentOS box.
> Examining BackupPC.conf, it appears to me to have
> nothing relevant.
> 
Of course you realise that nobody here is really believing you..
;-)
I mean, here are two Linux systems, with the same kind of processor, not 
too different versions of kernel nor apache, and identical (to some 
extent) config files.
Yet one appears to do tons of DNS lookups, and the other one not.
Logic would have it that there /is/ something different somewhere in 
either the configuration, or some file(s) you are not thinking about 
right now, that make the difference.
You mentioned that you saw the DNS lookups in wireshark.
Can you tell what these DNS lookups are for ?
Have you compared the DNS setup of the servers (meaning their 
/etc/resolv.conf and e.g. whether one of them is running dnsmasq and the 
other one not, or something like that) ?


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: excessive dns slows httpd

Posted by Mike -- EMAIL IGNORED <m_...@yahoo.com>.
On Tue, 04 Aug 2009 16:00:54 +0000, Mike -- EMAIL IGNORED wrote:

> On Tue, 04 Aug 2009 16:24:25 +0100, Tom Evans wrote:
> 
>> HostNameLookups
> 
> HostNameLookups Off
> in the only place it appears.  Perhaps I should compare all files in
> conf.d on my two machines.
> 
> Mike.
> 
[...]

I did the comparison and the contents of conf.d on my two
boxes are identical except that the Fedora box has
BackupPC.conf, which does not appear on the CentOS box.
Examining BackupPC.conf, it appears to me to have
nothing relevant.

Mike.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: excessive dns slows httpd

Posted by Mike -- EMAIL IGNORED <m_...@yahoo.com>.
On Tue, 04 Aug 2009 16:24:25 +0100, Tom Evans wrote:

> HostNameLookups

HostNameLookups Off
in the only place it appears.  Perhaps I should
compare all files in conf.d on my two machines.

Mike.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: excessive dns slows httpd

Posted by Tom Evans <te...@googlemail.com>.
On Tue, 2009-08-04 at 15:05 +0000, Mike -- EMAIL IGNORED wrote:
> On Tue, 04 Aug 2009 08:22:14 +0200, André Warnier wrote:
> 
> [...]
> >> 
> > You could first check the obvious, such as whether in your Apache/Centos
> > config, you are not *asking* the server to do these DNS lookups. Look
> > here for instance :
> > http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats (I mean
> > that if your are asking Apache to log the *name* of the client in the
> > access log for instance, then it has to do a lookup) Any time, anywhere,
> > where Apache, for whatever reason, has to translate an IP address into a
> > name or vice-versa, may result in this behaviour.
> > 
> [...]
> 
> In LogFormat, I changed %h to %a, which hat no effect.  I did "solve"
> the problem by completely blocking DNS in iptables.  As it happens,
> in my present configuration, this is not a problem, but I would hardly
> call it a satisfactory solution.  Remember, the httpd.config works
> without problem on a neighboring box (with a different OS).
> 
> Mike.
> 

I doubt they are running identical configurations then. At a pinch, I
would guess that the slow one has 'HostNameLookups On' (or worse:
Double). 
It would be impossible to help further on this without actual conf
files.

Tom


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org