[users@httpd] DNS lookups made by NoProxy directive

[Amol Thorat <am...@persistent.co.in>]

Hi,

I am an Apache newbie so please bear with me if I am off track :)

We are using Apache 1.3.x on Linux and use the mod_proxy module with 
plenty of NoProxy rules. Some of the rules are for host names and some 
are for IP addresses. e.g.
-----------------------------------------------------------------------------
NoProxy www2.a-create.jp .ofoto.com .plspictures.com .r4mobile.com 
.thqwireless.com .volantis.net
NoProxy 12.47.197.55 127.0.0.1 144.230.114.33 144.230.162.12 
144.230.32.17 144.230.37.11 144.230.37.12
-----------------------------------------------------------------------------

What we found was that for each Apache mod_proxy request, number of 
identical DNS requests for the request URL hostname are made equal to 
the number of IP addresses in the NoProxy list. e.g. With above 
configuration, if a request is made to http://www.yahoo.com then DNS 
requests for "www.yahoo.com" are fired 7 times (equal to number of IP 
addresses in the NoProxy list). What I was wondering is - why can't the 
IP address obtained in the first DNS call be reused? Because of this 
behavior, the DNS server is getting overloaded.

I also tried to search the net for any bug reported of this kind, but 
could not find anything relevant. However, I found this: 
http://httpd.markmail.org/message/ad54tf6lvh7gce5g?q=mod_proxy+noproxy+dns+ip&page=4.#query:mod_proxy%20noproxy%20dns%20ip+page:4+mid:svpnf6hdfvalwm3o+state:results 
<http://httpd.markmail.org/message/ad54tf6lvh7gce5g?q=mod_proxy+noproxy+dns+ip&page=4.>
Note the following content on the page: "It could be improved 
DNS-lookup-wise (only look up a given host once)". I am not sure if this 
is referring to the same problem I am having. Can someone confirm and if 
yes, whether this is fixed in any of the later releases?

The rest of the message is a walk through the Apache code which confirms 
this behavior:

When any URL request comes in, Apache needs to check if it matches any 
of the
entries in the NoProxy list. For this, when the configuration is being 
read,
for each entry, the code sets a "matcher" (a handler that does the
match of exclusion list with the URL ) to be called.

In file "mod_proxy.c", in function set_proxy_dirconn (Irrelevant
debug statements are removed),
--------------------------------------------------------------------------------
if (ap_proxy_is_ipaddr(New, parms->pool)) {
....
}
else if (ap_proxy_is_domainname(New, parms->pool)) {
ap_str_tolower(New->name);
....
else if (ap_proxy_is_hostname(New, parms->pool)) {
ap_str_tolower(New->name);
.....
}
else {
ap_proxy_is_word(New, parms->pool);
#if DEBUGGING
fprintf(stderr, "Parsed word %s\n", New->name);
#endif
}
--------------------------------------------------------------------------------

ap_proxy_is_ipaddr sets the matcher to "proxy_match_ipaddr" and
ap_proxy_is_hostname sets the matcher to "proxy_match_hostname". So
for each IP address entry in the exclusion list, function 
proxy_match_ipaddr
will be called and for every hostname entry, the proxy_match_hostname is
called.

proxy_match_hostname does simple string matches for hostname in URL and in
exclusion list. If string matches, it returns 1 otherwise returns 0.

proxy_match_ipaddr checks if the incoming URL has IP address in it and 
if yes,
whether it matches the IP address in the exclusion list. If IP address 
is found
in the exclusion list, it returns 1 otherwise returns 0. Now, if the URL 
does
not have an IP address, it calls ap_proxy_host2addr (which internally calls
gethostbyname) for the host name in the URL. If one of the IP addresses
returned by the DNS lookup matches the IP address in the exclusion list, it
returns 1 otherwise returns 0.

Now when a URL request comes to mod_proxy, it passes the request through 
each
entry in the exclusion list till it matches or till you run out of loop:

--------------------------------------------------------------------------------
int ii;
struct dirconn_entry *list = (struct dirconn_entry
*)conf->dirconn->elts;

for (direct_connect = ii = 0; ii < conf->dirconn->nelts
&& !direct_connect; ii++) {
direct_connect = list[ii].matcher(&list[ii], r);
}
--------------------------------------------------------------------------------

This results in the DNS request made for every IP address entry in the 
NoProxy list.


-- 

Thanks and Regards,

Amol



DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org