You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by wglass <wg...@gmail.com> on 2007/08/31 11:23:38 UTC
[users@httpd] performance problem in one virtual host
Hi,
I've had a bizarre performance problem earlier this week and I can't figure
it out. It'd be great if anyone had suggestions.
Twice recently, I've had extremely slow performance serving static files
from a single virtual host on my Apache server. Files that were sent to
Tomcat with mod_jk returned quickly, and files on other virtual hosts on the
same machine returned quickly. But every 1 in 3 static files served by
httpd in the affected virtual host took several seconds to return,
irrespective of size. (e.g. a 12 kb image could take 6 seconds to return).
A couple of notes:
* Server load was low and swap space was barely being used
* The problem appeared out of nowhere (no config changes) happened for an
hour and went away. A week later it appeared for about 4 hours, then again
fixed without warning.
* When I copied all the web files to a different virtual host on the same
box,it was lightening fast.
* I suspected DNS/reverse DNS issues. But I ran a tcpdump on port 53 and
saw no signs of DNS queries
* I had large log files. I renamed them and stopped/started httpd. Still
slow.
* I ran a web analyzer on the Apache logs. Traffic was exactly average.
* When I accessed the server from a machine on the same subnet, access was
fast! So network distance (or maybe IP address) plays a role.
One of my colleagues suspected the problem was strange registry settings at
our top level domain. We were using nameservers from two different vendors
for reliability. (and when we tested them, there was a serial number
mismatch between them). But I hardcoded the DNS on my client in /etc/hosts
(or the equivalent) and still saw the problem.
Any pointers on what I should look for to diagnose this problem / prevent it
in the future? I'm really at a loss as to what to do.
We're running Apache 2.0.52 on RHEL 4, on a dual core, dual Woodcrest
processor machine with 4GB Ram (also runs MySQL and Tomcat). The machine is
in a data farm in Texas somewhere.
Thanks,
WILL
--
View this message in context: http://www.nabble.com/performance-problem-in-one-virtual-host-tf4359125.html#a12423274
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] performance problem in one virtual host
Posted by wglass <wg...@gmail.com>.
Just a quick followup on this message. I think Nick's theory on DNS was
right. I talked to my ISP provider, and they do a DNS lookup in the switch
for every web request. So, even though the server and client weren't doing
DNS lookups, the intermediate network was. And the symptoms of the problem
were certainly reminiscent of sporadic DNS failures. We've reconfigured the
DNS for the server and the problem hasn't reoccurred - I'm hoping this takes
care of it.
A belated thanks, Nick! for the help.
WILL
wglass wrote:
>
> Hi Nick,
>
> I'm puzzled by the client DNS theory, since I had the host (in Windows,
> mind you) directly mapped in /etc/hosts. (the Windows equivalent).
>
> The argument in favor of the DNS theory is that we ran a web test of DNS
> with www.dnsreports.com and it reported several problems. (we were using
> two vendors and they had inconsistent serial numbers).
>
> New idea-- could DNS issues be causing cache performance problems at the
> ISP? (I'm accessing via a Comcast cable modem). Anyone have experience
> with this type of issue?
>
> WILL
>
>
>
> Nick Kew wrote:
>>
>> On Fri, 31 Aug 2007 06:46:37 -0700 (PDT)
>> wglass <wg...@gmail.com> wrote:
>>
>>>
>>> Thanks, Nick.
>>>
>>> Can you explain your question about DNS?
>>
>> If it's one vhost, that suggests the possibility of an issue
>> with DNS lookups at the client end.
>>
>>> What's the easiest way to get and test an HTTP/1.0 client?
>>
>> I usually use lynx for it.
>>
>> --
>> Nick Kew
>>
>> Application Development with Apache - the Apache Modules Book
>> http://www.apachetutor.org/
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/performance-problem-in-one-virtual-host-tf4359125.html#a12639610
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] performance problem in one virtual host
Posted by wglass <wg...@gmail.com>.
Hi Nick,
I'm puzzled by the client DNS theory, since I had the host (in Windows, mind
you) directly mapped in /etc/hosts. (the Windows equivalent).
The argument in favor of the DNS theory is that we ran a web test of DNS
with www.dnsreports.com and it reported several problems. (we were using
two vendors and they had inconsistent serial numbers).
New idea-- could DNS issues be causing cache performance problems at the
ISP? (I'm accessing via a Comcast cable modem). Anyone have experience
with this type of issue?
WILL
Nick Kew wrote:
>
> On Fri, 31 Aug 2007 06:46:37 -0700 (PDT)
> wglass <wg...@gmail.com> wrote:
>
>>
>> Thanks, Nick.
>>
>> Can you explain your question about DNS?
>
> If it's one vhost, that suggests the possibility of an issue
> with DNS lookups at the client end.
>
>> What's the easiest way to get and test an HTTP/1.0 client?
>
> I usually use lynx for it.
>
> --
> Nick Kew
>
> Application Development with Apache - the Apache Modules Book
> http://www.apachetutor.org/
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/performance-problem-in-one-virtual-host-tf4359125.html#a12430944
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] performance problem in one virtual host
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 31 Aug 2007 06:46:37 -0700 (PDT)
wglass <wg...@gmail.com> wrote:
>
> Thanks, Nick.
>
> Can you explain your question about DNS?
If it's one vhost, that suggests the possibility of an issue
with DNS lookups at the client end.
> What's the easiest way to get and test an HTTP/1.0 client?
I usually use lynx for it.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] performance problem in one virtual host
Posted by wglass <wg...@gmail.com>.
Thanks, Nick.
Can you explain your question about DNS? My assumption is that if I had a
reverse DNS issue, I could identify it by running tcpdump on port 53 on the
server. (e.g. with HostnameLookup) I did this and saw no TCP/UDP traffic.
To eliminate the possibility of client DNS issues, I hardcoded the IP
address to the domain name in /etc/hosts on the client. Is there something
I'm missing here?
The files are accessed directly on the same hard drive by httpd, though they
are also made available via NFS and SMB to other systems.
What's the easiest way to get and test an HTTP/1.0 client?
WILL
Nick Kew wrote:
>
> On Fri, 31 Aug 2007 02:23:38 -0700 (PDT)
> wglass <wg...@gmail.com> wrote:
>
>> * I suspected DNS/reverse DNS issues. But I ran a tcpdump on port 53
>> and saw no signs of DNS queries
>
> It does look an obvious candidate. And why would DNS be generating
> traffic (let alone TCP traffic) on the server?
>
> The other quick check I'd run is to see if an HTTP/1.0 client
> suffers the same delays. But it would be bizarre indeed for that
> to happen as you describe, with one vhost and static files.
>
> Talking of which, are that vhost's files mounted on anything exotic
> or non-local?
>
> --
> Nick Kew
>
> Application Development with Apache - the Apache Modules Book
> http://www.apachetutor.org/
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/performance-problem-in-one-virtual-host-tf4359125.html#a12426754
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] performance problem in one virtual host
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 31 Aug 2007 02:23:38 -0700 (PDT)
wglass <wg...@gmail.com> wrote:
> * I suspected DNS/reverse DNS issues. But I ran a tcpdump on port 53
> and saw no signs of DNS queries
It does look an obvious candidate. And why would DNS be generating
traffic (let alone TCP traffic) on the server?
The other quick check I'd run is to see if an HTTP/1.0 client
suffers the same delays. But it would be bizarre indeed for that
to happen as you describe, with one vhost and static files.
Talking of which, are that vhost's files mounted on anything exotic
or non-local?
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org