[users@httpd] strange random delays in a proxy configuration

[André Warnier <aw...@ice-sa.com>]

Hi guys.

A customer of ours have an internal Apache 2.2 server under Linux 
Debian.  Static pages of their applications are located on their own server.
Some URLs are proxied by their server to our own Apache 2.2 server 
(Linux Debian), through the Internet, by means of a line like

ProxyPass /getthumb http://our-server.company.com
and the URLs are like
/getthumb?id=(thumb-id)

These URLs obtain thumbnails from our server, corresponding to 
documents.  Each thumbnail is generated on the fly and is about 10 KB in 
size. It takes our server on average 150 ms to process such a URL and 
return the thumbnail to their server.
The number of thumbnails to serve for one customer application page is 
variable, between just a few and sometimes several hundred, with an 
average of 20-30 per page.
In the vast majority of cases, each thumbnail request is served in 
around 150 ms, and we serve thousands such requests per day.

Yet sometimes, for no apparent reason that we can link to the process of 
generating and returning the thumbnail, nor to any load of our own 
Apache server or theirs, returning one arbitrary thumbnail in the series 
takes about 30 seconds.
It is always either about 150 ms, or about 30000 ms.  It is never 
in-between, like 10 or 20 seconds or so.

The delay does not always affect the same thumbnails either.  For the 
same requested page with the same series of thumbnails, sometimes the 30 
seconds delay will happen at the first and second thumbnail, sometimes 4 
of them in the middle, and so on. Sometimes the whole page-worth of 
thumbnails loads in exactly (number of thumbnails X 150 ms), which is 
what it should always be.

If I watch our server's processes with "top", and simultaneously the end 
of the access log or the screen of one of their workstations, I can see 
nothing untoward when the browser suddenly "hangs" for 30 seconds 
waiting fo the next thumbnail.  The Apache processes on our side seem 
mostly idle at that time, and there are plenty of Apache processes 
available to serve any request.
It looks as if the browser on the customer side just hangs waiting for 
the thumbnail, and our server is just idle.

These times are per our own server's access log, and per their Firefox 
browser, to which we added the HttpFox add-on, to see what happens from 
their side.
Their browsers do issue the requests (almost all at once), and the 
symptoms are the same whether they use IE or Firefox.

My first question is : does there exist any tool in Apache itself, that 
would show me what is happening during such a 30-second pause ?
The access log does not, as it just adds a line at the end of the 30 
seconds, when the request is terminated, which is too late.  There are 
no errors in our error log.
mod_log_io and mod_forensic, from their documentation, do not seem to 
provide much more relevant info.

For example, since their server proxies these calls to ours, and we see 
nothing in our logs, is there a way whereby I can set the logging of the 
proxy calls (and only those, preferably) to some "debug" setting ?

(I have full access to their and our Apache servers configurations)

My second question would be : does anyone have an idea of what could 
explain sudden random 30 second pauses in such a schema ?

I am trying to avoid having to go through a TCP protocol analyser, 
because I am unfamiliar with them, at least at the Linux level.


Thanks

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] [SOLVED] strange random delays in a proxy configuration

[André Warnier <aw...@ice-sa.com>]

André Warnier wrote:
> Hi guys.
> 
> A customer of ours have an internal Apache 2.2 server
...
Never mind, I believe we found the problem, and it has nothing to do 
with the network or Apache or proxying per se.
Sorry to have wasted your time if I did.

It seems to be the application that is misbehaving, reloading some big 
table in each Apache child from time to time, when it has no reason to.
That explains why the pattern seemed random too, because when it happens 
depends on how many requests are sent by each client and directed to 
which Apache child, and the settings of MaxKeepAliveRequests and so on 
with which I have been fiddling in attempts to circumscribe the issue.



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org