You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Shanti Suresh <sh...@umich.edu> on 2013/04/15 21:14:49 UTC
Apache HTTPD + Tomcat restarts
All,
I made a simple change to the Apache HTTPD configs - I reduced the number
of loaded modules - and then stopped and started Apache. On the QA
environment, Tomcat did not need a restart. On the PROD (production)
environment, soon after Apache was restarted, the application URLs resolved
okay. But a minute later, I received an Internal Server Error from
Apache. Hitting Tomcat directly did not work either. It required a Tomcat
restart to start receiving the URLs again. I am not sure why the two
environments behaved differently. It was not my browser cache, 'cause I
reloaded the URL. It was not the loadbalancer RAM Cache as the
loadbalancer RAM Cache is disabled. Didn't find anything in the logs. I
wanted to know your thoughts.
Thanks.
-Shanti
Re: Apache HTTPD + Tomcat restarts
Posted by Shanti Suresh <sh...@umich.edu>.
Hi André, Hi all,
On Mon, Apr 15, 2013 at 7:02 PM, André Warnier <aw...@ice-sa.com> wrote:
>
> Well, you wrote that you wanted to know our thoughts. Yes or No would not
> have been an appropriate response to that, so we elaborated a bit.
> ;-)
>
>
> Yes, true. Thank you! I am a bit unsure what's going on. So greatly
appreciate your feedback.
>
>> Here's all I have from Apache's error log. Several entries looking like
>> so
>> starting at:
>> ----------
>> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
>> timeout specified has expired: proxy: error reading status line from
>> remote
>> server servername.do.ma.in
>> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
>> reading from remote server returned by /web-app/url1/
>> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
>> reading from remote server returned by /web-app/url2, referer:
>> http://example.com/people/**employees/regularemployee<http://example.com/people/employees/regularemployee>
>> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
>> timeout specified has expired: proxy: error reading status line from
>> remote
>> server servername.do.ma.in, referer: https://www.google.com/
>> ---------
>>
>> Apache and Tomcat are in the same city, and in the same data-center.
>>
>> And Apr 11 21:05 is when I restarted Apache after removing several unused
>> modules - hard stop and start. The way I restarted Apache is:
>> sudo /usr/sbin/service httpd stop
>> sudo /usr/bin/servive httpd start
>>
>>
> So you wrote that you restarted Apache at 21:05, and that about a minute
> later you had some Server Errors. The log messages above are dated of
> 21:24. That is 20 minutes after the restart, not really 1 minute.
>
Here are the some facts from the HTTPD access log(!):
On 4/11/2013:
HTTPD was down from 21:03 - 21:04:07
HTTPD started returning several Error 502's from: 21:20:35 to 21:37:58
Tomcat seemed to hang from 21:20 onwards - per our monitoring software
Tomcat restart was issued at: 21:46
Tomcat was OK at 21:49
It appears to me that restarting Apache HTTPD caused Tomcat to be confused
about something. About what? Some things I did in order to trim Apache
HTTPD include:
(1) I renamed the proxy_ajp.conf to proxy_ajp.conf.notinclude and
balancers.conf to balancers.conf.notinclude since these are not being used
( I use mod_proxy and mod_proxy_http)
(2) I trimmed HTTPD modules from 47 to 17 by commenting out "LoadModule"
statements and removed offending directives in httpd.conf
(3) I restarted HTTPD
If I have to express a thought based on that conflicting information, then
> I would guess that the errors are probably not related to the restart of
> Apache per se.
> It may just have been a coincidence. Although using the term
> "coincidence" for events that happen at a 20-minute interval on a
> production server seems a bit of a stretch.
>
I agree that that theory would be a bit if a stretch.
>
>
> In any case, there is nothing there that specifically links the Apache
> restart, with the errors that happened later.
>
I know. The funny thing is that happened independently on more than one
server - same behavior.
> If Tomcat tried to send a response later (after Apache httpd timed-out),
> then there should probably be an error message also in the Tomcat logs,
> since it would try to write back a response onto a connection that did not
> exist anymore.
>
> Have you checked the Tomcat logs ? (which may be in /var/log/daemon,
> depending on the script which starts Tomcat).
There is surprisingly nothing that stands out in the Tomcat logs from 4/11/
evening. I do not have the access logs enabled on the Tomcat side. And
one log has rolled over. But I doubt if that will have anything to say.
Although I will look into restoring that log file to check.
Some questions I have include - if HTTPD is restarted, what happens to
Tomcat's thread pool? Will they timeout after writing to a half-closed
connection? When HTTPD starts up, will it open new connections to Tomcat's
ports - I think the answer is "yes". HTTPD should start minServer
processes and connect to Tomcat?
What might cause Tomcat to work somewhat and to not work somewhat, at the
same time. Perhaps old Tomcat threads connecting to the half-closed HTTPD
connections might be confused, like you suggest? But the new connections
with HTTPD after the restart might be okay?
Thanks much.
-Shanti
Re: Apache HTTPD + Tomcat restarts
Posted by André Warnier <aw...@ice-sa.com>.
Shanti Suresh wrote:
> Hi all,
>
> Apologies :-) I was expecting a simple yes/no response.
Well, you wrote that you wanted to know our thoughts. Yes or No would not have been an
appropriate response to that, so we elaborated a bit.
;-)
>
>
> On Mon, Apr 15, 2013 at 4:01 PM, André Warnier <aw...@ice-sa.com> wrote:
>
>> Mark Eggers wrote:
>>
>>> Apache HTTPD modules do not influence how Apache Tomcat runs.
>>>
> That's what I thought.
>
>
>
>>>
>>> If that cures your problem, then you should take things up with the
>>> mod_security mailing list.
>>>
>>> Thank you! I am not running mod_security.
>
>
>>> Versions of:
>>>
>>> Apache HTTPD
>>>
>> Server version: Apache/2.2.3
>
>> Apache Tomcat
>> apache-tomcat-7.0.23
>
>> Java
>> jre-1.6.0-sun.x86_64
>
>>> Operating system and version
>>>
>> RedHat Linux 2.6.18-348.2.1.el5
>
>>> Method you use to connect Apache HTTPD to Apache Tomcat
>>>
>> mod_proxy + mod_proxy_http
>
>
>>> Text of the Internal Server Error. If you've not configured an error page
>>> for the 500 error, then you'll see a stack trace if it's from Tomcat.
>>>
> Don't have the Internal Server error's exact text. There was no stack
> trace from Tomcat. It was simply an Internal Server 500 error text on the
> Apache side. I have not configured an error page. Gosh! I don't
> recollect anything on the Tomcat direct-hit either. Otherwise, I would be
> a happy camper.
>
>
>> And you are guessing quite a lot. With the information provided by the OP
>> initially, I was going to ask if Apache httpd and Tomcat were located in
>> the same city and/or connected in any way. Oh, and he also mentioned a
>> loadbalancer cache. Either Pid or Quantum Theory might be of help here.
>>
>
> My apologies.
>
> Here's all I have from Apache's error log. Several entries looking like so
> starting at:
> ----------
> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
> timeout specified has expired: proxy: error reading status line from remote
> server servername.do.ma.in
> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
> reading from remote server returned by /web-app/url1/
> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
> reading from remote server returned by /web-app/url2, referer:
> http://example.com/people/employees/regularemployee
> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
> timeout specified has expired: proxy: error reading status line from remote
> server servername.do.ma.in, referer: https://www.google.com/
> ---------
>
> Apache and Tomcat are in the same city, and in the same data-center.
>
> And Apr 11 21:05 is when I restarted Apache after removing several unused
> modules - hard stop and start. The way I restarted Apache is:
> sudo /usr/sbin/service httpd stop
> sudo /usr/bin/servive httpd start
>
So you wrote that you restarted Apache at 21:05, and that about a minute later you had
some Server Errors. The log messages above are dated of 21:24. That is 20 minutes after
the restart, not really 1 minute.
If I have to express a thought based on that conflicting information, then I would guess
that the errors are probably not related to the restart of Apache per se.
It may just have been a coincidence. Although using the term "coincidence" for events
that happen at a 20-minute interval on a production server seems a bit of a stretch.
The (edited) error log shown above just tells us that at 21:24:14, Apache+mod_proxy
determined that there was a timeout following an earlier request to Tomcat. In other
words, Apache proxied a request to Tomcat, and Tomcat did not respond in time, so
mod_proxy gave up waiting and aborted the request with an error.
And then again 40 seconds later at 21:24:55, same issue.
Now why at that time Tomcat did not respond in time (or at all), is a mystery as well for
us as for you.
There is nothing there that would tell us when the request which failed was actually
started. We just see when it failed.
Maybe by looking at the same time at the Apache access log and at this error log, you can
determine which requests failed, and figure out if there is some logical reason at the
Tomcat level. Maybe these requests were just so that Tomcat really took "too long" to
answer, and mod_proxy lost patience...
In any case, there is nothing there that specifically links the Apache restart, with the
errors that happened later.
If Tomcat tried to send a response later (after Apache httpd timed-out), then there should
probably be an error message also in the Tomcat logs, since it would try to write back a
response onto a connection that did not exist anymore.
Have you checked the Tomcat logs ? (which may be in /var/log/daemon, depending on the
script which starts Tomcat).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache HTTPD + Tomcat restarts
Posted by Shanti Suresh <sh...@umich.edu>.
On Mon, Apr 15, 2013 at 5:02 PM, Shanti Suresh <sh...@umich.edu> wrote:
>
>
> And Apr 11 21:05 is when I restarted Apache after removing several unused
> modules - hard stop and start. The way I restarted Apache is:
> sudo /usr/sbin/service httpd stop
> sudo /usr/bin/servive httpd start
>
>
>
Typo. Should be read as:
sudo /sbin/service httpd stop
sudo /sbin/service httpd start
Thanks!
-Shanti
Re: Apache HTTPD + Tomcat restarts
Posted by Shanti Suresh <sh...@umich.edu>.
Hi all,
Apologies :-) I was expecting a simple yes/no response.
On Mon, Apr 15, 2013 at 4:01 PM, André Warnier <aw...@ice-sa.com> wrote:
> Mark Eggers wrote:
>
>>
>> Apache HTTPD modules do not influence how Apache Tomcat runs.
>>
>
That's what I thought.
>
>>
>>
>> If that cures your problem, then you should take things up with the
>> mod_security mailing list.
>>
>> Thank you! I am not running mod_security.
>
>> Versions of:
>>
>> Apache HTTPD
>>
> Server version: Apache/2.2.3
> Apache Tomcat
>>
> apache-tomcat-7.0.23
> Java
>>
> jre-1.6.0-sun.x86_64
>
>> Operating system and version
>>
> RedHat Linux 2.6.18-348.2.1.el5
>
>> Method you use to connect Apache HTTPD to Apache Tomcat
>>
> mod_proxy + mod_proxy_http
>
>> Text of the Internal Server Error. If you've not configured an error page
>> for the 500 error, then you'll see a stack trace if it's from Tomcat.
>>
>
Don't have the Internal Server error's exact text. There was no stack
trace from Tomcat. It was simply an Internal Server 500 error text on the
Apache side. I have not configured an error page. Gosh! I don't
recollect anything on the Tomcat direct-hit either. Otherwise, I would be
a happy camper.
>
> And you are guessing quite a lot. With the information provided by the OP
> initially, I was going to ask if Apache httpd and Tomcat were located in
> the same city and/or connected in any way. Oh, and he also mentioned a
> loadbalancer cache. Either Pid or Quantum Theory might be of help here.
>
My apologies.
Here's all I have from Apache's error log. Several entries looking like so
starting at:
----------
[Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
timeout specified has expired: proxy: error reading status line from remote
server servername.do.ma.in
[Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
reading from remote server returned by /web-app/url1/
[Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error
reading from remote server returned by /web-app/url2, referer:
http://example.com/people/employees/regularemployee
[Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The
timeout specified has expired: proxy: error reading status line from remote
server servername.do.ma.in, referer: https://www.google.com/
---------
Apache and Tomcat are in the same city, and in the same data-center.
And Apr 11 21:05 is when I restarted Apache after removing several unused
modules - hard stop and start. The way I restarted Apache is:
sudo /usr/sbin/service httpd stop
sudo /usr/bin/servive httpd start
Thanks.
-Shanti
Re: Apache HTTPD + Tomcat restarts
Posted by André Warnier <aw...@ice-sa.com>.
Mark Eggers wrote:
> On 4/15/2013 12:14 PM, Shanti Suresh wrote:
>> All,
>>
>> I made a simple change to the Apache HTTPD configs - I reduced the number
>> of loaded modules - and then stopped and started Apache. On the QA
>> environment, Tomcat did not need a restart. On the PROD (production)
>> environment, soon after Apache was restarted, the application URLs
>> resolved
>> okay. But a minute later, I received an Internal Server Error from
>> Apache. Hitting Tomcat directly did not work either. It required a
>> Tomcat
>> restart to start receiving the URLs again. I am not sure why the two
>> environments behaved differently. It was not my browser cache, 'cause I
>> reloaded the URL. It was not the loadbalancer RAM Cache as the
>> loadbalancer RAM Cache is disabled. Didn't find anything in the logs. I
>> wanted to know your thoughts.
>>
>> Thanks.
>>
>> -Shanti
>>
>
> Apache HTTPD modules do not influence how Apache Tomcat runs.
>
> So now I have a little better picture of what's running:
>
> Browser --> Apache HTTPD --> (multiple) Tomcats
>
> I don't have any idea what version of anything you're running.
>
> I'm going to guess (again) that you're using mod_jk and a load balancer
> configuration. However you could be using mod_proxy and
> mod_proxy_balancer. I don't know.
>
> If the 500 Internal Server Error is from Apache HTTPD, then it's an
> Apache HTTPD configuration issue and belongs on the Apache HTTPD mailing
> list.
>
> Are you running mod_security in production? I've seen some mod_security
> Internal Server Error issues when posting WAR files through Apache HTTPD
> / mod_jk / Tomcat. The log file for that should be where Apache HTTPD
> logs things.
>
> If the Internal Server Error comes from Tomcat, then it's most likely
> your application. You can check the log files in CATALINA_BASE/logs.
> There should be information in those logs.
>
> If you are running mod_security on a front-end Apache HTTPD server and
> want to quickly check, you could add the following to an Apache HTTPD
> server virtual host to turn it off (temporarily).
>
> <IfModule security2_module>
> SecRuleEngine Off
> </IfModule>
>
> If that cures your problem, then you should take things up with the
> mod_security mailing list.
>
> Of course, this is all a guess (I'll put your crystal ball back in a
> second, Pid). More information is needed:
>
> Versions of:
>
> Apache HTTPD
> Apache Tomcat
> Java
>
> Operating system and version
>
> Method you use to connect Apache HTTPD to Apache Tomcat
>
> Text of the Internal Server Error. If you've not configured an error
> page for the 500 error, then you'll see a stack trace if it's from Tomcat.
>
> Without all of this information (at a minimum), the best anyone can do
> is guess.
>
And you are guessing quite a lot. With the information provided by the OP initially, I was
going to ask if Apache httpd and Tomcat were located in the same city and/or connected in
any way. Oh, and he also mentioned a loadbalancer cache. Either Pid or Quantum Theory
might be of help here.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache HTTPD + Tomcat restarts
Posted by Mark Eggers <it...@yahoo.com>.
On 4/15/2013 12:14 PM, Shanti Suresh wrote:
> All,
>
> I made a simple change to the Apache HTTPD configs - I reduced the number
> of loaded modules - and then stopped and started Apache. On the QA
> environment, Tomcat did not need a restart. On the PROD (production)
> environment, soon after Apache was restarted, the application URLs resolved
> okay. But a minute later, I received an Internal Server Error from
> Apache. Hitting Tomcat directly did not work either. It required a Tomcat
> restart to start receiving the URLs again. I am not sure why the two
> environments behaved differently. It was not my browser cache, 'cause I
> reloaded the URL. It was not the loadbalancer RAM Cache as the
> loadbalancer RAM Cache is disabled. Didn't find anything in the logs. I
> wanted to know your thoughts.
>
> Thanks.
>
> -Shanti
>
Apache HTTPD modules do not influence how Apache Tomcat runs.
So now I have a little better picture of what's running:
Browser --> Apache HTTPD --> (multiple) Tomcats
I don't have any idea what version of anything you're running.
I'm going to guess (again) that you're using mod_jk and a load balancer
configuration. However you could be using mod_proxy and
mod_proxy_balancer. I don't know.
If the 500 Internal Server Error is from Apache HTTPD, then it's an
Apache HTTPD configuration issue and belongs on the Apache HTTPD mailing
list.
Are you running mod_security in production? I've seen some mod_security
Internal Server Error issues when posting WAR files through Apache HTTPD
/ mod_jk / Tomcat. The log file for that should be where Apache HTTPD
logs things.
If the Internal Server Error comes from Tomcat, then it's most likely
your application. You can check the log files in CATALINA_BASE/logs.
There should be information in those logs.
If you are running mod_security on a front-end Apache HTTPD server and
want to quickly check, you could add the following to an Apache HTTPD
server virtual host to turn it off (temporarily).
<IfModule security2_module>
SecRuleEngine Off
</IfModule>
If that cures your problem, then you should take things up with the
mod_security mailing list.
Of course, this is all a guess (I'll put your crystal ball back in a
second, Pid). More information is needed:
Versions of:
Apache HTTPD
Apache Tomcat
Java
Operating system and version
Method you use to connect Apache HTTPD to Apache Tomcat
Text of the Internal Server Error. If you've not configured an error
page for the 500 error, then you'll see a stack trace if it's from Tomcat.
Without all of this information (at a minimum), the best anyone can do
is guess.
. . . . just my two cents
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org