You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ofer Israeli <of...@checkpoint.com> on 2012/02/22 20:03:26 UTC
Tomcat with mod_jk becomes irresponsive after working for awhile
All,
I have a deployment on Windows 2003 Server using Apache 2.2.16, Tomcat 5.5.30 and mod_jk 1.2.31. Apache and Tomcat reside on the same machine. The communication is fine for days / weeks, but at some point all requests to the server (to Apache) that use mod_jk return HTTP 503.
The error that shows up in mod_jk.log is:
[Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_send_request::jk_ajp_common.c (1610): (all) failed sending request on a fresh connection (recoverable), socket -1 (errno=54)
[Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_service::jk_ajp_common.c (2562): (all) connecting to tomcat failed.
I've done a packet capture and I see that Apache initiates the 3-way handshake with Tomcat which looks fine and immediately after the connection is established, Tomcat sends a FIN packet to Apache whereby both sides gracefully close the connection.
Have any of you encountered this before?
Do you have any idea what debug logs I can turn on to get further information without bringing down the server so that we don't loose this situation?
Many thanks,
Ofer
Re: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by chris derham <ch...@derham.me.uk>.
>
> http://wiki.apache.org/tomcat/**HowTo#How_do_I_obtain_a_**
>>> thread_dump_of_my_running_**webapp_.3F<http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F>
>>> for more info, to get a thread dump of tomcat.
>>>
>> I have seen that page but actually can't use the //MS// option as
>> Tomcat is already running and in this bad state that I want to catch
>> without restarting the service. Is there some way to gather this
>> information without a restart?
>>
> If you look at that article, the paragraph above where is mentions //MS//
option, mentions using sendSignal to generate a thread dump. We've used
that in the past and it allows generation of a thread dump on an already
running process with no restart
Chris
Re: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by chris derham <ch...@derham.me.uk>.
Ofer,
Look at the articles in the link already given -
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F-
find the bit about sendsignal. Download this program, and run it. This
provides the windows equivalent of sending -3 to a *nix java process, e.g.
encourages it to provide a thread dump. I have used in the past on windows
with great success. Requires no changes to get it to run, e.g. you do not
have to restart a process or configure it
HTH
Chris
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Felix Schumacher wrote:
> Am 23.02.2012 19:50, schrieb Ofer Israeli:
>> Felix Schumacher wrote:
>>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>>> Caldarale, Charles R wrote:
>>>>>> From: Ofer Israeli [mailto:oferi@checkpoint.com]
>>>>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after
>>>>>> working for awhile
>>>>>
>>>>>> I am in the situation where the server is giving 503s for all
>>>>>> incoming requests, so I did a capture now to see what the
>>>>>> situation is and as mentioned above I see Tomcat FINing the
>>>>>> connections right away before mod_jk sends any content
>>>>>
>>>>> You might want to take a thread dump of Tomcat and see just what
>>>>> is going on. If all the threads are busy (or stuck), you may well
>>>>> have an application problem causing the 503.
>>>>>
>>>>> - Chuck
>>>>>
>>>> Hi Chuck,
>>>>
>>>> I've taken a thread dump and haven't found anything that looks
>>>> suspicious. What baffles me is that TaskManager shows that I have
>>>> 580 threads in the Tomcat process (which makes sense since I use
>>>> 500 worker threads for requests), but the thread dump only shows a
>>>> small portion of these. Below is the dump, I'll be happy if you
>>>> can give me some insight on this.
>>>>
>>>> 2012-02-23 20:08:16
>>>> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode):
>>>>
>>>> "Low Memory Detector" daemon prio=6 tid=0x16bc7400 nid=0x2b38
>>>> runnable [0x00000000] java.lang.Thread.State: RUNNABLE ...
>>>>
>>>> "main" prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native
>>>> Method) at
>>>>
>>>>
>>>> sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine
>>>> .java:82) at
>>>>
>>>>
>>>> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtual
>>>> Machine.java:195) at
>>>>
>>>>
>>>> sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156)
>>>> at sun.tools.jstack.JStack.runThreadDump(JStack.java:159)
>>>> at sun.tools.jstack.JStack.main(JStack.java:94)
>>> This is not a thread dump of tomcat, but rather jstack. Look at
>>>
>>> http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_
>>> my_running_webapp_.3F for more info, to get a thread dump of tomcat.
>>>
>>> Regards
>>> Felix
>>
>> Hi Felix,
>>
>> I have seen that page but actually can't use the //MS// option as
>> Tomcat is already running and in this bad state that I want to catch
>> without restarting the service. Is there some way to gather this
>> information without a restart?
>> From what I understood jstack is supposed to give me all the JVM
>> threads - am I missing something, can you elaborate on what it is I
>> see there and why there are portions missing? Hi Ofer,
>
> I haven't used tomcat under windows, so I might be a bad adviser.
> Maybe you could use a tool like visualvm or jconsole from the jdk to
> attach to the running tomcat jvm and have a look at the threads.
>
> We will probably have to wait for windows users to help you.
>
> Regards
> Felix
Hi all,
So far I haven't found any way to get a thread dump of Tomcat running as a service on Windows 2003 Server without restarting the service. Does anyone know how this can be done?
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 23.02.2012 19:50, schrieb Ofer Israeli:
> Felix Schumacher wrote:
>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>> Caldarale, Charles R wrote:
>>>>> From: Ofer Israeli [mailto:oferi@checkpoint.com]
>>>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after
>>>>> working
>>>>> for awhile
>>>>
>>>>> I am in the situation where the server is giving 503s for all
>>>>> incoming requests, so I did a capture now to see what the
>>>>> situation
>>>>> is and as mentioned above I see Tomcat FINing the connections
>>>>> right
>>>>> away before mod_jk sends any content
>>>>
>>>> You might want to take a thread dump of Tomcat and see just what
>>>> is
>>>> going on. If all the threads are busy (or stuck), you may well
>>>> have
>>>> an application problem causing the 503.
>>>>
>>>> - Chuck
>>>>
>>> Hi Chuck,
>>>
>>> I've taken a thread dump and haven't found anything that looks
>>> suspicious. What baffles me is that TaskManager shows that I have
>>> 580 threads in the Tomcat process (which makes sense since I use
>>> 500
>>> worker threads for requests), but the thread dump only shows a
>>> small
>>> portion of these. Below is the dump, I'll be happy if you can give
>>> me some insight on this.
>>>
>>> 2012-02-23 20:08:16
>>> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode):
>>>
>>> "Low Memory Detector" daemon prio=6 tid=0x16bc7400 nid=0x2b38
>>> runnable [0x00000000] java.lang.Thread.State: RUNNABLE
>> ...
>>>
>>> "main" prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000]
>>> java.lang.Thread.State: RUNNABLE
>>> at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native
>>> Method) at
>>>
>>>
>>> sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82)
>>> at
>>>
>>>
>>> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195)
>>> at
>>>
>>>
>>> sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156)
>>> at sun.tools.jstack.JStack.runThreadDump(JStack.java:159)
>>> at sun.tools.jstack.JStack.main(JStack.java:94)
>> This is not a thread dump of tomcat, but rather jstack. Look at
>>
>> http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
>> for more info, to get a thread dump of tomcat.
>>
>> Regards
>> Felix
>
> Hi Felix,
>
> I have seen that page but actually can't use the //MS// option as
> Tomcat is already running and in this bad state that I want to catch
> without restarting the service. Is there some way to gather this
> information without a restart?
> From what I understood jstack is supposed to give me all the JVM
> threads - am I missing something, can you elaborate on what it is I
> see there and why there are portions missing?
Hi Ofer,
I haven't used tomcat under windows, so I might be a bad adviser. Maybe
you could use a tool like visualvm or jconsole from the jdk to attach to
the running tomcat jvm and have a look at the threads.
We will probably have to wait for windows users to help you.
Regards
Felix
>
> Thanks,
> OFer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Felix Schumacher wrote:
> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>> Caldarale, Charles R wrote:
>>>> From: Ofer Israeli [mailto:oferi@checkpoint.com]
>>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>>>> for awhile
>>>
>>>> I am in the situation where the server is giving 503s for all
>>>> incoming requests, so I did a capture now to see what the situation
>>>> is and as mentioned above I see Tomcat FINing the connections right
>>>> away before mod_jk sends any content
>>>
>>> You might want to take a thread dump of Tomcat and see just what is
>>> going on. If all the threads are busy (or stuck), you may well have
>>> an application problem causing the 503.
>>>
>>> - Chuck
>>>
>> Hi Chuck,
>>
>> I've taken a thread dump and haven't found anything that looks
>> suspicious. What baffles me is that TaskManager shows that I have
>> 580 threads in the Tomcat process (which makes sense since I use 500
>> worker threads for requests), but the thread dump only shows a small
>> portion of these. Below is the dump, I'll be happy if you can give
>> me some insight on this.
>>
>> 2012-02-23 20:08:16
>> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode):
>>
>> "Low Memory Detector" daemon prio=6 tid=0x16bc7400 nid=0x2b38
>> runnable [0x00000000] java.lang.Thread.State: RUNNABLE
> ...
>>
>> "main" prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native
>> Method) at
>>
>> sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82)
>> at
>>
>> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195)
>> at
>>
>> sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156)
>> at sun.tools.jstack.JStack.runThreadDump(JStack.java:159)
>> at sun.tools.jstack.JStack.main(JStack.java:94)
> This is not a thread dump of tomcat, but rather jstack. Look at
> http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
> for more info, to get a thread dump of tomcat.
>
> Regards
> Felix
Hi Felix,
I have seen that page but actually can't use the //MS// option as Tomcat is already running and in this bad state that I want to catch without restarting the service. Is there some way to gather this information without a restart?
>From what I understood jstack is supposed to give me all the JVM threads - am I missing something, can you elaborate on what it is I see there and why there are portions missing?
Thanks,
OFer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 23.02.2012 19:32, schrieb Ofer Israeli:
> Caldarale, Charles R wrote:
>>> From: Ofer Israeli [mailto:oferi@checkpoint.com]
>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>>> for awhile
>>
>>> I am in the situation where the server is giving 503s for all
>>> incoming requests, so I did a capture now to see what the situation
>>> is and as mentioned above I see Tomcat FINing the connections right
>>> away before mod_jk sends any content
>>
>> You might want to take a thread dump of Tomcat and see just what is
>> going on. If all the threads are busy (or stuck), you may well have
>> an application problem causing the 503.
>>
>> - Chuck
>>
> Hi Chuck,
>
> I've taken a thread dump and haven't found anything that looks
> suspicious. What baffles me is that TaskManager shows that I have
> 580
> threads in the Tomcat process (which makes sense since I use 500
> worker threads for requests), but the thread dump only shows a small
> portion of these. Below is the dump, I'll be happy if you can give
> me
> some insight on this.
>
> 2012-02-23 20:08:16
> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode):
>
> "Low Memory Detector" daemon prio=6 tid=0x16bc7400 nid=0x2b38
> runnable [0x00000000]
> java.lang.Thread.State: RUNNABLE
...
>
> "main" prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000]
> java.lang.Thread.State: RUNNABLE
> at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method)
> at
>
> sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82)
> at
>
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195)
> at
>
> sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156)
> at sun.tools.jstack.JStack.runThreadDump(JStack.java:159)
> at sun.tools.jstack.JStack.main(JStack.java:94)
This is not a thread dump of tomcat, but rather jstack. Look at
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
for more info, to get a thread dump of tomcat.
Regards
Felix
>
> Thanks,
> Ofer
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Caldarale, Charles R wrote:
>> From: Ofer Israeli [mailto:oferi@checkpoint.com]
>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>> for awhile
>
>> I am in the situation where the server is giving 503s for all
>> incoming requests, so I did a capture now to see what the situation
>> is and as mentioned above I see Tomcat FINing the connections right
>> away before mod_jk sends any content
>
> You might want to take a thread dump of Tomcat and see just what is
> going on. If all the threads are busy (or stuck), you may well have
> an application problem causing the 503.
>
> - Chuck
>
Hi Chuck,
I've taken a thread dump and haven't found anything that looks suspicious. What baffles me is that TaskManager shows that I have 580 threads in the Tomcat process (which makes sense since I use 500 worker threads for requests), but the thread dump only shows a small portion of these. Below is the dump, I'll be happy if you can give me some insight on this.
2012-02-23 20:08:16
Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode):
"Low Memory Detector" daemon prio=6 tid=0x16bc7400 nid=0x2b38 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x16bbe400 nid=0x2828 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x16bbcc00 nid=0xc4c runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x16bbb800 nid=0x1378 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x16baac00 nid=0x2f4c in Object.wait() [0x16d1f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x029f1158> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x029f1158> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x16ba6000 nid=0x2c5c in Object.wait() [0x16ccf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x029f1058> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x029f1058> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000]
java.lang.Thread.State: RUNNABLE
at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method)
at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82)
at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195)
at sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156)
at sun.tools.jstack.JStack.runThreadDump(JStack.java:159)
at sun.tools.jstack.JStack.main(JStack.java:94)
"VM Thread" prio=10 tid=0x16ba2400 nid=0x320 runnable
"VM Periodic Task Thread" prio=10 tid=0x16bcac00 nid=0x2cb4 waiting on condition
JNI global references: 913
Heap
def new generation total 4928K, used 756K [0x029f0000, 0x02f40000, 0x07f40000)
eden space 4416K, 17% used [0x029f0000, 0x02aad0a0, 0x02e40000)
from space 512K, 0% used [0x02e40000, 0x02e40000, 0x02ec0000)
to space 512K, 0% used [0x02ec0000, 0x02ec0000, 0x02f40000)
tenured generation total 10944K, used 0K [0x07f40000, 0x089f0000, 0x129f0000)
the space 10944K, 0% used [0x07f40000, 0x07f40000, 0x07f40200, 0x089f0000)
compacting perm gen total 12288K, used 2379K [0x129f0000, 0x135f0000, 0x169f0000)
the space 12288K, 19% used [0x129f0000, 0x12c42d68, 0x12c42e00, 0x135f0000)
No shared spaces configured.
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Ofer Israeli [mailto:oferi@checkpoint.com]
> Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile
> I am in the situation where the server is giving 503s for
> all incoming requests, so I did a capture now to see what
> the situation is and as mentioned above I see Tomcat FINing
> the connections right away before mod_jk sends any content
You might want to take a thread dump of Tomcat and see just what is going on. If all the threads are busy (or stuck), you may well have an application problem causing the 503.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ofer,
>
> On 2/27/12 6:11 AM, Ofer Israeli wrote:
>> Christopher Schultz wrote:
>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>
>>> Ofer,
>>>
>>> On 2/23/12 1:57 PM, Ofer Israeli wrote:
>>>> Felix Schumacher wrote:
>>>>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>>>>> Caldarale, Charles R wrote:
>>>>>>>> From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk
>>>>>>>> becomes irresponsive after working for awhile
>>>>>>>
>>>>>>>> You might want to take a thread dump of Tomcat and see just
>>>>>>>> what is going on.
>>>>>>>
>>>>>>> Also look in the Tomcat logs (there are several) to see if
>>>>>>> errors are being reported there.
>>>>>>>
>>>>>>> - Chuck
>>>>>>
>>>>>> Which logs are you referring to? I've turned on org.apache logs
>>>>>> via log4j and haven't found anything related.
>>>>> Normally they can be found under $CATALINA_BASE/logs. The most
>>>>> important ones are usually catalina.out and localhost*log.
>>>>>
>>>>> Regards Felix
>>>>
>>>> I don't see these logs in the server. Could it be that this is due
>>>> to Tomcat being embedded?
>>>
>>> Wait, you are running Tomcat as an embedded service? That seems liek
>>> it would be reasonable information to include in your original post.
>>> Why did it take 6 posts to get this information out of you?
>>>
>>> If you are embedding Tomcat, you should have control over the
>>> logging setup. Where are you redirecting stdout?
>>>
>>> - -chris
>>
>> Hi Chris,
>>
>> Guess it's my bad for not including this information in previous
>> posts - I didn't know if this is related to the problem at hand or
>> not. On startup we redirect stdout and stderr to the same log file
>> that we log other information.
>
> How exactly do you do that?
>
> - -chris
In our code, on startup, we close stdout and stderr via the System class. Then we set the output and error streams to a propietary class which uses Log4J to write out to our file.
/Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Ofer,
On 2/27/12 6:11 AM, Ofer Israeli wrote:
> Christopher Schultz wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>
>> Ofer,
>>
>> On 2/23/12 1:57 PM, Ofer Israeli wrote:
>>> Felix Schumacher wrote:
>>>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>>>> Caldarale, Charles R wrote:
>>>>>>> From: Caldarale, Charles R Subject: RE: Tomcat with
>>>>>>> mod_jk becomes irresponsive after working for awhile
>>>>>>
>>>>>>> You might want to take a thread dump of Tomcat and see
>>>>>>> just what is going on.
>>>>>>
>>>>>> Also look in the Tomcat logs (there are several) to see
>>>>>> if errors are being reported there.
>>>>>>
>>>>>> - Chuck
>>>>>
>>>>> Which logs are you referring to? I've turned on org.apache
>>>>> logs via log4j and haven't found anything related.
>>>> Normally they can be found under $CATALINA_BASE/logs. The
>>>> most important ones are usually catalina.out and
>>>> localhost*log.
>>>>
>>>> Regards Felix
>>>
>>> I don't see these logs in the server. Could it be that this is
>>> due to Tomcat being embedded?
>>
>> Wait, you are running Tomcat as an embedded service? That seems
>> liek it would be reasonable information to include in your
>> original post. Why did it take 6 posts to get this information
>> out of you?
>>
>> If you are embedding Tomcat, you should have control over the
>> logging setup. Where are you redirecting stdout?
>>
>> - -chris
>
> Hi Chris,
>
> Guess it's my bad for not including this information in previous
> posts - I didn't know if this is related to the problem at hand or
> not. On startup we redirect stdout and stderr to the same log file
> that we log other information.
How exactly do you do that?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9LgnIACgkQ9CaO5/Lv0PBbLwCcCBXVIbmkQp3XkY5v3tZkShsl
F4sAoJEeIrN2ZB7/fPwB+r6NuVWVOOEj
=uaEk
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ofer,
>
> On 2/23/12 1:57 PM, Ofer Israeli wrote:
>> Felix Schumacher wrote:
>>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>>> Caldarale, Charles R wrote:
>>>>>> From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk
>>>>>> becomes irresponsive after working for awhile
>>>>>
>>>>>> You might want to take a thread dump of Tomcat and see just what
>>>>>> is going on.
>>>>>
>>>>> Also look in the Tomcat logs (there are several) to see if errors
>>>>> are being reported there.
>>>>>
>>>>> - Chuck
>>>>
>>>> Which logs are you referring to? I've turned on org.apache logs
>>>> via log4j and haven't found anything related.
>>> Normally they can be found under $CATALINA_BASE/logs. The most
>>> important ones are usually catalina.out and localhost*log.
>>>
>>> Regards Felix
>>
>> I don't see these logs in the server. Could it be that this is due
>> to Tomcat being embedded?
>
> Wait, you are running Tomcat as an embedded service? That seems liek
> it would be reasonable information to include in your original post.
> Why did it take 6 posts to get this information out of you?
>
> If you are embedding Tomcat, you should have control over the logging
> setup. Where are you redirecting stdout?
>
> - -chris
Hi Chris,
Guess it's my bad for not including this information in previous posts - I didn't know if this is related to the problem at hand or not. On startup we redirect stdout and stderr to the same log file that we log other information.
Cheers,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Ofer,
On 2/23/12 1:57 PM, Ofer Israeli wrote:
> Felix Schumacher wrote:
>> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>>> Caldarale, Charles R wrote:
>>>>> From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk
>>>>> becomes irresponsive after working for awhile
>>>>
>>>>> You might want to take a thread dump of Tomcat and see just
>>>>> what is going on.
>>>>
>>>> Also look in the Tomcat logs (there are several) to see if
>>>> errors are being reported there.
>>>>
>>>> - Chuck
>>>
>>> Which logs are you referring to? I've turned on org.apache
>>> logs via log4j and haven't found anything related.
>> Normally they can be found under $CATALINA_BASE/logs. The most
>> important ones are usually catalina.out and localhost*log.
>>
>> Regards Felix
>
> I don't see these logs in the server. Could it be that this is due
> to Tomcat being embedded?
Wait, you are running Tomcat as an embedded service? That seems liek
it would be reasonable information to include in your original post.
Why did it take 6 posts to get this information out of you?
If you are embedding Tomcat, you should have control over the logging
setup. Where are you redirecting stdout?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9K7o8ACgkQ9CaO5/Lv0PCuKwCeLiVAf6zxcJETxmqzDPV5DItm
nsoAn15lcniYHs/mKKrWzVxGfHVxsPnA
=bIJY
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Felix Schumacher wrote:
> Am 23.02.2012 19:32, schrieb Ofer Israeli:
>> Caldarale, Charles R wrote:
>>>> From: Caldarale, Charles R
>>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>>>> for awhile
>>>
>>>> You might want to take a thread dump of Tomcat and see just what is
>>>> going on.
>>>
>>> Also look in the Tomcat logs (there are several) to see if errors
>>> are being reported there.
>>>
>>> - Chuck
>>
>> Which logs are you referring to? I've turned on org.apache logs via
>> log4j and haven't found anything related.
> Normally they can be found under $CATALINA_BASE/logs. The most
> important ones are usually catalina.out and localhost*log.
>
> Regards
> Felix
I don't see these logs in the server. Could it be that this is due to Tomcat being embedded?
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 23.02.2012 19:32, schrieb Ofer Israeli:
> Caldarale, Charles R wrote:
>>> From: Caldarale, Charles R
>>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>>> for awhile
>>
>>> You might want to take a thread dump of Tomcat and see just what is
>>> going on.
>>
>> Also look in the Tomcat logs (there are several) to see if errors
>> are
>> being reported there.
>>
>> - Chuck
>
> Which logs are you referring to? I've turned on org.apache logs via
> log4j and haven't found anything related.
Normally they can be found under $CATALINA_BASE/logs. The most
important
ones are usually catalina.out and localhost*log.
Regards
Felix
>
> Thanks,
> Ofer
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
Caldarale, Charles R wrote:
>> From: Caldarale, Charles R
>> Subject: RE: Tomcat with mod_jk becomes irresponsive after working
>> for awhile
>
>> You might want to take a thread dump of Tomcat and see just what is
>> going on.
>
> Also look in the Tomcat logs (there are several) to see if errors are
> being reported there.
>
> - Chuck
Which logs are you referring to? I've turned on org.apache logs via log4j and haven't found anything related.
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Caldarale, Charles R
> Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile
> You might want to take a thread dump of Tomcat and see just
> what is going on.
Also look in the Tomcat logs (there are several) to see if errors are being reported there.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by Ofer Israeli <of...@checkpoint.com>.
André Warnier wrote:
> Ofer Israeli wrote:
>> All,
>>
>> I have a deployment on Windows 2003 Server using Apache 2.2.16,
>> Tomcat 5.5.30 and mod_jk 1.2.31. Apache and Tomcat reside on the
>> same machine. The communication is fine for days / weeks, but at
>> some point all requests to the server (to Apache) that use mod_jk
>> return HTTP 503. The error that shows up in mod_jk.log is: [Mon Feb
>> 20 12:07:01.464 2012] [7280:6912] [error]
>> ajp_send_request::jk_ajp_common.c (1610): (all) failed sending
>> request on a fresh connection (recoverable), socket -1 (errno=54)
>> [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error]
>> ajp_service::jk_ajp_common.c (2562): (all) connecting to tomcat
>> failed.
>>
>> I've done a packet capture and I see that Apache initiates the 3-way
>> handshake with Tomcat which looks fine and immediately after the
>> connection is established, Tomcat sends a FIN packet to Apache
>> whereby both sides gracefully close the connection.
>>
>> Have any of you encountered this before?
>>
>> Do you have any idea what debug logs I can turn on to get further
>> information without bringing down the server so that we don't loose
>> this situation?
>>
>>
> Hi.
> It sounds unlikely that you would have done a permanent packet
> capture during days/weeks, just waiting for this to happen. So what
> made you decide when to start your packet capture, or how do you
> force the problem to happen when you have started your packet capture
> ?
>
Hi Andre,
I do not have a permenant packet capture, but rather now I am in the situation where the server is giving 503s for all incoming requests, so I did a capture now to see what the situation is and as mentioned above I see Tomcat FINing the connections right away before mod_jk sends any content (putting aside the connection establishment).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat with mod_jk becomes irresponsive after working for awhile
Posted by André Warnier <aw...@ice-sa.com>.
Ofer Israeli wrote:
> All,
>
> I have a deployment on Windows 2003 Server using Apache 2.2.16, Tomcat 5.5.30 and mod_jk 1.2.31. Apache and Tomcat reside on the same machine. The communication is fine for days / weeks, but at some point all requests to the server (to Apache) that use mod_jk return HTTP 503.
> The error that shows up in mod_jk.log is:
> [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_send_request::jk_ajp_common.c (1610): (all) failed sending request on a fresh connection (recoverable), socket -1 (errno=54)
> [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_service::jk_ajp_common.c (2562): (all) connecting to tomcat failed.
>
> I've done a packet capture and I see that Apache initiates the 3-way handshake with Tomcat which looks fine and immediately after the connection is established, Tomcat sends a FIN packet to Apache whereby both sides gracefully close the connection.
>
> Have any of you encountered this before?
>
> Do you have any idea what debug logs I can turn on to get further information without bringing down the server so that we don't loose this situation?
>
>
Hi.
It sounds unlikely that you would have done a permanent packet capture during days/weeks,
just waiting for this to happen. So what made you decide when to start your packet
capture, or how do you force the problem to happen when you have started your packet capture ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org