You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ofer Israeli <of...@checkpoint.com> on 2012/04/04 18:02:17 UTC
Bug in Tomcat AJP Connector?
Hi all,
We have recently witnessed a strange situation. Our Tomcat (6.0.35) listens on 2 ports: 8080 and 8009, handling HTTP and AJP respectively. At some point in time we found that Apache is replying HTTP 503s to all clients and a look at netstat showed that indeed Apache could not communicate with Tomcat - there were many connections in state SYN_SENT, that were never replied. A test of attempting to connect to Tomcat's 8009 port turned out to give the same result, so the issue was obviously in Tomcat and not Apache. But netstat also showed that Tomcat was listening on port 8009. After digging into the logs, we found that there was an Out of Memory exception in the JK's accept() sequence and once this exception was caught there was a message of "terminating thread".
So although the exception is caught and handled by killing the thread (isn't that a little drastic?), the socket isn't closed beforehand and thus the OS thinks that the port is still fine (in LISTENING state), as although the thread is dead, the process is alive. Has anyone encountered this or knows if this bug is known?
Thanks,
Ofer
RE: Bug in Tomcat AJP Connector?
Posted by Ofer Israeli <of...@checkpoint.com>.
2012/4/6 Konstantin Kolinko <kn...@gmail.com>:
> 2012/4/6 Ofer Israeli <of...@checkpoint.com>:
> > Hi Konstantin,
> >
> > I agree regarding the OOM bringing TC to a state where it must be
> restored, but my point remains: if there is code that handles catching this
> exception and terminating the thread, why not terminate gracefully by
> closing the listening socket before killing the thread?
> >
>
> 1. In theory one can configure several acceptor threads. Though it is not
> recommended.
> So one has to check the count before closing the socket.
This sounds like the reason for not closing the socket. Is this a TC configuration?
> 2. The socket will be closed when Tomcat shuts down.
> Just closing the socket - what will it improve?
>
> Anyway you will need to shutdown this Tomcat instance before you can start
> the second one, because you'll need to free the shutdown port etc.
The point is to do proper cleanup when terminating a thread. One good reason for this is the strange situation the system is left in where you can see that the Tomcat process is listening on port 8009 via netstat, yet it actually isn't listening on that port as the acceptor thread has died. This makes it very difficult to troubleshoot as the system is showing "false" information.
> > if there is code that handles catching this exception and terminating
> > the thread
>
> 3. There is no such special code. The thread terminates by itself, because its
> run() throws an exception and thus finishes execution.
>
> There is ThreadGroup.uncaughtException() method in JRE which by default
> prints the exception to stdout.
I see specific Tomcat code that catches Throwable, prints out this exception and then terminates the thread in ThreadPool:
ThreadPool.log.error(sm.getString
("threadpool.thread_error", t, toRun.toString()));
By the way, this is not relevant only for OOME, but also for any other exception that was not handled and reached this code.
> If you consider to propose a patch - feel free to file an enhancement request
> in Bugzilla and attach it.
In accordance with your comments on the above, I'll file a patch if it looks in place.
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/4/6 Ofer Israeli <of...@checkpoint.com>:
> Hi Konstantin,
>
> I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread?
>
1. In theory one can configure several acceptor threads. Though it is
not recommended.
So one has to check the count before closing the socket.
2. The socket will be closed when Tomcat shuts down.
Just closing the socket - what will it improve?
Anyway you will need to shutdown this Tomcat instance before you can
start the second one, because you'll need to free the shutdown port
etc.
> if there is code that handles catching this exception and terminating the thread
3. There is no such special code. The thread terminates by itself,
because its run() throws an exception and thus finishes execution.
There is ThreadGroup.uncaughtException() method in JRE which by
default prints the exception to stdout.
If you consider to propose a patch - feel free to file an enhancement
request in Bugzilla and attach it.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Pid <pi...@pidster.com>.
On 09/04/2012 09:18, Ofer Israeli wrote:
> On 08/04/2012 23:14, Stefan Mayr <st...@mayr-stefan.de> wrote:
>> Am 08.04.2012 18:41, schrieb Ofer Israeli:
>>> 2012/4/6 Pid<pi...@pidster.com>:
>>>> On 05/04/2012 22:17, Ofer Israeli wrote:
>>>>> Y
>>>>>
>>>>> On 5 באפר 2012, at 18:58, "Konstantin
>>>>> Kolinko"<kn...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> 2012/4/5 Ofer Israeli<of...@checkpoint.com>:
>>>>>>> Mark Thomas wrote:
>>>>>>>> On 04/04/2012 17:02, Ofer Israeli wrote:
>>>>>>>>
>>>>>>>> Once you have an OOME all bets are off. The JVM needs to be
>>>> restarted.
>>>>>>>> There is no guarantee of reliable operation after an OOME.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>
>>>>>>> Hi Mark,
>>>>>>> I agree that there in such a situation the JVM should be
>>>>>>> restarted, but it
>>>> isn't restarted by Tomcat. On the other hand, Tomcat does take some
>>>> precautious actions and kills the accepting thread, but in such a
>>>> case it should also close the socket that thread is listening on
>>>> otherwise it is leaving garbage around after the thread's death.
>>>>>>> Do you see any reason as not to close the listening socket?
>>>>>>>
>>>>>>
>>>>>> 1. Tomcat does not start JVM thus it cannot restart it.
>>>>>>
>>>>>> You need some external tool or script or admin to perform
>>>>>> monitoring and (re)starts.
>>>>>>
>>>>>> 2. OOM can happen at a random place. Once it happens, it is likely
>>>>>> that other places will also start to fail randomly. It is also
>>>>>> likely that your attempts to recover will fail as well.
>>>>>>
>>>>>> Mark already mentioned it: "all bets are off".
>>>>>>
>>>>>> Best regards,
>>>>>> Konstantin Kolinko
>>>>>>
>>>>> Hi Konstantin,
>>>>>
>>>>> I agree regarding the OOM bringing TC to a state where it must be
>>>> restored, but my point remains: if there is code that handles
>>>> catching this exception and terminating the thread, why not terminate
>>>> gracefully by closing the listening socket before killing the thread?
>>>>
>>>> And your point has been answered. After an OOM the JVM is in an
>>>> unknown, unsafe state so a restart MUST occur to restore service.
>>>>
>>>> Closing a socket gracefully after an OOM is a bit like trying to shut
>>>> one of the portholes on the Titanic, shortly after hearing a large crashing
>> sound.
>>>>
>>>>
>>>> There's only one place I know of where Tomcat attempts to interact
>>>> with OOM conditions and this is not one of them, so I don't believe
>>>> it's safe to say that Tomcat is deliberately handling this exception.
>>>>
>>>> NB an OOM is an Error, not an Exception - it is a subclass of
>>>> VirtualMachineError, which is thrown to indicate that the Java
>>>> Virtual Machine is broken or has run out of resources necessary for
>>>> it to continue operating.
>>>>
>>>> An Error is a subclass of Throwable that indicates serious problems
>>>> that a reasonable application should not try to catch.
>>>> </end-quote>
>>>>
>>>> If anything, the locations where Tomcat catches a Throwable should be
>>>> modified so it does *not* catch Errors, rather than continuing to do
>>>> so and then attempting a trivial tidy-up.
>>>>
>>>>
>>>> p
>>>
>>> Thanks for your input - you're right regarding the error and the fact that
>> Tomcat is indeed catching a Throwable and not an Exception. I assume that if
>> the Throwable were not caught, then the thread would die in any case.
>> Although stated before that Tomcat could not kill itself in such a situation, I
>> still wonder if it would be possible to do so. Or taking a different perspective
>> on this: if the JVM specification is such that it cannot be trusted to continue
>> running after an OOM, then why does it not kill itself or restart itself?
>>>
>>
>> I guess you can do this with some vendor specific JVM arguments as
>> SUNs/Oracles -XX:OnOutOfMemoryError:
>> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-
>> 140102.html
>>
>> Different findings like "kill -9 %p" let me suspect that you can use %p as a
>> variable for your current pid. With that you can either kill your current
>> instance and let your monitoring handle the rest or try to initiate the restart
>> by yourself.
>>
>> Give it a try
>>
>> Stefan
>>
> Thanks Stefan - will look into this option.
Be careful using that option; if you are also producing a heap dump you
should wait until the heap dump has finished writing before stopping the
process, or the heap will be incomplete & therefore useless.
p
--
[key:62590808]
RE: Bug in Tomcat AJP Connector?
Posted by Ofer Israeli <of...@checkpoint.com>.
On 08/04/2012 23:14, Stefan Mayr <st...@mayr-stefan.de> wrote:
> Am 08.04.2012 18:41, schrieb Ofer Israeli:
> > 2012/4/6 Pid<pi...@pidster.com>:
> >> On 05/04/2012 22:17, Ofer Israeli wrote:
> >>> Y
> >>>
> >>> On 5 באפר 2012, at 18:58, "Konstantin
> >>> Kolinko"<kn...@gmail.com>
> >> wrote:
> >>>
> >>>> 2012/4/5 Ofer Israeli<of...@checkpoint.com>:
> >>>>> Mark Thomas wrote:
> >>>>>> On 04/04/2012 17:02, Ofer Israeli wrote:
> >>>>>>
> >>>>>> Once you have an OOME all bets are off. The JVM needs to be
> >> restarted.
> >>>>>> There is no guarantee of reliable operation after an OOME.
> >>>>>>
> >>>>>> Mark
> >>>>>
> >>>>> Hi Mark,
> >>>>> I agree that there in such a situation the JVM should be
> >>>>> restarted, but it
> >> isn't restarted by Tomcat. On the other hand, Tomcat does take some
> >> precautious actions and kills the accepting thread, but in such a
> >> case it should also close the socket that thread is listening on
> >> otherwise it is leaving garbage around after the thread's death.
> >>>>> Do you see any reason as not to close the listening socket?
> >>>>>
> >>>>
> >>>> 1. Tomcat does not start JVM thus it cannot restart it.
> >>>>
> >>>> You need some external tool or script or admin to perform
> >>>> monitoring and (re)starts.
> >>>>
> >>>> 2. OOM can happen at a random place. Once it happens, it is likely
> >>>> that other places will also start to fail randomly. It is also
> >>>> likely that your attempts to recover will fail as well.
> >>>>
> >>>> Mark already mentioned it: "all bets are off".
> >>>>
> >>>> Best regards,
> >>>> Konstantin Kolinko
> >>>>
> >>> Hi Konstantin,
> >>>
> >>> I agree regarding the OOM bringing TC to a state where it must be
> >> restored, but my point remains: if there is code that handles
> >> catching this exception and terminating the thread, why not terminate
> >> gracefully by closing the listening socket before killing the thread?
> >>
> >> And your point has been answered. After an OOM the JVM is in an
> >> unknown, unsafe state so a restart MUST occur to restore service.
> >>
> >> Closing a socket gracefully after an OOM is a bit like trying to shut
> >> one of the portholes on the Titanic, shortly after hearing a large crashing
> sound.
> >>
> >>
> >> There's only one place I know of where Tomcat attempts to interact
> >> with OOM conditions and this is not one of them, so I don't believe
> >> it's safe to say that Tomcat is deliberately handling this exception.
> >>
> >> NB an OOM is an Error, not an Exception - it is a subclass of
> >> VirtualMachineError, which is thrown to indicate that the Java
> >> Virtual Machine is broken or has run out of resources necessary for
> >> it to continue operating.
> >>
> >> An Error is a subclass of Throwable that indicates serious problems
> >> that a reasonable application should not try to catch.
> >> </end-quote>
> >>
> >> If anything, the locations where Tomcat catches a Throwable should be
> >> modified so it does *not* catch Errors, rather than continuing to do
> >> so and then attempting a trivial tidy-up.
> >>
> >>
> >> p
> >
> > Thanks for your input - you're right regarding the error and the fact that
> Tomcat is indeed catching a Throwable and not an Exception. I assume that if
> the Throwable were not caught, then the thread would die in any case.
> Although stated before that Tomcat could not kill itself in such a situation, I
> still wonder if it would be possible to do so. Or taking a different perspective
> on this: if the JVM specification is such that it cannot be trusted to continue
> running after an OOM, then why does it not kill itself or restart itself?
> >
>
> I guess you can do this with some vendor specific JVM arguments as
> SUNs/Oracles -XX:OnOutOfMemoryError:
> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-
> 140102.html
>
> Different findings like "kill -9 %p" let me suspect that you can use %p as a
> variable for your current pid. With that you can either kill your current
> instance and let your monitoring handle the rest or try to initiate the restart
> by yourself.
>
> Give it a try
>
> Stefan
>
Thanks Stefan - will look into this option.
Re: Bug in Tomcat AJP Connector?
Posted by Stefan Mayr <st...@mayr-stefan.de>.
Am 08.04.2012 18:41, schrieb Ofer Israeli:
> 2012/4/6 Pid<pi...@pidster.com>:
>> On 05/04/2012 22:17, Ofer Israeli wrote:
>>> Y
>>>
>>> On 5 באפר 2012, at 18:58, "Konstantin Kolinko"<kn...@gmail.com>
>> wrote:
>>>
>>>> 2012/4/5 Ofer Israeli<of...@checkpoint.com>:
>>>>> Mark Thomas wrote:
>>>>>> On 04/04/2012 17:02, Ofer Israeli wrote:
>>>>>>
>>>>>> Once you have an OOME all bets are off. The JVM needs to be
>> restarted.
>>>>>> There is no guarantee of reliable operation after an OOME.
>>>>>>
>>>>>> Mark
>>>>>
>>>>> Hi Mark,
>>>>> I agree that there in such a situation the JVM should be restarted, but it
>> isn't restarted by Tomcat. On the other hand, Tomcat does take some
>> precautious actions and kills the accepting thread, but in such a case it should
>> also close the socket that thread is listening on otherwise it is leaving garbage
>> around after the thread's death.
>>>>> Do you see any reason as not to close the listening socket?
>>>>>
>>>>
>>>> 1. Tomcat does not start JVM thus it cannot restart it.
>>>>
>>>> You need some external tool or script or admin to perform monitoring
>>>> and (re)starts.
>>>>
>>>> 2. OOM can happen at a random place. Once it happens, it is likely
>>>> that other places will also start to fail randomly. It is also likely
>>>> that your attempts to recover will fail as well.
>>>>
>>>> Mark already mentioned it: "all bets are off".
>>>>
>>>> Best regards,
>>>> Konstantin Kolinko
>>>>
>>> Hi Konstantin,
>>>
>>> I agree regarding the OOM bringing TC to a state where it must be
>> restored, but my point remains: if there is code that handles catching this
>> exception and terminating the thread, why not terminate gracefully by
>> closing the listening socket before killing the thread?
>>
>> And your point has been answered. After an OOM the JVM is in an
>> unknown, unsafe state so a restart MUST occur to restore service.
>>
>> Closing a socket gracefully after an OOM is a bit like trying to shut one of the
>> portholes on the Titanic, shortly after hearing a large crashing sound.
>>
>>
>> There's only one place I know of where Tomcat attempts to interact with
>> OOM conditions and this is not one of them, so I don't believe it's safe to say
>> that Tomcat is deliberately handling this exception.
>>
>> NB an OOM is an Error, not an Exception - it is a subclass of
>> VirtualMachineError, which is thrown to indicate that the Java Virtual
>> Machine is broken or has run out of resources necessary for it to continue
>> operating.
>>
>> An Error is a subclass of Throwable that indicates serious problems that a
>> reasonable application should not try to catch.
>> </end-quote>
>>
>> If anything, the locations where Tomcat catches a Throwable should be
>> modified so it does *not* catch Errors, rather than continuing to do so and
>> then attempting a trivial tidy-up.
>>
>>
>> p
>
> Thanks for your input - you're right regarding the error and the fact that Tomcat is indeed catching a Throwable and not an Exception. I assume that if the Throwable were not caught, then the thread would die in any case. Although stated before that Tomcat could not kill itself in such a situation, I still wonder if it would be possible to do so. Or taking a different perspective on this: if the JVM specification is such that it cannot be trusted to continue running after an OOM, then why does it not kill itself or restart itself?
>
I guess you can do this with some vendor specific JVM arguments as
SUNs/Oracles -XX:OnOutOfMemoryError:
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
Different findings like "kill -9 %p" let me suspect that you can use %p
as a variable for your current pid. With that you can either kill your
current instance and let your monitoring handle the rest or try to
initiate the restart by yourself.
Give it a try
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Bug in Tomcat AJP Connector?
Posted by Ofer Israeli <of...@checkpoint.com>.
2012/4/6 Pid <pi...@pidster.com>:
> On 05/04/2012 22:17, Ofer Israeli wrote:
> > Y
> >
> > On 5 באפר 2012, at 18:58, "Konstantin Kolinko" <kn...@gmail.com>
> wrote:
> >
> >> 2012/4/5 Ofer Israeli <of...@checkpoint.com>:
> >>> Mark Thomas wrote:
> >>>> On 04/04/2012 17:02, Ofer Israeli wrote:
> >>>>
> >>>> Once you have an OOME all bets are off. The JVM needs to be
> restarted.
> >>>> There is no guarantee of reliable operation after an OOME.
> >>>>
> >>>> Mark
> >>>
> >>> Hi Mark,
> >>> I agree that there in such a situation the JVM should be restarted, but it
> isn't restarted by Tomcat. On the other hand, Tomcat does take some
> precautious actions and kills the accepting thread, but in such a case it should
> also close the socket that thread is listening on otherwise it is leaving garbage
> around after the thread's death.
> >>> Do you see any reason as not to close the listening socket?
> >>>
> >>
> >> 1. Tomcat does not start JVM thus it cannot restart it.
> >>
> >> You need some external tool or script or admin to perform monitoring
> >> and (re)starts.
> >>
> >> 2. OOM can happen at a random place. Once it happens, it is likely
> >> that other places will also start to fail randomly. It is also likely
> >> that your attempts to recover will fail as well.
> >>
> >> Mark already mentioned it: "all bets are off".
> >>
> >> Best regards,
> >> Konstantin Kolinko
> >>
> > Hi Konstantin,
> >
> > I agree regarding the OOM bringing TC to a state where it must be
> restored, but my point remains: if there is code that handles catching this
> exception and terminating the thread, why not terminate gracefully by
> closing the listening socket before killing the thread?
>
> And your point has been answered. After an OOM the JVM is in an
> unknown, unsafe state so a restart MUST occur to restore service.
>
> Closing a socket gracefully after an OOM is a bit like trying to shut one of the
> portholes on the Titanic, shortly after hearing a large crashing sound.
>
>
> There's only one place I know of where Tomcat attempts to interact with
> OOM conditions and this is not one of them, so I don't believe it's safe to say
> that Tomcat is deliberately handling this exception.
>
> NB an OOM is an Error, not an Exception - it is a subclass of
> VirtualMachineError, which is thrown to indicate that the Java Virtual
> Machine is broken or has run out of resources necessary for it to continue
> operating.
>
> An Error is a subclass of Throwable that indicates serious problems that a
> reasonable application should not try to catch.
> </end-quote>
>
> If anything, the locations where Tomcat catches a Throwable should be
> modified so it does *not* catch Errors, rather than continuing to do so and
> then attempting a trivial tidy-up.
>
>
> p
Thanks for your input - you're right regarding the error and the fact that Tomcat is indeed catching a Throwable and not an Exception. I assume that if the Throwable were not caught, then the thread would die in any case. Although stated before that Tomcat could not kill itself in such a situation, I still wonder if it would be possible to do so. Or taking a different perspective on this: if the JVM specification is such that it cannot be trusted to continue running after an OOM, then why does it not kill itself or restart itself?
Thanks,
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Pid <pi...@pidster.com>.
On 05/04/2012 22:17, Ofer Israeli wrote:
> Y
>
> On 5 באפר 2012, at 18:58, "Konstantin Kolinko" <kn...@gmail.com> wrote:
>
>> 2012/4/5 Ofer Israeli <of...@checkpoint.com>:
>>> Mark Thomas wrote:
>>>> On 04/04/2012 17:02, Ofer Israeli wrote:
>>>>
>>>> Once you have an OOME all bets are off. The JVM needs to be restarted.
>>>> There is no guarantee of reliable operation after an OOME.
>>>>
>>>> Mark
>>>
>>> Hi Mark,
>>> I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death.
>>> Do you see any reason as not to close the listening socket?
>>>
>>
>> 1. Tomcat does not start JVM thus it cannot restart it.
>>
>> You need some external tool or script or admin to perform monitoring
>> and (re)starts.
>>
>> 2. OOM can happen at a random place. Once it happens, it is likely
>> that other places will also start to fail randomly. It is also likely
>> that your attempts to recover will fail as well.
>>
>> Mark already mentioned it: "all bets are off".
>>
>> Best regards,
>> Konstantin Kolinko
>>
> Hi Konstantin,
>
> I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread?
And your point has been answered. After an OOM the JVM is in an
unknown, unsafe state so a restart MUST occur to restore service.
Closing a socket gracefully after an OOM is a bit like trying to shut
one of the portholes on the Titanic, shortly after hearing a large
crashing sound.
There's only one place I know of where Tomcat attempts to interact with
OOM conditions and this is not one of them, so I don't believe it's safe
to say that Tomcat is deliberately handling this exception.
NB an OOM is an Error, not an Exception - it is a subclass of
VirtualMachineError, which is thrown to indicate that the Java Virtual
Machine is broken or has run out of resources necessary for it to
continue operating.
An Error is a subclass of Throwable that indicates serious problems that
a reasonable application should not try to catch.
</end-quote>
If anything, the locations where Tomcat catches a Throwable should be
modified so it does *not* catch Errors, rather than continuing to do so
and then attempting a trivial tidy-up.
p
--
[key:62590808]
Re: Bug in Tomcat AJP Connector?
Posted by Ofer Israeli <of...@checkpoint.com>.
Y
On 5 באפר 2012, at 18:58, "Konstantin Kolinko" <kn...@gmail.com> wrote:
> 2012/4/5 Ofer Israeli <of...@checkpoint.com>:
>> Mark Thomas wrote:
>>> On 04/04/2012 17:02, Ofer Israeli wrote:
>>>
>>> Once you have an OOME all bets are off. The JVM needs to be restarted.
>>> There is no guarantee of reliable operation after an OOME.
>>>
>>> Mark
>>
>> Hi Mark,
>> I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death.
>> Do you see any reason as not to close the listening socket?
>>
>
> 1. Tomcat does not start JVM thus it cannot restart it.
>
> You need some external tool or script or admin to perform monitoring
> and (re)starts.
>
> 2. OOM can happen at a random place. Once it happens, it is likely
> that other places will also start to fail randomly. It is also likely
> that your attempts to recover will fail as well.
>
> Mark already mentioned it: "all bets are off".
>
> Best regards,
> Konstantin Kolinko
>
Hi Konstantin,
I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread?
Best regards,
Ifer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 4/5/12 12:18 PM, Konstantin Kolinko wrote:
> 2012/4/5 Christopher Schultz <ch...@christopherschultz.net>:
>>
>> On 4/5/12 11:57 AM, Konstantin Kolinko wrote:
>>> 1. Tomcat does not start JVM thus it cannot restart it.
>>>
>>> You need some external tool or script or admin to perform
>>> monitoring and (re)starts.
>>
>> Asking Tomcat to restart itself after OOME would be like
>> expecting you to defibrillate yourself. It works in theory, but
>> not in practice.
>>
>
> Yes. It reminds me of that Bond movie
>
> http://www.youtube.com/watch?v=Jhm5up5NQwk
:)
Note that he still didn't restart his own heart... he needed help.
Maybe her name was 'cron' ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk993y4ACgkQ9CaO5/Lv0PAoCgCgsRm42v55a0XGWcfnJ/FF0Do8
AcMAmwWYowivPHKEsybuLmTie7FwV8eb
=2V9h
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Bug in Tomcat AJP Connector?
Posted by "Filip Hanik (mailing lists)" <de...@hanik.com>.
> -----Original Message-----
> From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
> Sent: Thursday, April 05, 2012 10:18 AM
> To: Tomcat Users List
> Subject: Re: Bug in Tomcat AJP Connector?
>
> 2012/4/5 Christopher Schultz <ch...@christopherschultz.net>:
> >
> > On 4/5/12 11:57 AM, Konstantin Kolinko wrote:
> >> 1. Tomcat does not start JVM thus it cannot restart it.
> >>
> >> You need some external tool or script or admin to perform
> >> monitoring and (re)starts.
> >
> > Asking Tomcat to restart itself after OOME would be like expecting you
> > to defibrillate yourself. It works in theory, but not in practice.
> >
>
> Yes. It reminds me of that Bond movie
[Filip Hanik]
I would not discard this as a joke, software is headed that way, there are
solutions for this today, and more of them will come tomorrow.
Self healing - it's gonna be fairly important in the new paradigm
>
> http://www.youtube.com/watch?v=Jhm5up5NQwk
>
> Best regards,
> Konstantin Kolinko
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/4/5 Christopher Schultz <ch...@christopherschultz.net>:
>
> On 4/5/12 11:57 AM, Konstantin Kolinko wrote:
>> 1. Tomcat does not start JVM thus it cannot restart it.
>>
>> You need some external tool or script or admin to perform
>> monitoring and (re)starts.
>
> Asking Tomcat to restart itself after OOME would be like expecting you
> to defibrillate yourself. It works in theory, but not in practice.
>
Yes. It reminds me of that Bond movie
http://www.youtube.com/watch?v=Jhm5up5NQwk
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 4/5/12 11:57 AM, Konstantin Kolinko wrote:
> 1. Tomcat does not start JVM thus it cannot restart it.
>
> You need some external tool or script or admin to perform
> monitoring and (re)starts.
Asking Tomcat to restart itself after OOME would be like expecting you
to defibrillate yourself. It works in theory, but not in practice.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk99waMACgkQ9CaO5/Lv0PCVogCgtEiQFk+fuEAHKc3lXkNAM0Ak
MrAAnR+NKO5LK4EAmxviUTnBjTJGg7Sv
=5S+d
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/4/5 Ofer Israeli <of...@checkpoint.com>:
> Mark Thomas wrote:
>> On 04/04/2012 17:02, Ofer Israeli wrote:
>>
>> Once you have an OOME all bets are off. The JVM needs to be restarted.
>> There is no guarantee of reliable operation after an OOME.
>>
>> Mark
>
> Hi Mark,
> I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death.
> Do you see any reason as not to close the listening socket?
>
1. Tomcat does not start JVM thus it cannot restart it.
You need some external tool or script or admin to perform monitoring
and (re)starts.
2. OOM can happen at a random place. Once it happens, it is likely
that other places will also start to fail randomly. It is also likely
that your attempts to recover will fail as well.
Mark already mentioned it: "all bets are off".
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Bug in Tomcat AJP Connector?
Posted by Ofer Israeli <of...@checkpoint.com>.
Mark Thomas wrote:
> On 04/04/2012 17:02, Ofer Israeli wrote:
>> Hi all,
>>
>> We have recently witnessed a strange situation. Our Tomcat (6.0.35)
>> listens on 2 ports: 8080 and 8009, handling HTTP and AJP
>> respectively. At some point in time we found that Apache is replying
>> HTTP 503s to all clients and a look at netstat showed that indeed
>> Apache could not communicate with Tomcat - there were many
>> connections in state SYN_SENT, that were never replied. A test of
>> attempting to connect to Tomcat's 8009 port turned out to give the
>> same result, so the issue was obviously in Tomcat and not Apache.
>> But netstat also showed that Tomcat was listening on port 8009.
>> After digging into the logs, we found that there was an Out of Memory
>> exception in the JK's accept() sequence and once this exception was
>> caught there was a message of "terminating thread".
>>
>> So although the exception is caught and handled by killing the thread
>> (isn't that a little drastic?), the socket isn't closed beforehand
>> and thus the OS thinks that the port is still fine (in LISTENING
>> state), as although the thread is dead, the process is alive. Has
>> anyone encountered this or knows if this bug is known?
>
> Once you have an OOME all bets are off. The JVM needs to be restarted.
> There is no guarantee of reliable operation after an OOME.
>
> Mark
Hi Mark,
I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death.
Do you see any reason as not to close the listening socket?
Ofer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Posted by Mark Thomas <ma...@apache.org>.
On 04/04/2012 17:02, Ofer Israeli wrote:
> Hi all,
>
> We have recently witnessed a strange situation. Our Tomcat (6.0.35)
> listens on 2 ports: 8080 and 8009, handling HTTP and AJP
> respectively. At some point in time we found that Apache is replying
> HTTP 503s to all clients and a look at netstat showed that indeed
> Apache could not communicate with Tomcat - there were many
> connections in state SYN_SENT, that were never replied. A test of
> attempting to connect to Tomcat's 8009 port turned out to give the
> same result, so the issue was obviously in Tomcat and not Apache.
> But netstat also showed that Tomcat was listening on port 8009.
> After digging into the logs, we found that there was an Out of Memory
> exception in the JK's accept() sequence and once this exception was
> caught there was a message of "terminating thread".
>
> So although the exception is caught and handled by killing the thread
> (isn't that a little drastic?), the socket isn't closed beforehand
> and thus the OS thinks that the port is still fine (in LISTENING
> state), as although the thread is dead, the process is alive. Has
> anyone encountered this or knows if this bug is known?
Once you have an OOME all bets are off. The JVM needs to be restarted.
There is no guarantee of reliable operation after an OOME.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org