You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Madhuri Patwardhan <ma...@yahoo.com> on 2009/08/04 00:42:47 UTC

what exactly happens when timeouts are set and tomcat has outofmemory errors.

Hi,

Can somebody go over the details of what exactly happens when tomcat instance has outOfmemory errors and there are following timeouts set in a loadbalancer worker config. 

This is my understanding.

ping_mode=A, ping_timeout=10000, reply_timeout=60000 and recorver_options=3

If the timeouts are set as above then CPing/Cpong test for connect and prepost will work. When a new real request is sent to this tomcat instance, it should hit reply_timeout because it has outofmemory errors. Then depending on no. of retries set (it is default, for me so 2), this instance will be tried once again. Second time also it should not get reply within reply_timeout and at that point it should be marked in "ERR" state. 
However, I don't see that. I don't see it being marked in "ERR" state. The state is still "OK" inspite of this tomcat instance having outofmemory errors. Could somebody explain what am I missing?

Also, what does recorvery option mean in this case? Is it possible to explain what happens when I set recovery options? what happens when I set it to 3? What will happen if I set it to 7?

I am really hoping to get more understanding of how this works.

Thanks,
Madhuri

--- On Mon, 8/3/09, Madhuri Patwardhan <ma...@yahoo.com> wrote:

> From: Madhuri Patwardhan <ma...@yahoo.com>
> Subject: Re: after setting timeouts got puzzling results
> To: "Tomcat Users List" <us...@tomcat.apache.org>, aw@ice-sa.com
> Date: Monday, August 3, 2009, 6:12 PM
> 
> Hi,
> 
> I am using apache 2.2.11 and mod_jk 1.2.28 and tomcat
> version 5.5.27 and 5.5.20
> 
> Thanks,
> Madhuri
> 
> --- On Mon, 8/3/09, André Warnier <aw...@ice-sa.com>
> wrote:
> 
> > From: André Warnier <aw...@ice-sa.com>
> > Subject: Re: after setting timeouts got puzzling
> results
> > To: "Tomcat Users List" <us...@tomcat.apache.org>
> > Date: Monday, August 3, 2009, 1:48 PM
> > Madhuri Patwardhan wrote:
> > > Hi,
> > > 
> > > We are running with apache loadbalancer and 19
> tomcat
> > workers loadbalanced.
> > > I went through the timeout document and other
> > documents, and my apache access log in detail before
> > choosing the timeout values. I am seeing a bit
> surprising
> > results after setting timeouts.
> > > 
> > > Earlier we did not have connect_timeout,
> > prepost_timeout, reply_timeout, recovery_options and
> > ping_mode set. After going through the documents and
> > analyzing our application logs for timings I set the
> > following values.
> > > 
> > > ping_mode = A
> > > reply_timeout=60000
> > > recorvery_options=3
> > > socket_connect_timeout=5000
> > > 
> > > Ping_timeout is set to the default value that is
> > 10000.
> > > 
> > > I saw that one of the tomcat instances had out
> of
> > memory errors and loadbalacer still showed that worker
> in
> > good state for 24 hours. I felt after setting these
> timeouts
> > infact it should have detected conditions like this
> quickly.
> > What am I missing?
> > > 
> > One thing that is missing above, and would help
> someone to
> > answer, are the versions of Apache, mod_jk and Tomcat
> you
> > are using.
> > (At least I suppose it is mod_jk).
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> > 
> > 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.

Posted by Anantha Padmanabha <an...@bankbazaar.com>.
Madhuri,

Its better to use max_reply_timeouts also in config so that LB does not mark
a worker as 'down' once in a while.
We dont use socket_connect_timeout and similar config you are mentioning had
worked fine while we did OutOfMemory testing as far as I remember.

If someone is not looking at why OOM is happening, I suggest you to give
more focus on that now before config changes

Thanks
Anantha

On Tue, Aug 4, 2009 at 4:54 PM, Madhuri Patwardhan <
madhuri_patwardhan@yahoo.com> wrote:

>
> Thanks for the reply.
>
> Even after reading documents I have some doubts.
>
> 1] How is a particular tomcat instance marked to "ERR" state?
> This is what I understand. If apache is not able to connect to that tomcat
> instance port ( typically when tomcat is not running) OR if it does not
> receive reply from the tomcat instance. The second part is were I am not
> very clear. If reply_timeout is not set then it waits indefinitely for the
> answer then how does it mark it in state "ERR". If reply_timeout is set then
> it will wait for that time for no. of retries and then mark it in state ERR.
> Is there any other way a instance would get marked to state ERR?
>
> 2] Once instance is marked to state ERR then how does recovery options
> affect it?
> As per the document: "Recovery options influence, how we should handle
> retries, in case we detect a problem with Tomcat. How often we will retry is
> controlled by the attribute retries."
> So, if for example I have recovery options set to 3 how would it work? If I
> have set it to 7 how would it work?
>
> I am trying to understand the details of all this so that I can get to the
> bottom of the problem that I am facing. The problem that I am having is
> after setting these timeouts I am seeing worse performance from the website
> There are many website access errors reported.
> Earlier the following was not set and now I have set ping_mode=A,
> reply_timeout=60000, socket_connect_timeout=5000 and recovery_options=3.
>
> Thanks,
> Madhuri
>
> --- On Tue, 8/4/09, Mark Thomas <ma...@apache.org> wrote:
>
> > From: Mark Thomas <ma...@apache.org>
> > Subject: Re: what exactly happens when timeouts are set and tomcat has
> outofmemory errors.
> > To: "Tomcat Users List" <us...@tomcat.apache.org>
> > Date: Tuesday, August 4, 2009, 5:05 AM
> > Madhuri Patwardhan wrote:
> >
> > > Do you know anything about recovery options?
> >
> > No. You could try reading the docs and/or the source code.
> >
> > Mark
> > >
> > > Thanks,
> > > Madhuri
> > >
> > > --- On Mon, 8/3/09, Mark Thomas <ma...@apache.org>
> > wrote:
> > >
> > >> From: Mark Thomas <ma...@apache.org>
> > >> Subject: Re: what exactly happens when timeouts
> > are set and tomcat has outofmemory errors.
> > >> To: "Tomcat Users List" <us...@tomcat.apache.org>
> > >> Date: Monday, August 3, 2009, 6:49 PM
> > >> Madhuri Patwardhan wrote:
> > >>> Hi,
> > >>>
> > >>> Can somebody go over the details of what
> > exactly
> > >> happens when tomcat instance has outOfmemory
> > errors and
> > >> there are following timeouts set in a loadbalancer
> > worker
> > >> config.
> > >>> This is my understanding.
> > >>>
> > >>> ping_mode=A, ping_timeout=10000,
> > reply_timeout=60000
> > >> and recorver_options=3
> > >>> If the timeouts are set as above then
> > CPing/Cpong test
> > >> for connect and prepost will work. When a new real
> > request
> > >> is sent to this tomcat instance, it should hit
> > reply_timeout
> > >> because it has outofmemory errors.
> > >>
> > >> That depends on the state of the JVM. It might
> > appear to
> > >> work normally,
> > >> it might fail to respond at all. Once on OOM has
> > occurred
> > >> Tomcat has to
> > >> be assumed to be in an unknown state and should
> > be
> > >> restarted. I have
> > >> seen Tomcat instances apparently recover from an
> > OOM. It
> > >> all depends why
> > >> the OOM occurred and what Tomcat was doing at the
> > time.
> > >>
> > >>> Then depending on no. of retries set (it is
> > default,
> > >> for me so 2), this instance will be tried once
> > again. Second
> > >> time also it should not get reply within
> > reply_timeout and
> > >> at that point it should be marked in "ERR" state.
> >
> > >>> However, I don't see that. I don't see it
> > being marked
> > >> in "ERR" state. The state is still "OK" inspite of
> > this
> > >> tomcat instance having outofmemory errors. Could
> > somebody
> > >> explain what am I missing?
> > >>
> > >> In short, you are incorrectly assuming that once
> > an OOM has
> > >> occurred
> > >> Tomcat will no longer respond to requests.
> > >>
> > >> Mark
> > >>
> > >>
> > >>
> > >>
> > >>
> > ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>
> > >>
> > >
> > >
> > >
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.

Posted by Madhuri Patwardhan <ma...@yahoo.com>.
Thanks for the reply.

Even after reading documents I have some doubts.

1] How is a particular tomcat instance marked to "ERR" state?
This is what I understand. If apache is not able to connect to that tomcat instance port ( typically when tomcat is not running) OR if it does not receive reply from the tomcat instance. The second part is were I am not very clear. If reply_timeout is not set then it waits indefinitely for the answer then how does it mark it in state "ERR". If reply_timeout is set then it will wait for that time for no. of retries and then mark it in state ERR. Is there any other way a instance would get marked to state ERR?

2] Once instance is marked to state ERR then how does recovery options affect it?
As per the document: "Recovery options influence, how we should handle retries, in case we detect a problem with Tomcat. How often we will retry is controlled by the attribute retries." 
So, if for example I have recovery options set to 3 how would it work? If I have set it to 7 how would it work?

I am trying to understand the details of all this so that I can get to the bottom of the problem that I am facing. The problem that I am having is after setting these timeouts I am seeing worse performance from the website There are many website access errors reported.
Earlier the following was not set and now I have set ping_mode=A, reply_timeout=60000, socket_connect_timeout=5000 and recovery_options=3.

Thanks,
Madhuri

--- On Tue, 8/4/09, Mark Thomas <ma...@apache.org> wrote:

> From: Mark Thomas <ma...@apache.org>
> Subject: Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: Tuesday, August 4, 2009, 5:05 AM
> Madhuri Patwardhan wrote:
> 
> > Do you know anything about recovery options?
> 
> No. You could try reading the docs and/or the source code.
> 
> Mark
> > 
> > Thanks,
> > Madhuri
> > 
> > --- On Mon, 8/3/09, Mark Thomas <ma...@apache.org>
> wrote:
> > 
> >> From: Mark Thomas <ma...@apache.org>
> >> Subject: Re: what exactly happens when timeouts
> are set and tomcat has outofmemory errors.
> >> To: "Tomcat Users List" <us...@tomcat.apache.org>
> >> Date: Monday, August 3, 2009, 6:49 PM
> >> Madhuri Patwardhan wrote:
> >>> Hi,
> >>>
> >>> Can somebody go over the details of what
> exactly
> >> happens when tomcat instance has outOfmemory
> errors and
> >> there are following timeouts set in a loadbalancer
> worker
> >> config. 
> >>> This is my understanding.
> >>>
> >>> ping_mode=A, ping_timeout=10000,
> reply_timeout=60000
> >> and recorver_options=3
> >>> If the timeouts are set as above then
> CPing/Cpong test
> >> for connect and prepost will work. When a new real
> request
> >> is sent to this tomcat instance, it should hit
> reply_timeout
> >> because it has outofmemory errors.
> >>
> >> That depends on the state of the JVM. It might
> appear to
> >> work normally,
> >> it might fail to respond at all. Once on OOM has
> occurred
> >> Tomcat has to
> >> be assumed to be in an unknown state and should
> be
> >> restarted. I have
> >> seen Tomcat instances apparently recover from an
> OOM. It
> >> all depends why
> >> the OOM occurred and what Tomcat was doing at the
> time.
> >>
> >>> Then depending on no. of retries set (it is
> default,
> >> for me so 2), this instance will be tried once
> again. Second
> >> time also it should not get reply within
> reply_timeout and
> >> at that point it should be marked in "ERR" state.
> 
> >>> However, I don't see that. I don't see it
> being marked
> >> in "ERR" state. The state is still "OK" inspite of
> this
> >> tomcat instance having outofmemory errors. Could
> somebody
> >> explain what am I missing?
> >>
> >> In short, you are incorrectly assuming that once
> an OOM has
> >> occurred
> >> Tomcat will no longer respond to requests.
> >>
> >> Mark
> >>
> >>
> >>
> >>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> > 
> > 
> >       
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> > 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.

Posted by Mark Thomas <ma...@apache.org>.
Madhuri Patwardhan wrote:

> Do you know anything about recovery options?

No. You could try reading the docs and/or the source code.

Mark
> 
> Thanks,
> Madhuri
> 
> --- On Mon, 8/3/09, Mark Thomas <ma...@apache.org> wrote:
> 
>> From: Mark Thomas <ma...@apache.org>
>> Subject: Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.
>> To: "Tomcat Users List" <us...@tomcat.apache.org>
>> Date: Monday, August 3, 2009, 6:49 PM
>> Madhuri Patwardhan wrote:
>>> Hi,
>>>
>>> Can somebody go over the details of what exactly
>> happens when tomcat instance has outOfmemory errors and
>> there are following timeouts set in a loadbalancer worker
>> config. 
>>> This is my understanding.
>>>
>>> ping_mode=A, ping_timeout=10000, reply_timeout=60000
>> and recorver_options=3
>>> If the timeouts are set as above then CPing/Cpong test
>> for connect and prepost will work. When a new real request
>> is sent to this tomcat instance, it should hit reply_timeout
>> because it has outofmemory errors.
>>
>> That depends on the state of the JVM. It might appear to
>> work normally,
>> it might fail to respond at all. Once on OOM has occurred
>> Tomcat has to
>> be assumed to be in an unknown state and should be
>> restarted. I have
>> seen Tomcat instances apparently recover from an OOM. It
>> all depends why
>> the OOM occurred and what Tomcat was doing at the time.
>>
>>> Then depending on no. of retries set (it is default,
>> for me so 2), this instance will be tried once again. Second
>> time also it should not get reply within reply_timeout and
>> at that point it should be marked in "ERR" state. 
>>> However, I don't see that. I don't see it being marked
>> in "ERR" state. The state is still "OK" inspite of this
>> tomcat instance having outofmemory errors. Could somebody
>> explain what am I missing?
>>
>> In short, you are incorrectly assuming that once an OOM has
>> occurred
>> Tomcat will no longer respond to requests.
>>
>> Mark
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 
> 
>       
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.

Posted by Madhuri Patwardhan <ma...@yahoo.com>.
Before I set the timeouts thats how it worked. The tomcat instance in outof memory state would be eventually marked in "ERR" state. I thought by using timeouts it would set it to "ERR" state early.

I do need to restart the tomcat instance in outofmemory error but how would I know tomcat instance is in that state unless I keep checking logs of all tomcat instances. I used the "ERR" indication on jkstatus page to know if there is something wrong with tomcat instance.

Actually, before setting the following timeouts, I would see tomcat instances get marked to "ERR" state. After, setting these timeouts that is not happening as much as it happened before. After setting these timeouts I have got lot more error reports of website access errors.

Do you know anything about recovery options?

Thanks,
Madhuri

--- On Mon, 8/3/09, Mark Thomas <ma...@apache.org> wrote:

> From: Mark Thomas <ma...@apache.org>
> Subject: Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: Monday, August 3, 2009, 6:49 PM
> Madhuri Patwardhan wrote:
> > Hi,
> > 
> > Can somebody go over the details of what exactly
> happens when tomcat instance has outOfmemory errors and
> there are following timeouts set in a loadbalancer worker
> config. 
> > 
> > This is my understanding.
> > 
> > ping_mode=A, ping_timeout=10000, reply_timeout=60000
> and recorver_options=3
> > 
> > If the timeouts are set as above then CPing/Cpong test
> for connect and prepost will work. When a new real request
> is sent to this tomcat instance, it should hit reply_timeout
> because it has outofmemory errors.
> 
> That depends on the state of the JVM. It might appear to
> work normally,
> it might fail to respond at all. Once on OOM has occurred
> Tomcat has to
> be assumed to be in an unknown state and should be
> restarted. I have
> seen Tomcat instances apparently recover from an OOM. It
> all depends why
> the OOM occurred and what Tomcat was doing at the time.
> 
> > Then depending on no. of retries set (it is default,
> for me so 2), this instance will be tried once again. Second
> time also it should not get reply within reply_timeout and
> at that point it should be marked in "ERR" state. 
> > However, I don't see that. I don't see it being marked
> in "ERR" state. The state is still "OK" inspite of this
> tomcat instance having outofmemory errors. Could somebody
> explain what am I missing?
> 
> In short, you are incorrectly assuming that once an OOM has
> occurred
> Tomcat will no longer respond to requests.
> 
> Mark
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: what exactly happens when timeouts are set and tomcat has outofmemory errors.

Posted by Mark Thomas <ma...@apache.org>.
Madhuri Patwardhan wrote:
> Hi,
> 
> Can somebody go over the details of what exactly happens when tomcat instance has outOfmemory errors and there are following timeouts set in a loadbalancer worker config. 
> 
> This is my understanding.
> 
> ping_mode=A, ping_timeout=10000, reply_timeout=60000 and recorver_options=3
> 
> If the timeouts are set as above then CPing/Cpong test for connect and prepost will work. When a new real request is sent to this tomcat instance, it should hit reply_timeout because it has outofmemory errors.

That depends on the state of the JVM. It might appear to work normally,
it might fail to respond at all. Once on OOM has occurred Tomcat has to
be assumed to be in an unknown state and should be restarted. I have
seen Tomcat instances apparently recover from an OOM. It all depends why
the OOM occurred and what Tomcat was doing at the time.

> Then depending on no. of retries set (it is default, for me so 2), this instance will be tried once again. Second time also it should not get reply within reply_timeout and at that point it should be marked in "ERR" state. 
> However, I don't see that. I don't see it being marked in "ERR" state. The state is still "OK" inspite of this tomcat instance having outofmemory errors. Could somebody explain what am I missing?

In short, you are incorrectly assuming that once an OOM has occurred
Tomcat will no longer respond to requests.

Mark




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org