You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2009/03/06 12:51:32 UTC
mod_jk local error states vs. global error states
On 06.03.2009 09:22, Mladen Turk wrote:
> BZ 46808 is valid but it brings us back where we were before.
> It solves (well doesn't actually) one thing,
> but breaks the sticky sessions.
> Unless the patch reliably detects the cause of failure
> I'm -1 for committing that. The problem is that returned
> codes from socket operations are platform specific, and
> then what if some one pulls the cable in the middle of
> AJP request? The possibilities are endless, and trying to
> solve each and every one is over engineering thought.
> We should keep the things simple:
>
> "If there are opened connections to the backend do not
> mark that worker in error"
> We simply do not know the status of the network beyond
> the current connection. It might be Tomcat rejecting
> new connections, firewall reaching the limit, ulimit,
> ssh tunnel restart, etc, etc ...
> So although our connection fails other might still
> work. If we can for sure detect something as absolute
> failure then we could save some concurrent client
> cycles, but then again we have a watchdog that can
> check all the client connections in background.
I don't want to simply commit the patch of BZ 46808, I first want to
fully understand the construction and implications of the local error
states. I did understand the concept in general (even before 1.2.27 was
released), but nevertheless I have some questions:
1) New busy introduced in r647085
=================================
We have three busy counters:
a) one for the lb in total
b) one for each lb sub
c) one for each ajp worker
In status worker we use only a) and c). In lb we use a) and b). Your
comment to BZ 46808 seems to indicate, that using c) instead ob b) in lb
would be better. We could then again remove b).
Right?
So we could drop the rec->s->busy++ and --, because that's done for the
ajp busyness alredy in jk_ajp_common.c and we would test against
aw->s->busy instead of rec->s->busy.
OK?
2) Local states vs. global states
=================================
I went through all places were we use and set states in lb. I have some
comments:
a) Setting local states and global states to different values
-------------------------------------------------------------
Of course the whole idea of the local state is to allow situations,
where we behave different for a single request and for the rest of the
system. On the other hand if we have to many such differences, the
system gets very complex, because as you can see further down, we need
to carefully decide whether we need to test against the global or
against the local state.
First the easier cases (I think)
(i) JK_CLIENT_ERROR
1258 else if (service_stat == JK_CLIENT_ERROR) {
1259 /*
1260 * Client error !!!
1261 * Since this is bad request do not fail over.
1262 */
1263 rec->s->state = JK_LB_STATE_OK;
1264 p->states[rec->i] = JK_LB_STATE_ERROR;
1265 rec->s->error_time = 0;
1266 rc = JK_CLIENT_ERROR;
1267 recoverable = JK_FALSE;
1268 }
recoverable=JK_FALSE means we no longer loop in service() and thus
states are no longer used. I would therefore suggest to keep the two
states equal to "OK" here. It doesn't matter for the logic, but makes
keeping track of the differences easier.
(ii) JK_STATUS_FATAL_ERROR
1291 else if (service_stat == JK_STATUS_FATAL_ERROR) {
1292 /*
1293 * Status code configured as service is down.
1294 * Mark the node as bad.
1295 * Failing over to another node could help.
1296 */
1297 rec->s->errors++;
1298 if (rec->s->busy) {
1299 rec->s->state = JK_LB_STATE_OK;
1300 }
1301 else {
1302 rec->s->state = JK_LB_STATE_ERROR;
1303 }
1304 p->states[rec->i] = JK_LB_STATE_ERROR;
1305 rec->s->error_time = time(NULL);
1306 rc = JK_FALSE;
1307 }
This is the situation, where fail_on_status was triggered with a
positive status code, so the admin configured to use this as ERROR. And
despite busyness, we actually got a response back. So I propose to not
make a distinction based on busyness here and to set the global state to
ERROR too.
OK?
(iii) JK_REPLY_TIMEOUT
1308 else if (service_stat == JK_REPLY_TIMEOUT) {
1309 if (aw->s->reply_timeouts >
(unsigned)p->worker->max_reply_timeouts) {
1310 /*
1311 * Service failed - to many reply timeouts
1312 * Take this node out of service.
1313 */
1314 rec->s->errors++;
1315 if (rec->s->busy) {
1316 rec->s->state = JK_LB_STATE_OK;
1317 }
1318 else {
1319 rec->s->state = JK_LB_STATE_ERROR;
1320 }
1321 p->states[rec->i] = JK_LB_STATE_ERROR;
1322 rec->s->error_time = time(NULL);
We are already over our max_reply_timeouts. So this should trigger
global ERROR, even when busy. The fact we have max_reply_timeouts is
exactly to prevent isolated reply_timeouts to trigger ERROR. If we
really go over that limit, we should go into ERROR without further
conditions. So I propose global ERROR here, without checking busy.
Now the more difficult cases:
(iv) JK_SERVER_ERROR
We only get this, if a memory allocation fails.
1269 else if (service_stat == JK_SERVER_ERROR) {
1270 /*
1271 * Internal JK server error.
1272 * Don't mark the node as bad.
1273 * Failing over to another node could help.
1274 */
1275 rec->s->state = JK_LB_STATE_OK;
1276 p->states[rec->i] = JK_LB_STATE_ERROR;
1277 rec->s->error_time = 0;
1278 rc = JK_FALSE;
1279 }
I'm fine with what you decided, although actually I see no reason why
allocation should work better or worse for one of the lb members.
We can leave it, to keep track of the differences it would be easier to
set local state to OK too.
(v) the "else"
else {
/*
* Service failed !!!
* Time for fault tolerance (if possible)...
*/
rec->s->errors++;
if (rec->s->busy) {
rec->s->state = JK_LB_STATE_OK;
}
else {
rec->s->state = JK_LB_STATE_ERROR;
}
p->states[rec->i] = JK_LB_STATE_ERROR;
rec->s->error_time = time(NULL);
rc = JK_FALSE;
}
This is the hard case. It handles connect problems, protocol failures
and a lot of various problems.
I agree, that it is hard to decide whether an error here should be
handled as a global failover decision. On the other hand in production
it is likely that busy>0 so we will never set a global error in this
"else" cases.
Of course now we are in good shape with cping/cpong (if a connection
exists) and the new socket connect timeout (if not) to detect problems
on the network layer really fast and before the request is wasted und
unrecoverable.
My idea would be: leave this as is, but add a second check based on the
error_time. We still set it, even when the global state is OK. In the
other cases, were you use the local state to simply prevent using the
same member in the service loop twice, we don't set it. So it's a good
indicator for this "else" tree.
Something along the lines of:
Index: jk_lb_worker.h
===================================================================
--- jk_lb_worker.h (revision 750052)
+++ jk_lb_worker.h (working copy)
@@ -123,6 +123,9 @@
/* Time to wait before retry. */
#define WAIT_BEFORE_RECOVER (60)
+/* Time after a local error switches to global error (seconds) */
+#define MAX_LOCAL_ERROR_TIME 10
+
/* We accept doing global maintenance if we are */
/* JK_LB_MAINTAIN_TOLERANCE seconds early. */
#define JK_LB_MAINTAIN_TOLERANCE (2)
Index: jk_lb_worker.c
===================================================================
--- jk_lb_worker.c (revision 750069)
+++ jk_lb_worker.c (working copy)
@@ -1338,15 +1338,17 @@
* Service failed !!!
* Time for fault tolerance (if possible)...
*/
+ time_t now = time(NULL);
rec->s->errors++;
- if (rec->s->busy) {
- rec->s->state = JK_LB_STATE_OK;
- }
- else {
+ if (rec->s->busy == 0 ||
+ (rec->s->error_time > 0 &&
+ (int)difftime(now, rec->s->error_time) >
MAX_LOCAL_ERROR_TIME)) {
rec->s->state = JK_LB_STATE_ERROR;
}
p->states[rec->i] = JK_LB_STATE_ERROR;
- rec->s->error_time = time(NULL);
+ if (rec->s->error_time == 0) {
+ rec->s->error_time = now;
+ }
rc = JK_FALSE;
}
if (p->worker->lblock == JK_LB_LOCK_PESSIMISTIC)
This also has another important side effect: it prevents switching an
established global error state back to OK, if we are in the "else" case
and busy==0.
What do you think?
a) Local states and forced recovery
--------------------------------------
I think we should test against the local state in forced recovery
(whenever global is ERROR, local is as well, but local can be without
global). Since we use forced recovery in maintenance (there we have no
local states) and in normal service, I suggest:
Index: jk_lb_worker.c
===================================================================
--- jk_lb_worker.c (revision 750069)
+++ jk_lb_worker.c (working copy)
@@ -570,17 +570,23 @@
}
static int force_recovery(lb_worker_t *p,
+ int *states,
jk_logger_t *l)
{
unsigned int i;
int forced = 0;
+ int state;
lb_sub_worker_t *w = NULL;
ajp_worker_t *aw = NULL;
JK_TRACE_ENTER(l);
for (i = 0; i < p->num_of_workers; i++) {
w = &p->lb_workers[i];
- if (w->s->state == JK_LB_STATE_ERROR) {
+ if (states == NULL)
+ state = w->s->state;
+ else
+ state = states[i];
+ if (state == JK_LB_STATE_ERROR) {
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_INFO,
"worker %s is marked for forced recovery",
@@ -588,6 +594,8 @@
aw = (ajp_worker_t *)w->worker->worker_private;
aw->s->reply_timeouts = 0;
w->s->state = JK_LB_STATE_FORCE;
+ if (states != NULL)
+ states[i] = JK_LB_STATE_FORCE;
forced++;
}
}
@@ -659,7 +667,7 @@
JK_LB_DECAY_MULT * delta / lb->maintain_time);
curmax = decay_load(lb, JK_LB_DECAY_MULT * delta /
lb->maintain_time, l);
if (!recover_workers(lb, curmax, now, l)) {
- force_recovery(lb, l);
+ force_recovery(lb, NULL, l);
}
}
@@ -1396,7 +1406,7 @@
* If it still fails, Tomcat is still disconnected.
*/
jk_shm_lock();
- nf = force_recovery(p->worker, l);
+ nf = force_recovery(p->worker, states, l);
jk_shm_unlock();
was_forced = 1;
if (nf) {
OK?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Mladen Turk <mt...@apache.org>.
Rainer Jung wrote:
> On 06.03.2009 13:32, Mladen Turk wrote:
>> For the rest it's simply too much to cope in a single email ;)
>
> I put the force recovery fix and the "else" suggestion in a patch at:
>
> http://people.apache.org/~rjung/mod_jk-dev/patches/local_states.patch
>
> Everything apart from Hunk number 3 and the small change to
> jklb_worker.h is for forced recovery, Hunk 3 and jk_lb_worker.h are for
> "else".
>
I've added the new JK_AJP_PROTOCOL_ERROR, so IMO you can
now apply your proposed patch to 'else'
'else' now means that we've send the request but the tomcat
immediately dropped the connection (first read after
send failed). Although I'm not sure why you've choose the
10 second recovery delay. Can't we stay with the configured
recovery_timeout here as well?
Regards
--
^(TM)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Mladen Turk <mt...@apache.org>.
Rainer Jung wrote:
> On 06.03.2009 13:32, Mladen Turk wrote:
>> For the rest it's simply too much to cope in a single email ;)
>
> I put the force recovery fix and the "else" suggestion in a patch at:
>
> http://people.apache.org/~rjung/mod_jk-dev/patches/local_states.patch
>
I've added 'in_error' for ajp worker.
We already have 'errors' counter, but this is
statistical one. New 'in_error' holds the number of connections
that are currently in error state.
If it's number get higher then busy/2 (more then half are in
error state) then the entire worker is marked as invalid.
Think this is much better then having additional retry for
local workers. The value busy/2 can probably be configured
via some directive similar to max_reply_timeouts,
eg, max_reply_errors.
Regards
--
^(TM)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 13:32, Mladen Turk wrote:
> For the rest it's simply too much to cope in a single email ;)
I put the force recovery fix and the "else" suggestion in a patch at:
http://people.apache.org/~rjung/mod_jk-dev/patches/local_states.patch
Everything apart from Hunk number 3 and the small change to
jklb_worker.h is for forced recovery, Hunk 3 and jk_lb_worker.h are for
"else".
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Mladen Turk <mt...@apache.org>.
Rainer Jung wrote:
> On 06.03.2009 14:19, Mladen Turk wrote:
>>
>> JkMount /foo aw
>> JkMount /bar aw
>>
>> Now, if /bar is slow and gets timeout it would mean that
>> /foo will be banned as well (although it might work perfectly)
>>
>> But I see your point. Since configured it should be banned
>> immediately. However this requires that admins behave 'smart'
>> and deploy their applications to different instances and use
>> different workers.
>>
>> Ideal would be for us to have per JkMount status in shared
>> memory. This is something for the future definitely.
>
> Yeah, there's such a dependency between mounts and workers. But exactly
> for this case we now have reply timeouts which can be set per mount.
> Because here one size doesn't fit all. By default all reply timeouts are
> off though.
>
Right, but although the reply timeouts are per mount the
consequences are per worker, because if one mount triggers
timeout all others will be affected.
> So I think going into global error here is safe.
>
For now yes, until we'll have a shared memory for each mount.
With that we'll be able to completely decouple
connection from application logic.
Regards
--
^(TM)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 14:19, Mladen Turk wrote:
> Rainer Jung wrote:
>> On 06.03.2009 13:32, Mladen Turk wrote:
>>> Rainer Jung wrote:
>>> All this should never touch the global state
>>> if there are live connections.
>>> Let the live connection decides for itself when it gets serviced.
>>> Anything else is just plain 'guessing'
>>>
>>> That was my general rule of thumb,
>>> because the point is to be robust as much as possible
>>
>> JK_CLIENT_ERROR: it does not touch the global state (well it sets it
>> to OK), but you do touch the local state. I argued, why I would set
>> the local state also to OK. Any answer?
>>
>
> Well you said:
> "It doesn't matter for the logic, but makes keeping track of the
> differences easier."
>
> I think opposite. Marking local as error and global as ok makes
> things track easier. local error means that this worker won't
> be tested in next lb loop, global error means it won't be tested
> until retry timeout.
Since it is no functional change, I can stick to your definition of
"easier", so we keep it as is.
> Explicitly setting global to OK reads as
> 'It wasn't our fault, it was client's fault, we are still in
> contact with backend'
But that's actaully what we do, we do set global explicitely to OK, and
we always did. I was talking about the local part here. But I think it's
OK to leave it as it is.
> But I agree, setting anything here is irrelevant, but like
> you said "It makes things easier to track and read"
OK.
>> JK_STATUS_FATAL_ERROR: The whole purpose of the fail_on_status
>> configuration item is to tell something else. E.g. there is a status
>> if the context is not available, or the app could have a filter
>> returning a special status. For me it does not make sense to simply
>> ignore, what the admin configured using fail_on_status.
>>
>
> OK, this should probably be set to global as well if configured
> explicitly. It will mean that all the sessions will be lost however.
OK. Yes, they'll be lost, but it depends on the admin to choose (or even
set by a filter) good status codes. But I think the most popular case
is, where the app is not deployed for some time.
>> JK_REPLY_TIMEOUT: Again I'm talking about the situation we have more
>> timeouts than max_reply_timeouts. By default we do not have any reply
>> timeout set, so the admin instructed us to react on reply timeouts.
>>
>
> JkMount /foo aw
> JkMount /bar aw
>
> Now, if /bar is slow and gets timeout it would mean that
> /foo will be banned as well (although it might work perfectly)
>
> But I see your point. Since configured it should be banned
> immediately. However this requires that admins behave 'smart'
> and deploy their applications to different instances and use
> different workers.
>
> Ideal would be for us to have per JkMount status in shared
> memory. This is something for the future definitely.
Yeah, there's such a dependency between mounts and workers. But exactly
for this case we now have reply timeouts which can be set per mount.
Because here one size doesn't fit all. By default all reply timeouts are
off though.
So I think going into global error here is safe.
Note: it's only in the first half of the timeout handling, when we are
already above max_reply_timeouts, so we are not talking about isolated
timeouts.
>>>> (iv) JK_SERVER_ERROR
>>>>
>>>> We only get this, if a memory allocation fails.
>>>>
>>>> I'm fine with what you decided, although actually I see no reason why
>>>> allocation should work better or worse for one of the lb members.
>>>> We can leave it, to keep track of the differences it would be easier
>>>> to set local state to OK too.
>>>>
>>>
>>> Well, again I disagree. Actually in worker or prefork the child
>>> will simply die without setting the global error in shm.
>>> If we set here the global error it will again kill all the
>>> sessions if one child had some memory issues.
>>
>> I didn't suggest setting global to ERROR, I suggested setting local to
>> OK, because I don't get, hwat local ERROR helps here, and keeping both
>> equal is easier to understand.
>>
>
> Hmm, right, but setting local to OK won't probably help.
> It might end up in some garbage send to Tomcat, so it's
> better to mark it as local error.
OK. Agreed.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 14:19, Mladen Turk wrote:
> Rainer Jung wrote:
>> On 06.03.2009 13:32, Mladen Turk wrote:
>>> Huge one Rainer ;)
>>
>> I know, but I went through it in depth.
>>
>>> Rainer Jung wrote:
>>>>
>>>> We have three busy counters:
>>>>
>>>> a) one for the lb in total
>>>> b) one for each lb sub
>>>> c) one for each ajp worker
>>>>
>>>> In status worker we use only a) and c). In lb we use a) and b). Your
>>>> comment to BZ 46808 seems to indicate, that using c) instead ob b) in
>>>> lb would be better. We could then again remove b).
>>>>
>>>> Right?
>>>>
>>>> So we could drop the rec->s->busy++ and --, because that's done for
>>>> the ajp busyness alredy in jk_ajp_common.c and we would test against
>>>> aw->s->busy instead of rec->s->busy.
>>>>
>>>> OK?
>>>>
>>>
>>> No because rec->s->busy is per-lb info,
>>> and aw->s-busy >= res->s->busy if aw is member
>>> of multiple lb's
>>
>> But isn't the whole purpose of your changes to give the backend still
>> a chance, if it is processing requests? Why does it then matter,
>> whether those requests come from the same lb??
Any answer to that one? You are using this busy only for the purpose to
differentiate between a busy backend (do not put it into global error
under certain circumstances) and an idle backend (take most errors more
seriously, because they are not simply triggered by ovrload).
So the real load should be the best differentiator, and aw->s-busy is
close to the real load than res->s->busy.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Mladen Turk <mt...@apache.org>.
Rainer Jung wrote:
> On 06.03.2009 13:32, Mladen Turk wrote:
>> Huge one Rainer ;)
>
> I know, but I went through it in depth.
>
>> Rainer Jung wrote:
>>>
>>> We have three busy counters:
>>>
>>> a) one for the lb in total
>>> b) one for each lb sub
>>> c) one for each ajp worker
>>>
>>> In status worker we use only a) and c). In lb we use a) and b). Your
>>> comment to BZ 46808 seems to indicate, that using c) instead ob b) in
>>> lb would be better. We could then again remove b).
>>>
>>> Right?
>>>
>>> So we could drop the rec->s->busy++ and --, because that's done for
>>> the ajp busyness alredy in jk_ajp_common.c and we would test against
>>> aw->s->busy instead of rec->s->busy.
>>>
>>> OK?
>>>
>>
>> No because rec->s->busy is per-lb info,
>> and aw->s-busy >= res->s->busy if aw is member
>> of multiple lb's
>
> But isn't the whole purpose of your changes to give the backend still a
> chance, if it is processing requests? Why does it then matter, whether
> those requests come from the same lb??
>
>>> 2) Local states vs. global states
>>> =================================
>>>
>>> I went through all places were we use and set states in lb. I have
>>> some comments:
>>>
>>> a) Setting local states and global states to different values
>>> -------------------------------------------------------------
>>>
>>>
>>> First the easier cases (I think)
>>>
>>> (i) JK_CLIENT_ERROR
>>>
>>> (ii) JK_STATUS_FATAL_ERROR
>>>
>>> (iii) JK_REPLY_TIMEOUT
>>>
>>
>> All this should never touch the global state
>> if there are live connections.
>> Let the live connection decides for itself when it gets serviced.
>> Anything else is just plain 'guessing'
>>
>> That was my general rule of thumb,
>> because the point is to be robust as much as possible
>
> JK_CLIENT_ERROR: it does not touch the global state (well it sets it to
> OK), but you do touch the local state. I argued, why I would set the
> local state also to OK. Any answer?
>
Well you said:
"It doesn't matter for the logic, but makes keeping track of the
differences easier."
I think opposite. Marking local as error and global as ok makes
things track easier. local error means that this worker won't
be tested in next lb loop, global error means it won't be tested
until retry timeout. Explicitly setting global to OK reads as
'It wasn't our fault, it was client's fault, we are still in
contact with backend'
But I agree, setting anything here is irrelevant, but like
you said "It makes things easier to track and read"
> JK_STATUS_FATAL_ERROR: The whole purpose of the fail_on_status
> configuration item is to tell something else. E.g. there is a status if
> the context is not available, or the app could have a filter returning a
> special status. For me it does not make sense to simply ignore, what the
> admin configured using fail_on_status.
>
OK, this should probably be set to global as well if configured
explicitly. It will mean that all the sessions will be lost however.
> JK_REPLY_TIMEOUT: Again I'm talking about the situation we have more
> timeouts than max_reply_timeouts. By default we do not have any reply
> timeout set, so the admin instructed us to react on reply timeouts.
>
JkMount /foo aw
JkMount /bar aw
Now, if /bar is slow and gets timeout it would mean that
/foo will be banned as well (although it might work perfectly)
But I see your point. Since configured it should be banned
immediately. However this requires that admins behave 'smart'
and deploy their applications to different instances and use
different workers.
Ideal would be for us to have per JkMount status in shared
memory. This is something for the future definitely.
>>>
>>> Now the more difficult cases:
>>>
>>> (iv) JK_SERVER_ERROR
>>>
>>> We only get this, if a memory allocation fails.
>>>
>>> I'm fine with what you decided, although actually I see no reason why
>>> allocation should work better or worse for one of the lb members.
>>> We can leave it, to keep track of the differences it would be easier
>>> to set local state to OK too.
>>>
>>
>> Well, again I disagree. Actually in worker or prefork the child
>> will simply die without setting the global error in shm.
>> If we set here the global error it will again kill all the
>> sessions if one child had some memory issues.
>
> I didn't suggest setting global to ERROR, I suggested setting local to
> OK, because I don't get, hwat local ERROR helps here, and keeping both
> equal is easier to understand.
>
Hmm, right, but setting local to OK won't probably help.
It might end up in some garbage send to Tomcat, so it's
better to mark it as local error.
Regards
--
^(TM)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 13:32, Mladen Turk wrote:
> Huge one Rainer ;)
I know, but I went through it in depth.
> Rainer Jung wrote:
>>
>> We have three busy counters:
>>
>> a) one for the lb in total
>> b) one for each lb sub
>> c) one for each ajp worker
>>
>> In status worker we use only a) and c). In lb we use a) and b). Your
>> comment to BZ 46808 seems to indicate, that using c) instead ob b) in
>> lb would be better. We could then again remove b).
>>
>> Right?
>>
>> So we could drop the rec->s->busy++ and --, because that's done for
>> the ajp busyness alredy in jk_ajp_common.c and we would test against
>> aw->s->busy instead of rec->s->busy.
>>
>> OK?
>>
>
> No because rec->s->busy is per-lb info,
> and aw->s-busy >= res->s->busy if aw is member
> of multiple lb's
But isn't the whole purpose of your changes to give the backend still a
chance, if it is processing requests? Why does it then matter, whether
those requests come from the same lb??
>> 2) Local states vs. global states
>> =================================
>>
>> I went through all places were we use and set states in lb. I have
>> some comments:
>>
>> a) Setting local states and global states to different values
>> -------------------------------------------------------------
>>
>>
>> First the easier cases (I think)
>>
>> (i) JK_CLIENT_ERROR
>>
>> (ii) JK_STATUS_FATAL_ERROR
>>
>> (iii) JK_REPLY_TIMEOUT
>>
>
> All this should never touch the global state
> if there are live connections.
> Let the live connection decides for itself when it gets serviced.
> Anything else is just plain 'guessing'
>
> That was my general rule of thumb,
> because the point is to be robust as much as possible
JK_CLIENT_ERROR: it does not touch the global state (well it sets it to
OK), but you do touch the local state. I argued, why I would set the
local state also to OK. Any answer?
JK_STATUS_FATAL_ERROR: The whole purpose of the fail_on_status
configuration item is to tell something else. E.g. there is a status if
the context is not available, or the app could have a filter returning a
special status. For me it does not make sense to simply ignore, what the
admin configured using fail_on_status.
JK_REPLY_TIMEOUT: Again I'm talking about the situation we have more
timeouts than max_reply_timeouts. By default we do not have any reply
timeout set, so the admin instructed us to react on reply timeouts.
>>
>> Now the more difficult cases:
>>
>> (iv) JK_SERVER_ERROR
>>
>> We only get this, if a memory allocation fails.
>>
>> I'm fine with what you decided, although actually I see no reason why
>> allocation should work better or worse for one of the lb members.
>> We can leave it, to keep track of the differences it would be easier
>> to set local state to OK too.
>>
>
> Well, again I disagree. Actually in worker or prefork the child
> will simply die without setting the global error in shm.
> If we set here the global error it will again kill all the
> sessions if one child had some memory issues.
I didn't suggest setting global to ERROR, I suggested setting local to
OK, because I don't get, hwat local ERROR helps here, and keeping both
equal is easier to understand.
> For the rest it's simply too much to cope in a single email ;)
Right!
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: mod_jk local error states vs. global error states
Posted by Mladen Turk <mt...@apache.org>.
Huge one Rainer ;)
Rainer Jung wrote:
>
> We have three busy counters:
>
> a) one for the lb in total
> b) one for each lb sub
> c) one for each ajp worker
>
> In status worker we use only a) and c). In lb we use a) and b). Your
> comment to BZ 46808 seems to indicate, that using c) instead ob b) in lb
> would be better. We could then again remove b).
>
> Right?
>
> So we could drop the rec->s->busy++ and --, because that's done for the
> ajp busyness alredy in jk_ajp_common.c and we would test against
> aw->s->busy instead of rec->s->busy.
>
> OK?
>
No because rec->s->busy is per-lb info,
and aw->s-busy >= res->s->busy if aw is member
of multiple lb's
> 2) Local states vs. global states
> =================================
>
> I went through all places were we use and set states in lb. I have some
> comments:
>
> a) Setting local states and global states to different values
> -------------------------------------------------------------
>
>
> First the easier cases (I think)
>
> (i) JK_CLIENT_ERROR
>
> (ii) JK_STATUS_FATAL_ERROR
>
> (iii) JK_REPLY_TIMEOUT
>
All this should never touch the global state
if there are live connections.
Let the live connection decides for itself when it gets serviced.
Anything else is just plain 'guessing'
That was my general rule of thumb,
because the point is to be robust as much as possible
>
> Now the more difficult cases:
>
> (iv) JK_SERVER_ERROR
>
> We only get this, if a memory allocation fails.
>
> I'm fine with what you decided, although actually I see no reason why
> allocation should work better or worse for one of the lb members.
> We can leave it, to keep track of the differences it would be easier to
> set local state to OK too.
>
Well, again I disagree. Actually in worker or prefork the child
will simply die without setting the global error in shm.
If we set here the global error it will again kill all the
sessions if one child had some memory issues.
For the rest it's simply too much to cope in a single email ;)
Cheers
--
^(TM)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org