You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Dominik Pospisil <dp...@redhat.com> on 2008/01/11 13:57:11 UTC
mod_jk maintenance, recovery
Hello,
I am having following problem with following failover test scenario.
Cluster setup:
- 1 apache load balancer
- 2 nodes with equal LB factor
- sticky session turned on
- Apache/2.0.52, mod_jk/1.2.26
Test scenario:
1. start 1st node
2. start load driver
3. start 2nd node
4. wait for state transfer (2 minutes)
5. kill 1st node
My experience is that after stage 1 and 2, all clients are handled correctly
by 1st node and the second node is set correctly to ERR state. After while,
the second none switches to ERR/REC state.
However at stage 4 (after starting 2nd node) the second node will never come
up to OK state. I have set both worker maintain period and LB recovery_time
to 30s. So i guess that in 2 minutes, the second node should have been
re-checked. When I press manually "Reset worker state" button, it comes up
immediatelly, but it never happend automatically during maintenance phase.
Eventually, after killing 1st node, and after returning couple of "503 Service
Temporarily Unavailable" exceptions, mod_jk finally recheck 2nd node status,
reroute requests to 2nd node and resumes correct operation.
My question is: Why the second node is not recognized before failover? Did I
missed something? Or is it a bug?
Thanks,
- Dominik
Attaching worker.properties
---------
worker.list=loadbalancer,status
worker.maintain=30
# modify the host as your host IP or DNS name.
worker.cluster01.port=8009
worker.cluster01.host=172.17.0.39
worker.cluster01.type=ajp13
worker.cluster01.lbfactor=1
#worker.cluster01.redirect=cluster02
# modify the host as your host IP or DNS name.
worker.cluster02.port=8009
worker.cluster02.host=172.17.1.39
worker.cluster02.type=ajp13
worker.cluster02.lbfactor=1
#worker.cluster02.redirect=cluster01
# modify the host as your host IP or DNS name.
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.method=Session
worker.loadbalancer.balance_workers=cluster01,cluster02
worker.loadbalancer.sticky_session=1
worker.loadbalancer.recover_time=30
#worker.list=loadbalancer
# Status worker for managing load balancer
worker.status.type=status
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Dominik Pospisil <dp...@redhat.com>.
Rainer,
thanks for valuable info.
>> Let's see, if this matches your experience: assuming recovery_options=0,
> the request should be retried on other lb member workers (if such exist
> and are OK), unless it's a POST and the POST body is bigger than 8186
> bytes and more than the first 8186 bytes have already been sent to the
> backend.
>
> Why doesn't it get retried in this case? Because we only buffer that
> many data for retry.
>
> POST bodies can get very big (uploads), so it's not good to buffer
> complete POST bodies (could take a lot of memory). AJP tries to stream
> packets through the web server to the backend as they arrive. The actual
> design simply decides to buffer one ajp packet, which is 8192 bytes
> minus ajp head = 8186 bytes.
>
> Does that fit your observations?
>
No, it does not. I am sending GET requests only. And again, errors are
returned for couple of first request only - i am using 4000 clients and about
100 clients receives this error. Than the second node switches from ERR/REC
to OK state and rest of clients are redirected correctly.
>... adding to previous post:
>
>It also does not retry/fail over, in the following (more or less obvious
>cases):
>
>- there was a problem with the client (browser) connection
>- we already started to send back parts of the response to the client,
>typically the headers (depending on recovery_options; default:
>retry/fail over)
>- Internal JK error (not observed in practise yet)
>- depending on recovery_options: if the backend received the request
>(default: retry/fail over)
None of this options seems to be the case too. If some data were send back to
client before failover, i would detect it in the load generator.
mod_jk.log fragments:
couple of first lines after failover:
[Mon Jan 21 08:47:26 2008][27428:1262557536] [error]
ajp_send_request::jk_ajp_common.c (1359): (cluster01) connecting to backend
failed. Tomcat is probably not started or is listening on the wrong port
(errno=110)
[Mon Jan 21 08:47:26 2008][22609:1252067680] [error]
ajp_send_request::jk_ajp_common.c (1359): (cluster01) connecting to backend
failed. Tomcat is probably not started or is listening on the wrong port
(errno=110)
[Mon Jan 21 08:47:26 2008][29495:1115699552] [error]
ajp_send_request::jk_ajp_common.c (1359): (cluster01) connecting to backend
failed. Tomcat is probably not started or is listening on the wrong port
(errno=110)
...
[Mon Jan 21 08:48:40 2008][20105:1335986528] [error]
ajp_get_reply::jk_ajp_common.c (1766): (cluster01) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Mon Jan 21 08:48:40 2008][20629:1157658976] [error]
ajp_get_reply::jk_ajp_common.c (1766): (cluster01) Tomcat is down or refused
connection. No response has been sent to the client (yet)
There is absolutelly no indication, that mod_jk is unsuccessfuly trying to
connect cluster02 at this point, there are no errors related to cluster02
(except couple of requests at the beginning of the tests).
Thanks,
Dominik
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Rainer Jung <ra...@kippdata.de>.
... adding to previous post:
It also does not retry/fail over, in the following (more or less obvious
cases):
- there was a problem with the client (browser) connection
- we already started to send back parts of the response to the client,
typically the headers (depending on recovery_options; default:
retry/fail over)
- Internal JK error (not observed in practise yet)
- depending on recovery_options: if the backend received the request
(default: retry/fail over)
Regards,
Rainer
Dominik Pospisil wrote:
>> Indeed. It would be OK to return 503, for requests, that already have
>> been received by the first node, but not returned yet. New requests
>
> That's the case. All errors I am getting are from requests which were allready
> in processing by failing node. Why it is OK to return 503 for such requests?
> Why after mod_jk detect failure does not redirect them to second node? I have
> recovery_option attribute set to zero.
>
> I am sorry for disturbing you with questins again, but it is really essential
> for me to learn how mod_jk failover works in detail.
>
> Thanks again,
>
> Dominik
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Rainer Jung <ra...@kippdata.de>.
Dominik Pospisil wrote:
>> Indeed. It would be OK to return 503, for requests, that already have
>> been received by the first node, but not returned yet. New requests
>
> That's the case. All errors I am getting are from requests which were allready
> in processing by failing node. Why it is OK to return 503 for such requests?
> Why after mod_jk detect failure does not redirect them to second node? I have
> recovery_option attribute set to zero.
>
> I am sorry for disturbing you with questins again, but it is really essential
> for me to learn how mod_jk failover works in detail.
Let's see, if this matches your experience: assuming recovery_options=0,
the request should be retried on other lb member workers (if such exist
and are OK), unless it's a POST and the POST body is bigger than 8186
bytes and more than the first 8186 bytes have already been sent to the
backend.
Why doesn't it get retried in this case? Because we only buffer that
many data for retry.
POST bodies can get very big (uploads), so it's not good to buffer
complete POST bodies (could take a lot of memory). AJP tries to stream
packets through the web server to the backend as they arrive. The actual
design simply decides to buffer one ajp packet, which is 8192 bytes
minus ajp head = 8186 bytes.
Does that fit your observations?
>
> Thanks again,
>
> Dominik
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Dominik Pospisil <dp...@redhat.com>.
> Indeed. It would be OK to return 503, for requests, that already have
> been received by the first node, but not returned yet. New requests
That's the case. All errors I am getting are from requests which were allready
in processing by failing node. Why it is OK to return 503 for such requests?
Why after mod_jk detect failure does not redirect them to second node? I have
recovery_option attribute set to zero.
I am sorry for disturbing you with questins again, but it is really essential
for me to learn how mod_jk failover works in detail.
Thanks again,
Dominik
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Dominik,
>>> Eventually, after killing 1st node, and after returning couple of "503
>>> Service Temporarily Unavailable" exceptions, mod_jk finally recheck 2nd
>>> node status, reroute requests to 2nd node and resumes correct operation.
>>>
>
> Still, I it is not clear to me, why I am getting 503 exceptions. I believe
> that when there is one or more servers up and ready to serve request, this
> should not happend. Why the requests are not immidiatelly rerouted to second
> node, which is couple of minutes up and running (in ERR/REC state) ?
Indeed. It would be OK to return 503, for requests, that already have
been received by the first node, but not returned yet. New requests
should not run into errors though. But: it also depends on the type of
node failure, more precisely on its exact behaviour on the network. To
make problem detection more robust, you can especially enable connect
and prepost timeout. See
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
I can imagine, that with your default configuration (timeouts disabled)
you might run into long TCP timeouts and e.g. the httpd default timeout
of 5 minutes or an overload situation because to many requests where
waiting to long results in the error.
If cping/cpong doesn't help: To find out the reason for the 503, you can
have a look at the JK log file. Look for log lines with "[error]", only
those indicate final errors. To find out about more detailed reasons of
those errors, it might help to look at the associated "[info]" log
lines. Associated means shortly before in time and having the same
process and thread id "[pid:tid]".
>
> Thanks,
>
> Dominik
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Dominik Pospisil <dp...@redhat.com>.
Hi Rainer,
thanks a lot for prompt response.
> Dominik Pospisil wrote:
> > Hello,
> > I am having following problem with following failover test scenario.
> >
> > Cluster setup:
> > - 1 apache load balancer
> > - 2 nodes with equal LB factor
> > - sticky session turned on
> > - Apache/2.0.52, mod_jk/1.2.26
> >
> > Test scenario:
> > 1. start 1st node
> > 2. start load driver
> > 3. start 2nd node
> > 4. wait for state transfer (2 minutes)
> > 5. kill 1st node
> >
> > My experience is that after stage 1 and 2, all clients are handled
> > correctly by 1st node and the second node is set correctly to ERR state.
> > After while, the second none switches to ERR/REC state.
> >
> > However at stage 4 (after starting 2nd node) the second node will never
> > come up to OK state. I have set both worker maintain period and LB
> > recovery_time to 30s. So i guess that in 2 minutes, the second node
> > should have been re-checked. When I press manually "Reset worker state"
> > button, it comes up immediatelly, but it never happend automatically
> > during maintenance phase.
>
> I would expect, that your load driver only send sticky requests, i.e.
> requests with eiher cookie or URL encoding for node cluster01. At least
> that would fit to your observation.
>
> mod_jk detects during maintenance, if a worker was in error state long
> enough to try again. This happens in your setup, as you can see by the
> ERR/REC state. The next request that comes in *and does not contain a
> session id of another node* will be routed to the REC node. Under load,
> you won't see this state often, because most of the time it should turn
> into ERR or OK very quick.
>
Hmm, I see. But I would not agree that under load this would necessairly turn
ERR or OK quickly. I am also generating heavy load, just there are no
clients / sessions arriving. This could happend in real screnario too.
> Maybe your app sets a cookie and the load driver always presrnts that
> cookie. That way all further requests would be handled as sticky and
> routed to the first node.
>
> You can find out by logging %{Cookie}i in your httpd access log. If you
> include this in your LogFormat, you can see the incoming Cookie header
> for each request.
Yes, that's true, all of my clients are initialized at the beginning of the
test and corresponding sessions are created.
> > Eventually, after killing 1st node, and after returning couple of "503
> > Service Temporarily Unavailable" exceptions, mod_jk finally recheck 2nd
> > node status, reroute requests to 2nd node and resumes correct operation.
> >
Still, I it is not clear to me, why I am getting 503 exceptions. I believe
that when there is one or more servers up and ready to serve request, this
should not happend. Why the requests are not immidiatelly rerouted to second
node, which is couple of minutes up and running (in ERR/REC state) ?
Thanks,
Dominik
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk maintenance, recovery
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Dominik,
Dominik Pospisil wrote:
> Hello,
> I am having following problem with following failover test scenario.
>
> Cluster setup:
> - 1 apache load balancer
> - 2 nodes with equal LB factor
> - sticky session turned on
> - Apache/2.0.52, mod_jk/1.2.26
>
> Test scenario:
> 1. start 1st node
> 2. start load driver
> 3. start 2nd node
> 4. wait for state transfer (2 minutes)
> 5. kill 1st node
>
> My experience is that after stage 1 and 2, all clients are handled correctly
> by 1st node and the second node is set correctly to ERR state. After while,
> the second none switches to ERR/REC state.
>
> However at stage 4 (after starting 2nd node) the second node will never come
> up to OK state. I have set both worker maintain period and LB recovery_time
> to 30s. So i guess that in 2 minutes, the second node should have been
> re-checked. When I press manually "Reset worker state" button, it comes up
> immediatelly, but it never happend automatically during maintenance phase.
I would expect, that your load driver only send sticky requests, i.e.
requests with eiher cookie or URL encoding for node cluster01. At least
that would fit to your observation.
mod_jk detects during maintenance, if a worker was in error state long
enough to try again. This happens in your setup, as you can see by the
ERR/REC state. The next request that comes in *and does not contain a
session id of another node* will be routed to the REC node. Under load,
you won't see this state often, because most of the time it should turn
into ERR or OK very quick.
Maybe your app sets a cookie and the load driver always presrnts that
cookie. That way all further requests would be handled as sticky and
routed to the first node.
You can find out by logging %{Cookie}i in your httpd access log. If you
include this in your LogFormat, you can see the incoming Cookie header
for each request.
> Eventually, after killing 1st node, and after returning couple of "503 Service
> Temporarily Unavailable" exceptions, mod_jk finally recheck 2nd node status,
> reroute requests to 2nd node and resumes correct operation.
>
> My question is: Why the second node is not recognized before failover? Did I
> missed something? Or is it a bug?
>
> Thanks,
>
> - Dominik
>
> Attaching worker.properties
> ---------
> worker.list=loadbalancer,status
> worker.maintain=30
>
> # modify the host as your host IP or DNS name.
> worker.cluster01.port=8009
> worker.cluster01.host=172.17.0.39
> worker.cluster01.type=ajp13
> worker.cluster01.lbfactor=1
> #worker.cluster01.redirect=cluster02
> # modify the host as your host IP or DNS name.
> worker.cluster02.port=8009
> worker.cluster02.host=172.17.1.39
> worker.cluster02.type=ajp13
> worker.cluster02.lbfactor=1
> #worker.cluster02.redirect=cluster01
> # modify the host as your host IP or DNS name.
>
> # Load-balancing behaviour
> worker.loadbalancer.type=lb
> worker.loadbalancer.method=Session
> worker.loadbalancer.balance_workers=cluster01,cluster02
> worker.loadbalancer.sticky_session=1
> worker.loadbalancer.recover_time=30
>
> #worker.list=loadbalancer
> # Status worker for managing load balancer
> worker.status.type=status
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org