You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Artur <ar...@upi.pl> on 2006/08/08 16:37:03 UTC
Mod_jk balancing, session problem
Hello,
We have environment with Apache (2.2.0) + Tomcat (5.5.17) connected via
mod_jk (1.2.18). Further tomcat makes some queries to databese through
Tuxedo services (BEA).
Everything works fine until heavy load when something stops responding
(Service not available).
We think it is tomcat or mod_jk. It occures when "Max busy connections" on
mod_jk status manager shows about 200. Mod_jk is in error status then.
So we decided to use mod_jk load balancing feature and we installed another
instance of tomcat (on the same machine).
Now service is available even under heavy load but we are losing sessions.
Of course sticky_session is set to "true".
Below I paste our config files. Please help if you only can.
Best regards
Artur
###Jk status for one day:
Name Type jvmRoute Host Addr Stat D F M V Acc Err Wr Rd Busy Max RR Cd
worker2 ajp13 worker2 localhost:8009 127.0.0.1:8009 OK 0 1 1 2144 824825 512
480M 5.8G 6 138 worker3
###httpd.mpm section:
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
StartServers 5
MaxClients 448
MinSpareThreads 5
MaxSpareThreads 25
ThreadsPerChild 7
MaxRequestsPerChild 0
</IfModule>
###workers.properties:
# Define 3 workers, 2 real workers using ajp13 and the last one being a
loadbalancing worker
worker.list=worker1,jkstatus
# Set properties for worker2 (ajp13)
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=8009
worker.worker2.lbfactor=1
worker.worker2.connection_pool_size=7
worker.worker2.connection_pool_timeout=300
worker.worker2.socket_timeout=60
# Set properties for worker4 (lb) which use worker1 and worker2
worker.worker3.type=ajp13
worker.worker3.host=localhost
worker.worker3.port=8019
worker.worker3.lbfactor=1
worker.worker3.connection_pool_size=7
worker.worker3.connection_pool_timeout=300
worker.worker3.socket_timeout=60
# Define a 'jkstatus' worker using status
worker.jkstatus.type=status
worker.worker1.type=lb
worker.worker1.sticky_session=true
worker.worker1.method=R
worker.worker1.balance_workers=worker2,worker3
###server1.xml - connector section:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker2">
###server2.xml - connector section
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8019"
enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker3">
--
View this message in context: http://www.nabble.com/Mod_jk-balancing%2C-session-problem-tf2073073.html#a5707787
Sent from the Tomcat - User forum at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Mod_jk balancing, session problem
Posted by Artur <ar...@upi.pl>.
>Choose the suggested access log methods to improve observability (URLs
>and Cookie and Set-Cookie Headers.
We make some tests with error log set to debug, but it is so many
informations in the log (even the decrypted password !!, should it be so ??)
that it may take some time to analyse everything.
But here I have another question about the errors reported by JK status
manager.
In our production environment we have log level set to "error" and there are
about 200 requests per worker shown by JK status manager as error , but
there isn't any line in the mod_jk.log with "error".
Why is that so ?
Another question is about tomcat catalina.out. Please find below a cut from
log:
2006-08-10 06:17:20 org.apache.jk.core.MsgContext action
WARNING: Error sending end packet
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)
at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
2006-08-10 06:17:20 org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
2006-08-10 08:38:53 org.apache.jk.core.MsgContext action
WARNING: Error sending end packet
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)
at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
2006-08-10 08:38:53 org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
2006-08-10 08:57:51 org.apache.jk.core.MsgContext action
WARNING: Error sending end packet
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)
at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
2006-08-10 08:57:51 org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Do you have any idea what my be the reason ?
Best regards
Artur
--
View this message in context: http://www.nabble.com/Mod_jk-balancing%2C-session-problem-tf2073073.html#a5765997
Sent from the Tomcat - User forum at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Mod_jk balancing, session problem
Posted by Rainer Jung <ra...@kippdata.de>.
Hi,
Artur schrieb:
>> - does it happen in times, when you get errors in the mod_jk-log? Does
>> it happen at all times, or do you have peaks at special times?
>>
>> I can't figure it out, when this happens, but there isn't any special
>> moment I can observe.
>> It happens randomly. There isn't any error in mod_jk.log
Choose the suggested access log methods to improve observability (URLs
and Cookie and Set-Cookie Headers.
>> And here I need some help. In mod_jk.log I always see the same worker -
>> balancer worker, so I don't know which worker has been choosen. How can I
>> change it ? I assume by adding request log to mod_jk you mean something
>> like this:
>>
>> # JkRequestLogFormat set the request format
>> JkRequestLogFormat "%w %V %T"
OK, unfortunately mod_jk does not yet allow to log the sub worker of an
lb. I'll check for improvement in 1.2.19...
>>> <IfModule mpm_worker_module>
>>> StartServers 5
>>> MaxClients 448
>>> MinSpareThreads 5
>>> MaxSpareThreads 25
>>> ThreadsPerChild 7
>>> MaxRequestsPerChild 0
>>> </IfModule>
>> Interesting, very few ThreadsPerChild. Not really a problem, but I
>> wonder if this is efficient.
>>
>> We have decresed this value because we thought that maybe Apache is
>> opening too many threads to Tomcat. We had in Tomcat's log entries saying"
>> "All threads (500) are curently busy, increase maxThreds".
Threads in Tomcat = Sum of numbers of apache threads used during peak
times. So this relates more to MaxClients and not to ThreadsPerChild.
If you want to connect many Apaches with a lot of threads to tomcat you
need to use tricks like the APR connector on the tomcat side. A few
hundred threads should be no problem, on a good OS even 1.000-2.000
should work.
>>
>> Think about using connect_timeout and prepost_timeout. Maybe also set
>> recovery_options to 3.
>>
>> What values do you suggest for above parameters ?
connect_timeout=20000
This is in Milliseconds. Anything between 2000 and 20000 should be OK.
The right value depends on your trust in latency, reliability of your
network and the responsiveness of the target machine.
prepost_timeout=10000
Values between 5000 and 30000 are reasonable. Arguments the same as before.
recovery_options=3 or 7 (here you'll need to experimnt on your own. The
added "4" is very new, but maybe helpful).
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Mod_jk balancing, session problem
Posted by Artur <ar...@upi.pl>.
Rainer Jung-3 wrote:
>
> What do you mean by "losing sessions"? I expect you mean users work in a
> session and then they send a request and get a response indicating, that
> the requested session has not been found?
>
> Thats exactly what I meant.
>
> If so:
>
> - does it happen in times, when you get errors in the mod_jk-log? Does
> it happen at all times, or do you have peaks at special times?
>
> I can't figure it out, when this happens, but there isn't any special
> moment I can observe.
> It happens randomly. There isn't any error in mod_jk.log
>
> - if it happens even when there's no other problem at the same time:
>
> You can also add a request log to mod_jk, that logs, which worker has
> beend chosen for every request. That way you do not rely on your user
> feedback, but can measure the problem yourself (how many users/sessions,
> what time of day etc.).
>
> And here I need some help. In mod_jk.log I always see the same worker -
> balancer worker, so I don't know which worker has been choosen. How can I
> change it ? I assume by adding request log to mod_jk you mean something
> like this:
>
> # JkRequestLogFormat set the request format
> JkRequestLogFormat "%w %V %T"
>
>
>
>> Of course sticky_session is set to "true".
>> Below I paste our config files. Please help if you only can.
>>
>> Best regards
>> Artur
>>
>> ###Jk status for one day:
>>
>> Name Type jvmRoute Host Addr Stat D F M V Acc Err Wr Rd Busy Max RR Cd
>> worker2 ajp13 worker2 localhost:8009 127.0.0.1:8009 OK 0 1 1 2144 824825
>> 512
>> 480M 5.8G 6 138 worker3
>
> worker3 is in the original output and has iónly been removed by your
> copy&paste?
>
> Right
>
> What was happening, when you got these 2144 Errors? The time stamps
> should be available from the mod_jk log.
>
> "Loosing session" occurs even when there arnen't any errors in mod_jk
> status manager, it just looks like with next request you have to "log in"
> once more.
>
>
>> ###httpd.mpm section:
>> # worker MPM
>> # StartServers: initial number of server processes to start
>> # MaxClients: maximum number of simultaneous client connections
>> # MinSpareThreads: minimum number of worker threads which are kept spare
>> # MaxSpareThreads: maximum number of worker threads which are kept spare
>> # ThreadsPerChild: constant number of worker threads in each server
>> process
>> # MaxRequestsPerChild: maximum number of requests a server process serves
>>
>> <IfModule mpm_worker_module>
>> StartServers 5
>> MaxClients 448
>> MinSpareThreads 5
>> MaxSpareThreads 25
>> ThreadsPerChild 7
>> MaxRequestsPerChild 0
>> </IfModule>
>
> Interesting, very few ThreadsPerChild. Not really a problem, but I
> wonder if this is efficient.
>
> We have decresed this value because we thought that maybe Apache is
> opening too many threads to Tomcat. We had in Tomcat's log entries saying"
> "All threads (500) are curently busy, increase maxThreds".
>
> Think about using connect_timeout and prepost_timeout. Maybe also set
> recovery_options to 3.
>
> What values do you suggest for above parameters ?
>
>
--
View this message in context: http://www.nabble.com/Mod_jk-balancing%2C-session-problem-tf2073073.html#a5710742
Sent from the Tomcat - User forum at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Mod_jk balancing, session problem
Posted by Rainer Jung <ra...@kippdata.de>.
Artur wrote:
> Hello,
>
> We have environment with Apache (2.2.0) + Tomcat (5.5.17) connected via
> mod_jk (1.2.18). Further tomcat makes some queries to databese through
> Tuxedo services (BEA).
> Everything works fine until heavy load when something stops responding
> (Service not available).
> We think it is tomcat or mod_jk. It occures when "Max busy connections" on
> mod_jk status manager shows about 200. Mod_jk is in error status then.
Understanding this situation would be helpful to decide, if adding
another tomcat really helps (problem=database => more tomcats don't help
etc.).
>
> So we decided to use mod_jk load balancing feature and we installed another
> instance of tomcat (on the same machine).
> Now service is available even under heavy load but we are losing sessions.
What do you mean by "losing sessions"? I expect you mean users work in a
session and then they send a request and get a response indicating, that
the requested session has not been found?
If so:
- does it happen in times, when you get errors in the mod_jk-log? Does
it happen at all times, or do you have peaks at special times?
Stickyness sends users to the correct node, except when the node is not
reachable (error, stopped). In this case users will be sent to one of
the remaining nodes and - of course - without session replication their
session will not be there. Session repllication means to establish a
tomcat clusster, something you need to carefully test and that might
force changes in your app.
- if it happens even when there's no other problem at the same time:
make session IDs trackable. Either your app uses URL rewriting, then
your session ID is part of every request and logged in the apache access
log. Or you use Cookies, then add logging of incoming header "Cookie" to
your mod_log_config LogFormat. If you also add outgoing header
"Set-Cookie" you are able to follow the sessions, and you can decide, if
the users really send the correct session ID. Sometimes what you may
find is: the previous request produced an error in the app, which then
decided to invalidate the session.
You can also add a request log to mod_jk, that logs, which worker has
beend chosen for every request. That way you do not rely on your user
feedback, but can measure the problem yourself (how many users/sessions,
what time of day etc.).
> Of course sticky_session is set to "true".
> Below I paste our config files. Please help if you only can.
>
> Best regards
> Artur
>
> ###Jk status for one day:
>
> Name Type jvmRoute Host Addr Stat D F M V Acc Err Wr Rd Busy Max RR Cd
> worker2 ajp13 worker2 localhost:8009 127.0.0.1:8009 OK 0 1 1 2144 824825 512
> 480M 5.8G 6 138 worker3
worker3 is in the original output and has iónly been removed by your
copy&paste?
What was happening, when you got these 2144 Errors? The time stamps
should be available from the mod_jk log.
> ###httpd.mpm section:
> # worker MPM
> # StartServers: initial number of server processes to start
> # MaxClients: maximum number of simultaneous client connections
> # MinSpareThreads: minimum number of worker threads which are kept spare
> # MaxSpareThreads: maximum number of worker threads which are kept spare
> # ThreadsPerChild: constant number of worker threads in each server process
> # MaxRequestsPerChild: maximum number of requests a server process serves
>
> <IfModule mpm_worker_module>
> StartServers 5
> MaxClients 448
> MinSpareThreads 5
> MaxSpareThreads 25
> ThreadsPerChild 7
> MaxRequestsPerChild 0
> </IfModule>
Interesting, very few ThreadsPerChild. Not really a problem, but I
wonder if this is efficient.
>
> ###workers.properties:
> # Define 3 workers, 2 real workers using ajp13 and the last one being a
> loadbalancing worker
>
> worker.list=worker1,jkstatus
>
> # Set properties for worker2 (ajp13)
> worker.worker2.type=ajp13
> worker.worker2.host=localhost
> worker.worker2.port=8009
> worker.worker2.lbfactor=1
> worker.worker2.connection_pool_size=7
This should be set top the same value as ThreadsPerChild by mod_jk
automatically.
> worker.worker2.connection_pool_timeout=300
> worker.worker2.socket_timeout=60
Think about using connect_timeout and prepost_timeout. Maybe also set
recovery_options to 3.
>
> # Set properties for worker4 (lb) which use worker1 and worker2
>
> worker.worker3.type=ajp13
> worker.worker3.host=localhost
> worker.worker3.port=8019
> worker.worker3.lbfactor=1
> worker.worker3.connection_pool_size=7
> worker.worker3.connection_pool_timeout=300
> worker.worker3.socket_timeout=60
>
>
> # Define a 'jkstatus' worker using status
> worker.jkstatus.type=status
>
> worker.worker1.type=lb
> worker.worker1.sticky_session=true
> worker.worker1.method=R
> worker.worker1.balance_workers=worker2,worker3
>
>
> ###server1.xml - connector section:
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector port="8009"
> enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
> />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="worker2">
>
>
> ###server2.xml - connector section
> <!-- Define an AJP 1.3 Connector on port 8009 -->
>
>
> <Connector port="8019"
> enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
> />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="worker3">
regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org