You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Emmanuel Courreges <ec...@orb.com> on 2007/01/20 03:44:53 UTC
mod_jk request freeze with load balancing
Hello all,
Some answers to requests freeze and never return, which is a problem for
setting up the load balancing in production environment. Usually if the
user refreshes and the request is made again it doesn't freeze again, so
it's pretty random.
This seems to happen only:
- when the answer is big (>16kBytes)
- with the balanced tomcat which is not on localhost
My ethereal and mod_jk traces (cf. later) show me that data for the next
chunk is received by the server but mod_jk doesn't see it.
My setup is basically 2 Redhat Enterprise 4 32bits up to date,
- mod_jk 1.2.20 compiled by myself
- tomcat 5.5.17
- jdk1.6
Apache+mod_jk load balances to the local server and to a remote server.
port 8009 is open in iptables and they are on the same switch AND data
keeps getting received so a network/firewall thing is not the issue
here. It balances well 50/50 except that some requests that get balanced
to the remote server never return.
I have noticed this problem for more than a year now (with older
versions of everything), which is why I did hold off on load balancing.
Does anyone experience the same problem/have a solution?
Thank you!
Emmanuel Courreges.
The first chunks arrive fine, then you'll see from the last characters
of the last chunk displayed in mod_jk (e.g. search for QOe8), that it's
not the last thing sent at all in this tcp connection.
It's definitely blocked somewhere, because when the request works fine I
get a message: "AJP13 protocol: Reuse is OK" at the end, and here
nothing happens after this chunk.
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (893): 0300 62 37 00
FF 00 00 00 00 00 00 00 00 00 00
00 00 - b7..............
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_send_request::jk_ajp_common.c (1273): request body to send 0 -
request body to resend 0
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from
ajp13 pos=0 len=146 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000 04 00 C8
00 02 4F 4B 00 00 05 00 0D 43 61
63 68 - .....OK.....Cach
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0010 65 2D 43
6F 6E 74 72 6F 6C 00 00 11 6E 6F
2D 63 - e-Control...no-c
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0020 61 63 68
65 2C 20 70 72 69 76 61 74 65 00
00 06 - ache,.private...
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0030 50 72 61
67 6D 61 00 00 08 4E 6F 2D 43 61
63 68 - Pragma...No-Cach
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0040 65 00 00
07 45 78 70 69 72 65 73 00 00 02
2D 31 - e...Expires...-1
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0050 00 00 0C
43 6F 6E 74 65 6E 74 2D 54 79 70
65 00 - ...Content-Type.
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0060 00 17 74
65 78 74 2F 68 74 6D 6C 3B 63 68
61 72 - ..text/html;char
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0070 73 65 74
3D 55 54 46 2D 38 00 00 10 43 6F
6E 74 - set=UTF-8...Cont
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0080 65 6E 74
2D 4C 61 6E 67 75 61 67 65 00 00
02 65 - ent-Language...e
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0090 6E 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 - n...............
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (603): status = 200
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (610): Number of headers is = 5
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[0] [Cache-Control]
= [no-cache, private]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[1] [Pragma] =
[No-Cache]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[2] [Expires] = [-1]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[3] [Content-Type]
= [text/html;charset=UTF-8]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[4]
[Content-Language] = [en]
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from
ajp13 pos=0 len=8188 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000 03 1F F8
3C 68 74 6D 6C 3E 0D 0A 3C 68 65
61 64 - ...<html>..<head
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0010 3E 0D 0A
09 3C 74 69 74 6C 65 3E 4F 72 62
3C 2F - >...<title>Orb</
...
7F 20 - +..!G&5sSJT?p=..
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03f0 42 3D 7F
20 28 3E 33 7F 20 40 3D 4F 7F 20
21 3F - B=..(>3..@=O..!?
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] ws_write::mod_jk.c
(436): written 8184 out of 8184
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): received from
ajp13 pos=0 len=8188 max=8192
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 0000 03 1F F8
65 76 65 39 68 21 74 62 3E 40 5B
28 7F - ...eve9h!tb>@[(.
...
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03c0 2A 3D 4C
3E 24 76 47 7A 71 71 55 3D 2C 29
3A 31 - *=L>$vGzqqU=,):1
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03d0 55 7F 39
5F 3B 4D 6A 6C 4F 22 71 34 32 7F
24 5C - U.9_;MjlO"q42.$\
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03e0 40 23 5E
32 32 29 34 3D 32 2C 7A 3E 3F 5C
67 35 - @#^22)4=2,z>?\g5
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1029): 03f0 64 3E 5C
51 4F 65 38 23 34 4E 23 26 31 28
39 3F - d>\QOe8#4N#&1(9?
[Fri Jan 19 13:05:10 2007] [11579:64192] [debug] ws_write::mod_jk.c
(436): written 8184 out of 8184
ETHEREAL:
45311 13:05:10.600576 208.96.4.147 -> 208.96.4.140 AJP13 143:RSP:SEND
BODY CHUNK[Unreassembled Packet]
0000 00 11 11 3e c5 8d 00 30 48 56 4e c6 08 00 45 00 ...>...0HVN...E.
0010 05 dc 0f d7 40 00 40 06 7b 65 d0 60 04 93 d0 60 ....@.@.{e.`...`
0020 04 8c 1f 49 98 1a b7 2d 80 27 05 b6 71 2b 80 10 ...I...-.'..q+..
0030 21 4f ce cd 00 00 01 01 08 0a c4 8d 79 10 d8 0f !O..........y...
0040 c2 2d 41 42 1f fc 03 1f f8 65 76 65 39 68 21 74 .-AB.....eve9h!t
0050 62 3e 40 5b 28 7f 63 6a 59 5d 24 26 6d 21 54 37 b>@[(.cjY]$&m!T7
0060 27 3d 38 2e 69 55 65 2e 54 51 55 42 2f 32 6d 51 '=8.iUe.TQUB/2mQ
0070 28 25 55 76 38 20 73 21 2d 26 3b 65 27 29 63 29 (%Uv8 s!-&;e')c)
0080 2b 45 21 21 56 34 77 2f 3e 62 2b 29 36 2c 6a 38 +E!!V4w/>b+)6,j8
0090 5c 27 39 24 7f 20 4a 2a 33 24 2a 28 4d 5a 47 7d \'9$. J*3$*(MZG}
00a0 47 38 5a 32 25 37 2e 2f 34 25 3e 7c 70 38 4b 42 G8Z2%7./4%>|p8KB
00b0 24 46 26 6c 71 31 31 3c 67 71 26 46 3c 79 30 3e $F&lq11<gq&F<y0>
00c0 25 28 3a 34 2d 33 66 6c 7f 79 3d 2b 2c 4c 70 5f %(:4-3fl.y=+,Lp_
00d0 35 24 55 3a 7c 31 70 33 39 41 6e 26 2e 65 20 62 5$U:|1p39An&.e b
00e0 22 33 23 3e 36 34 72 7e 6e 3a 36 6d 24 3b 23 2c "3#>64r~n:6m$;#,
00f0 2b 6b 77 2c 7b 3a 5d 24 24 20 3a 3a 2f 66 2b 7a +kw,{:]$$ ::/f+z
0100 53 23 43 2a 27 37 2b 3d 38 35 5b 36 7b 3a 51 42 S#C*'7+=85[6{:QB
0110 4b 6e 37 2a 5a 49 5e 4e 42 21 31 73 47 4f 3a 33 Kn7*ZI^NB!1sGO:3
0120 57 62 69 39 22 49 62 33 77 68 39 60 25 4a 67 27 Wbi9"Ib3wh9`%Jg'
0130 27 71 3d 79 67 2a 71 6b 57 27 65 25 3d 59 6b 46 'q=yg*qkW'e%=YkF
0140 55 29 24 7b 2e 3e 58 56 40 47 66 2a 79 2c 31 3e U)${.>XV@Gf*y,1>
0150 77 63 5b 3d 6b 36 46 40 63 20 48 32 45 7f 20 5f wc[=k6F@c H2E. _
0160 36 3d 2c 4f 50 35 27 6e 2f 51 58 52 3b 5c 4d 3a 6=,OP5'n/QXR;\M:
0170 55 27 2a 55 35 7b 2b 71 24 2d 24 26 35 34 38 63 U'*U5{+q$-$&548c
0180 67 4c 6a 54 33 68 4c 78 77 48 2a 76 38 29 64 72 gLjT3hLxwH*v8)dr
0190 27 39 29 20 39 74 2e 45 2f 24 23 54 2f 67 5b 42 '9) 9t.E/$#T/g[B
01a0 56 70 26 77 31 2d 52 52 24 39 39 2b 42 38 2a 2a Vp&w1-RR$99+B8**
01b0 44 2e 44 73 20 25 40 6b 3f 71 29 56 72 2c 25 27 D.Ds %@k?q)Vr,%'
01c0 44 57 34 28 66 64 3b 21 39 60 7b 71 31 5c 5e 6d DW4(fd;!9`{q1\^m
01d0 36 7d 24 37 3a 23 4d 24 25 6d 24 37 6b 4d 24 2c 6}$7:#M$%m$7kM$,
01e0 55 43 39 28 30 22 2f 39 2e 41 2c 3f 51 57 35 61 UC9(0"/9.A,?QW5a
01f0 2a 21 2d 3f 5a 39 44 3f 3c 2c 29 20 77 27 3d 61 *!-?Z9D?<,) w'=a
0200 29 27 5a 2b 3a 32 3a 65 37 28 29 41 3c 56 34 24 )'Z+:2:e7()A<V4$
0210 5c 33 25 47 5a 64 2b 27 2f 66 30 2e 38 7e 68 77 \3%GZd+'/f0.8~hw
0220 2d 3d 23 4a 48 21 35 4c 4e 62 2d 5b 50 2d 68 47 -=#JH!5LNb-[P-hG
0230 63 5c 60 3f 4a 3b 62 2a 3c 3a 26 55 4e 3a 36 78 c\`?J;b*<:&UN:6x
0240 2e 54 75 2e 37 3b 72 24 4b 2b 29 36 3b 28 3c 34 .Tu.7;r$K+)6;(<4
0250 24 36 39 61 4f 20 7d 55 35 2e 22 24 5b 32 39 64 $69aO }U5."$[29d
0260 42 26 22 3b 42 35 23 54 77 2f 2a 2f 5c 64 47 33 B&";B5#Tw/*/\dG3
0270 4e 3a 2b 79 2e 23 2e 45 55 36 39 77 6e 37 6c 4e N:+y.#.EU69wn7lN
0280 2a 5b 6c 4b 2e 24 7f 31 53 29 40 29 3e 59 22 31 *[lK.$.1S)@)>Y"1
0290 22 2a 21 30 32 2c 60 3e 25 47 7f 7a 29 54 27 22 "*!02,`>%G.z)T'"
02a0 51 5a 21 53 67 27 26 53 39 63 4c 25 41 68 54 49 QZ!Sg'&S9cL%AhTI
02b0 24 60 43 76 36 66 51 30 39 4e 41 61 2e 7a 3a 2b $`Cv6fQ09NAa.z:+
02c0 70 40 2c 30 48 31 32 3c 38 4f 34 4d 76 31 54 64 p@,0H12<8O4Mv1Td
02d0 2f 3a 20 2f 4e 39 54 39 6a 6b 39 24 6a 37 4d 73 /: /N9T9jk9$j7Ms
02e0 2d 4b 76 31 7d 27 74 3e 53 3c 59 65 35 77 35 44 -Kv1}'t>S<Ye5w5D
02f0 51 5d 21 64 73 5b 5a 26 2f 49 5d 30 45 6c 56 50 Q]!ds[Z&/I]0ElVP
0300 2f 34 2e 4e 6a 2e 2e 7a 2f 7e 7a 3a 4a 50 7d 6b /4.Nj..z/~z:JP}k
0310 6b 53 33 3e 4e 32 2d 29 20 2d 22 34 47 3c 2b 30 kS3>N2-) -"4G<+0
0320 26 5f 21 38 6b 33 6b 2f 7a 30 32 79 30 2c 49 3b &_!8k3k/z02y0,I;
0330 34 31 3b 41 2b 4e 2b 32 23 23 2c 72 7d 2a 27 52 41;A+N+2##,r}*'R
0340 32 4a 2c 21 2a 35 27 48 5d 74 31 2f 39 5a 69 22 2J,!*5'H]t1/9Zi"
0350 4b 71 39 78 46 54 39 76 6c 5c 73 48 6c 2a 44 30 Kq9xFT9vl\sHl*D0
0360 47 75 3f 59 59 64 65 49 2b 5b 5d 6f 20 30 3d 7d Gu?YYdeI+[]o 0=}
0370 4f 78 57 6c 63 75 34 56 28 4a 3b 3d 39 2a 6d 6e OxWlcu4V(J;=9*mn
0380 2d 42 46 3e 66 65 26 31 22 3f 39 20 20 37 62 36 -BF>fe&1"?9 7b6
0390 7d 68 20 2f 6e 38 3a 5c 23 2b 3f 2d 26 70 3e 3f }h /n8:\#+?-&p>?
03a0 52 3a 2a 72 2f 2a 54 5a 32 55 76 3a 24 7e 43 5c R:*r/*TZ2Uv:$~C\
03b0 52 32 2c 76 6e 43 4d 2f 3c 21 24 46 33 75 47 3f R2,vnCM/<!$F3uG?
03c0 4e 2a 34 2b 2f 39 36 52 3f 3c 49 39 7b 42 41 27 N*4+/96R?<I9{BA'
03d0 36 26 3b 32 7a 73 49 2f 2a 35 2c 64 65 72 2a 34 6&;2zsI/*5,der*4
03e0 5f 3a 34 39 39 66 56 3d 6d 2b 23 31 36 2e 45 5e _:499fV=m+#16.E^
03f0 64 33 73 30 3d 7f 20 3a 75 68 64 7f 20 29 32 32 d3s0=. :uhd. )22
0400 41 22 4b 65 37 6e 2a 3d 4c 3e 24 76 47 7a 71 71 A"Ke7n*=L>$vGzqq
0410 55 3d 2c 29 3a 31 55 7f 39 5f 3b 4d 6a 6c 4f 22 U=,):1U.9_;MjlO"
0420 71 34 32 7f 24 5c 40 23 5e 32 32 29 34 3d 32 2c q42.$\@#^22)4=2,
0430 7a 3e 3f 5c 67 35 64 3e 5c 51 4f 65 38 23 34 4e z>?\g5d>\QOe8#4N
0440 23 26 31 28 39 3f 79 5b 2e 4d 7e 25 30 25 4a 35 #&1(9?y[.M~%0%J5
0450 7b 3f 38 71 67 35 27 29 4e 5d 4f 44 50 6d 56 20 {?8qg5')N]ODPmV
0460 40 2d 20 48 30 65 5f 36 21 29 23 21 7f 29 2b 2e @- H0e_6!)#!.)+.
0470 4d 26 72 30 49 25 33 3f 78 78 66 76 55 6c 2f 3c M&r0I%3?xxfvUl/<
0480 23 74 7a 3a 62 4c 35 27 28 32 3b 3a 3e 3f 24 34 #tz:bL5'(2;:>?$4
0490 5d 25 20 49 5e 35 36 63 65 36 7c 6d 2f 66 36 24 ]% I^56ce6|m/f6$
04a0 23 5b 3c 23 51 45 73 3b 61 20 2a 50 4b 3b 7e 63 #[<#QEs;a *PK;~c
04b0 2c 4e 3c 21 5e 2a 4a 4f 4d 31 6e 55 2b 2c 54 6d ,N<!^*JOM1nU+,Tm
04c0 23 23 33 40 4b 4a 37 2a 71 3f 3d 24 6c 4e 47 5c ##3@KJ7*q?=$lNG\
04d0 63 52 32 5b 37 5e 63 32 48 5d 48 2f 3e 32 2e 54 cR2[7^c2H]H/>2.T
04e0 2e 48 5e 7d 38 2d 5e 3a 35 7f 20 78 4e 7a 34 5a .H^}8-^:5. xNz4Z
04f0 37 2a 2d 5a 29 34 31 71 32 39 38 38 28 5d 41 42 7*-Z)41q2988(]AB
0500 63 20 33 53 27 74 27 46 38 62 60 4a 44 3f 34 6b c 3S't'F8b`JD?4k
0510 26 37 22 29 42 22 4f 2d 4d 73 53 2c 63 51 24 2d &7")B"O-MsS,cQ$-
0520 79 3d 3c 3e 25 79 4c 28 71 3b 27 59 3c 35 5d 36 y=<>%yL(q;'Y<5]6
0530 6c 32 43 2e 73 53 58 54 5b 52 56 72 30 62 4c 28 l2C.sSXT[RVr0bL(
0540 3c 4f 3b 2b 67 57 53 2f 73 21 72 53 6e 7b 39 4e <O;+gWS/s!rSn{9N
0550 23 29 25 27 22 46 24 4b 24 5b 24 46 43 3b 4b 66 #)%'"F$K$[$FC;Kf
0560 49 2b 36 57 27 3a 25 3d 27 21 27 35 56 2c 30 59 I+6W':%='!'5V,0Y
0570 48 21 4b 46 32 31 5e 4a 33 2c 64 71 6b 53 37 28 H!KF21^J3,dqkS7(
0580 46 23 72 36 26 5c 23 6e 31 2a 29 3b 34 72 2d 69 F#r6&\#n1*);4r-i
0590 45 27 39 3c 3d 7b 31 46 2e 7a 2c 55 38 76 23 28 E'9<={1F.z,U8v#(
05a0 2d 77 40 7d 5c 6a 42 4a 3d 3d 4d 4f 34 32 44 37 -w@}\jBJ==MO42D7
05b0 53 5e 47 45 3a 35 64 79 6a 32 5a 49 32 49 2a 4a S^GE:5dyj2ZI2I*J
05c0 2c 4c 64 2e 2b 32 44 7f 6e 3a 2e 7b 3d 39 2c 23 ,Ld.+2D.n:.{=9,#
05d0 49 3c 31 7c 69 2a 36 58 43 27 66 3e 7e 3d 2a 69 I<1|i*6XC'f>~=*i
05e0 53 23 53 30 26 5d 2d 2a 55 32 S#S0&]-*U2
45312 13:05:10.600592 208.96.4.147 -> 208.96.4.140 TCP [Continuation to
#45312] 8009 > 38938 [ACK] Seq=2204919 Ack=85681 Win=34108 Len=1448
TSV=3297605904 T
SER=3624911405
0000 00 11 11 3e c5 8d 00 30 48 56 4e c6 08 00 45 00 ...>...0HVN...E.
0010 05 dc 0f d9 40 00 40 06 7b 63 d0 60 04 93 d0 60 ....@.@.{c.`...`
0020 04 8c 1f 49 98 1a b7 2d 85 cf 05 b6 71 2b 80 10 ...I...-....q+..
0030 21 4f f2 aa 00 00 01 01 08 0a c4 8d 79 10 d8 0f !O..........y...
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org