You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "mohamedin@easy-dialog.info" <mo...@easy-dialog.info> on 2009/12/18 09:30:16 UTC
Errors in session replication and very high server load
Dear All,
I have a strange problem. When I added a new server to my tomcat cluster I have
noticed that the load is getting very high on the server. Tomcat log show a lot
of these lines
18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 8 -7 69
-88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:14
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 62 -53 -50 -43 81
75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:07:19
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNUNG: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75,
-127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={},
]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115
-83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 62 -53
-50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 65 84
-59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -120}:4000,{62, 75, -127, -120},4000, alive=65141440,id={-64 -42 103
97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
domain={}, ]] message. Will verify.
Also the load on the server is getting very high (while no CPU usage). This is
the output of top
top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
Swap: 999992k total, 676k used, 999316k free, 13818440k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:08.92 migration/1
This high load start after starting apache2 (with modjk) so users can enter the
server using the hardware balancer. And the load is increasing always to strange
levels like 99 only stopping tomcat fix it.
Please help
Re: Errors in session replication and very high server load
Posted by Mohamedin <mo...@easy-dialog.info>.
Thanks for the reply.
I have tried to figure out why the load is high but I couldn't. Any hints?
Thanks,
Mohamed Mohamedin
----- Original Message -----
From: "Filip Hanik - Dev Lists" <de...@hanik.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Sunday, December 20, 2009 3:56 PM
Subject: Re: Errors in session replication and very high server load
> Well, the log messages you see, are all based on timeouts.
> If your system has a load average of 12, unless you have a 12-way machine,
> that is very high, and could be the cause of your timeouts.
> You will need to figure out what is causing the high load average.
>
> Filip
>
> On 12/18/2009 01:30 AM, mohamedin@easy-dialog.info wrote:
>> Dear All,
>>
>> I have a strange problem. When I added a new server to my tomcat cluster
>> I have
>> noticed that the load is getting very high on the server. Tomcat log show
>> a lot
>> of these lines
>>
>> 18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster
>> memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97
>> 8 -7 69
>> -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
>> domain={}, ]
>> 18.12.2009 09:07:14
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still
>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15
>> 62 -53 -50 -43 81
>> 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
>> domain={}, ]]
>> 18.12.2009 09:07:19
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> performBasicCheck
>> WARNUNG: Member added, even though we werent
>> notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109
>> 18 -76
>> 79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={},
>> domain={}, ]
>> 18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster
>> memberAdded
>> INFO: Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109
>> 18 -76
>> 79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={},
>> domain={}, ]
>> 18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
>> memberDisappeared
>> INFO: Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75,
>> -127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91
>> 115 -83
>> 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
>> domain={},
>> ]
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> performBasicCheck
>> INFO: Suspect member, confirmed
>> dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
>> 80 64
>> 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
>> domain={}, ]]
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127, -121}:4000,{62, 75, -127, -121},4000,
>> alive=65045054,id={-87 -91 115
>> -83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
>> domain={}, ]] message. Will verify.
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Verification complete. Member still
>> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127,
>> -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83
>> 80 64
>> 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
>> domain={}, ]]
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15
>> 62 -53
>> -50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
>> domain={}, ]] message. Will verify.
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61
>> 65 84
>> -59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
>> domain={}, ]] message. Will verify.
>> 18.12.2009 09:08:10
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
>> memberDisappeared
>> INFO: Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
>> 75, -127, -120}:4000,{62, 75, -127, -120},4000,
>> alive=65141440,id={-64 -42 103
>> 97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={},
>> command={},
>> domain={}, ]] message. Will verify.
>>
>>
>> Also the load on the server is getting very high (while no CPU usage).
>> This is
>> the output of top
>>
>> top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61,
>> 8.35
>> Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
>> Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
>> Swap: 999992k total, 676k used, 999316k free, 13818440k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
>> 2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
>> 3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
>> 4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
>> 5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
>> 6 root RT -5 0 0 0 S 0 0.0 0:08.92 migration/1
>>
>> This high load start after starting apache2 (with modjk) so users can
>> enter the
>> server using the hardware balancer. And the load is increasing always to
>> strange
>> levels like 99 only stopping tomcat fix it.
>>
>>
>> Please help
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> __________ Information from ESET NOD32 Antivirus, version of virus
> signature database 4703 (20091220) __________
>
> The message was checked by ESET NOD32 Antivirus.
>
> http://www.eset.com
>
>
>
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4704 (20091220) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Errors in session replication and very high server load
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
Well, the log messages you see, are all based on timeouts.
If your system has a load average of 12, unless you have a 12-way
machine, that is very high, and could be the cause of your timeouts.
You will need to figure out what is causing the high load average.
Filip
On 12/18/2009 01:30 AM, mohamedin@easy-dialog.info wrote:
> Dear All,
>
> I have a strange problem. When I added a new server to my tomcat cluster I have
> noticed that the load is getting very high on the server. Tomcat log show a lot
> of these lines
>
> 18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 8 -7 69
> -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, domain={}, ]
> 18.12.2009 09:07:14
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 62 -53 -50 -43 81
> 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, domain={}, ]]
> 18.12.2009 09:07:19
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> performBasicCheck
> WARNUNG: Member added, even though we werent
> notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
> 79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
> 18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
> 79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
> 18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberDisappeared
> INFO: Received member
> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75,
> -127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
> 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={},
> ]
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> performBasicCheck
> INFO: Suspect member, confirmed
> dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 80 64
> 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
> 75, -127, -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115
> -83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
> -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 80 64
> 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
> 75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 62 -53
> -50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
> 75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 65 84
> -59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> 18.12.2009 09:08:10
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
> 75, -127, -120}:4000,{62, 75, -127, -120},4000, alive=65141440,id={-64 -42 103
> 97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
> domain={}, ]] message. Will verify.
>
>
> Also the load on the server is getting very high (while no CPU usage). This is
> the output of top
>
> top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
> Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
> Swap: 999992k total, 676k used, 999316k free, 13818440k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
> 2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
> 3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
> 4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
> 5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
> 6 root RT -5 0 0 0 S 0 0.0 0:08.92 migration/1
>
> This high load start after starting apache2 (with modjk) so users can enter the
> server using the hardware balancer. And the load is increasing always to strange
> levels like 99 only stopping tomcat fix it.
>
>
> Please help
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org