You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by "Walsh, Peter" <Pe...@disney.com> on 2011/11/08 00:46:10 UTC

Trouble getting full clustering working...

Hello,
I'm trying to get full clustering working between 2 Traffic Server instances I have.  I followed the steps in this document, http://trafficserver.staging.apache.org/docs/trunk/admin/cluster-howto/#FullClustering, however once the 2 instances start they each output some info indicating the other instance as down. So it seems they are initially aware of each other but the heartbeat is failing for some reason.  I'm working on getting Wireshark on these machines to troubleshoot but wanted to send out an email to see if anyone had advice. Thanks in advance.

Example output from traffic.out

[Nov  7 14:40:18.827] Server {47628384669248} NOTE: cache clustering enabled
[Nov  7 14:40:18.984] Server {47628384669248} NOTE: cache clustering enabled
[Nov  7 14:40:19.075] Server {47628384669248} NOTE: logging initialized[7], logging_mode = 3
[Nov  7 14:40:19.079] Server {47628384669248} NOTE: traffic server running
[Nov  7 14:40:19.081] Server {1098275136} NOTE: cache enabled
[Nov  7 14:40:44.233] Manager {1113639232} NOTE: adding node 10.195.10.57 to the cluster
[Nov  7 14:40:54.932] Server {1079224640} NOTE: machine up 10.195.10.57, protocol version=3.0
[Nov  7 14:41:05.060] Server {1079224640} NOTE: machine down 10.195.10.57
[Nov  7 14:41:05.130] Server {1079224640} NOTE: machine up 10.195.10.57, protocol version=3.0
[Nov  7 14:41:15.146] Server {1079224640} NOTE: machine down 10.195.10.57
[Nov  7 14:41:15.236] Server {1079224640} NOTE: machine up 10.195.10.57, protocol version=3.0
[Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking manager on node 10.195.10.57 as down
[Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking server on node 10.195.10.57 as down
[Nov  7 14:41:25.270] Server {1079224640} NOTE: machine down 10.195.10.57
[Nov  7 14:55:45.357] Manager {1113639232} NOTE: marking manager on node 10.195.10.57 as up
[Nov  7 14:55:45.363] Manager {1113639232} NOTE: marking server on node 10.195.10.57 as up
[Nov  7 14:55:46.381] Manager {1113639232} NOTE: [drainIncomingChannel] file req: records.config v: 71 bytes: 28164
[Nov  7 14:55:55.565] Server {1079224640} NOTE: machine up 10.195.10.57, protocol version=3.0
[Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking manager on node 10.195.10.57 as down
[Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking server on node 10.195.10.57 as down


- Pete

RE: Trouble getting full clustering working...

Posted by "Walsh, Peter" <Pe...@disney.com>.
After upgrading to 3.0.1 I can see in the logs and in cluster.config that instances are part of a cluster now.  However, despite being in full clustering mode I can't seem to get peer fetching to work, which I'll have to investigate some more.

Pete Walsh
Software Engineer
206-664-4150


-----Original Message-----
From: users-return-1177-Peter.Walsh=disney.com@trafficserver.apache.org [mailto:users-return-1177-Peter.Walsh=disney.com@trafficserver.apache.org] On Behalf Of Walsh, Peter
Sent: Tuesday, November 08, 2011 8:49 AM
To: users@trafficserver.apache.org
Subject: RE: Trouble getting full clustering working...

We are still on 2.1.8 at the moment but are planning on upgrading to 3.0.1 in the near future.  If you think it might be broken in 2.1.8 I'll hold off verifying the clustering capability until we've upgraded to the latest stable.

Pete Walsh
Software Engineer
206-664-4150


-----Original Message-----
From: ming.zym@gmail.com [mailto:ming.zym@gmail.com] 
Sent: Monday, November 07, 2011 8:02 PM
To: users@trafficserver.apache.org
Subject: Re: Trouble getting full clustering working...

which version in your testing? AFAIK, the v3.0.1 & v3.0.x branch is
working with cluster.

the trunk is in heave developing, and the cluster & other function may
broken without any notice.


在 2011-11-07一的 15:46 -0800,Walsh, Peter写道:
> Hello,
> I'm trying to get full clustering working between 2 Traffic Server
> instances I have.  I followed the steps in this
> document, http://trafficserver.staging.apache.org/docs/trunk/admin/cluster-howto/#FullClustering, however once the 2 instances start they each output some info indicating the other instance as down. So it seems they are initially aware of each other but the heartbeat is failing for some reason.  I'm working on getting Wireshark on these machines to troubleshoot but wanted to send out an email to see if anyone had advice. Thanks in advance.
> 
> 
> Example output from traffic.out
> 
> 
> [Nov  7 14:40:18.827] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:18.984] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:19.075] Server {47628384669248} NOTE: logging
> initialized[7], logging_mode = 3
> [Nov  7 14:40:19.079] Server {47628384669248} NOTE: traffic server
> running
> [Nov  7 14:40:19.081] Server {1098275136} NOTE: cache enabled
> [Nov  7 14:40:44.233] Manager {1113639232} NOTE: adding node
> 10.195.10.57 to the cluster
> [Nov  7 14:40:54.932] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:05.060] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:05.130] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:15.146] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:15.236] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> [Nov  7 14:41:25.270] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:55:45.357] Manager {1113639232} NOTE: marking manager on
> node 10.195.10.57 as up
> [Nov  7 14:55:45.363] Manager {1113639232} NOTE: marking server on
> node 10.195.10.57 as up
> [Nov  7 14:55:46.381] Manager {1113639232} NOTE:
> [drainIncomingChannel] file req: records.config v: 71 bytes: 28164
> [Nov  7 14:55:55.565] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> 
> 
> 
> 
> - Pete

-- 
zym, Zhao Yongming.
aka: yonghao @ taobao.com


RE: Trouble getting full clustering working...

Posted by "Walsh, Peter" <Pe...@disney.com>.
We are still on 2.1.8 at the moment but are planning on upgrading to 3.0.1 in the near future.  If you think it might be broken in 2.1.8 I'll hold off verifying the clustering capability until we've upgraded to the latest stable.

Pete Walsh
Software Engineer
206-664-4150


-----Original Message-----
From: ming.zym@gmail.com [mailto:ming.zym@gmail.com] 
Sent: Monday, November 07, 2011 8:02 PM
To: users@trafficserver.apache.org
Subject: Re: Trouble getting full clustering working...

which version in your testing? AFAIK, the v3.0.1 & v3.0.x branch is
working with cluster.

the trunk is in heave developing, and the cluster & other function may
broken without any notice.


在 2011-11-07一的 15:46 -0800,Walsh, Peter写道:
> Hello,
> I'm trying to get full clustering working between 2 Traffic Server
> instances I have.  I followed the steps in this
> document, http://trafficserver.staging.apache.org/docs/trunk/admin/cluster-howto/#FullClustering, however once the 2 instances start they each output some info indicating the other instance as down. So it seems they are initially aware of each other but the heartbeat is failing for some reason.  I'm working on getting Wireshark on these machines to troubleshoot but wanted to send out an email to see if anyone had advice. Thanks in advance.
> 
> 
> Example output from traffic.out
> 
> 
> [Nov  7 14:40:18.827] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:18.984] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:19.075] Server {47628384669248} NOTE: logging
> initialized[7], logging_mode = 3
> [Nov  7 14:40:19.079] Server {47628384669248} NOTE: traffic server
> running
> [Nov  7 14:40:19.081] Server {1098275136} NOTE: cache enabled
> [Nov  7 14:40:44.233] Manager {1113639232} NOTE: adding node
> 10.195.10.57 to the cluster
> [Nov  7 14:40:54.932] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:05.060] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:05.130] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:15.146] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:15.236] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> [Nov  7 14:41:25.270] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:55:45.357] Manager {1113639232} NOTE: marking manager on
> node 10.195.10.57 as up
> [Nov  7 14:55:45.363] Manager {1113639232} NOTE: marking server on
> node 10.195.10.57 as up
> [Nov  7 14:55:46.381] Manager {1113639232} NOTE:
> [drainIncomingChannel] file req: records.config v: 71 bytes: 28164
> [Nov  7 14:55:55.565] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> 
> 
> 
> 
> - Pete

-- 
zym, Zhao Yongming.
aka: yonghao @ taobao.com


Re: Trouble getting full clustering working...

Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
which version in your testing? AFAIK, the v3.0.1 & v3.0.x branch is
working with cluster.

the trunk is in heave developing, and the cluster & other function may
broken without any notice.


在 2011-11-07一的 15:46 -0800,Walsh, Peter写道:
> Hello,
> I'm trying to get full clustering working between 2 Traffic Server
> instances I have.  I followed the steps in this
> document, http://trafficserver.staging.apache.org/docs/trunk/admin/cluster-howto/#FullClustering, however once the 2 instances start they each output some info indicating the other instance as down. So it seems they are initially aware of each other but the heartbeat is failing for some reason.  I'm working on getting Wireshark on these machines to troubleshoot but wanted to send out an email to see if anyone had advice. Thanks in advance.
> 
> 
> Example output from traffic.out
> 
> 
> [Nov  7 14:40:18.827] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:18.984] Server {47628384669248} NOTE: cache clustering
> enabled
> [Nov  7 14:40:19.075] Server {47628384669248} NOTE: logging
> initialized[7], logging_mode = 3
> [Nov  7 14:40:19.079] Server {47628384669248} NOTE: traffic server
> running
> [Nov  7 14:40:19.081] Server {1098275136} NOTE: cache enabled
> [Nov  7 14:40:44.233] Manager {1113639232} NOTE: adding node
> 10.195.10.57 to the cluster
> [Nov  7 14:40:54.932] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:05.060] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:05.130] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:15.146] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:41:15.236] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:41:19.862] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> [Nov  7 14:41:25.270] Server {1079224640} NOTE: machine down
> 10.195.10.57
> [Nov  7 14:55:45.357] Manager {1113639232} NOTE: marking manager on
> node 10.195.10.57 as up
> [Nov  7 14:55:45.363] Manager {1113639232} NOTE: marking server on
> node 10.195.10.57 as up
> [Nov  7 14:55:46.381] Manager {1113639232} NOTE:
> [drainIncomingChannel] file req: records.config v: 71 bytes: 28164
> [Nov  7 14:55:55.565] Server {1079224640} NOTE: machine up
> 10.195.10.57, protocol version=3.0
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking manager
> on node 10.195.10.57 as down
> [Nov  7 14:56:20.341] Manager {47832443424192} NOTE: marking server on
> node 10.195.10.57 as down
> 
> 
> 
> 
> - Pete

-- 
zym, Zhao Yongming.
aka: yonghao @ taobao.com