You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alain RODRIGUEZ <ar...@gmail.com> on 2013/02/25 11:28:45 UTC
Ghost nodes
Hi,
I am having issues after decommissioning 3 nodes one by one of my 1.1.6 C*
cluster (RF=3):
On the "c.164" node, which was added a week after removing the 3 nodes,
with gossipinfo I have:
/a.135
RPC_ADDRESS:0.0.0.0
STATUS:NORMAL,127605887595351923798765477786913079296
RELEASE_VERSION:1.1.6
RACK:1b
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
DC:eu-west
LOAD:3.40954135223E11
/b.173
RPC_ADDRESS:0.0.0.0
STATUS:NORMAL,0
RELEASE_VERSION:1.1.6
RACK:1b
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
DC:eu-west
LOAD:3.32757832183E11
/c.164
RPC_ADDRESS:0.0.0.0
STATUS:NORMAL,85070591730234615865843651857942052864
RELEASE_VERSION:1.1.6
RACK:1b
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
DC:eu-west
LOAD:2.93726484252E11
/d.6
RPC_ADDRESS:0.0.0.0
STATUS:NORMAL,42535295865117307932921825928971026432
RELEASE_VERSION:1.1.6
RACK:1b
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
DC:eu-west
LOAD:2.85020693654E11
On the 3 other nodes I see this:
/a.135
RPC_ADDRESS:0.0.0.0
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:NORMAL,127605887595351923798765477786913079296
DC:eu-west
LOAD:3.40974023487E11
/10.64.167.32
RPC_ADDRESS:0.0.0.0
SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:LEFT,28356863910078203714492389662765613056,1359823927010
DC:eu-west
LOAD:1.47947624544E11
/10.250.202.154
RPC_ADDRESS:0.0.0.0
SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:LEFT,85070591730234615865843651857942052863,1359808901882
DC:eu-west
LOAD:1.45049060742E11
/b.173
RPC_ADDRESS:0.0.0.0
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:NORMAL,0
DC:eu-west
LOAD:3.32760540235E11
/c.164
RPC_ADDRESS:0.0.0.0
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
RELEASE_VERSION:1.1.6
RACK:1b
LOAD:2.93751485625E11
DC:eu-west
STATUS:NORMAL,85070591730234615865843651857942052864
/10.64.103.228
RPC_ADDRESS:0.0.0.0
SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:LEFT,141784319550391032739561396922763706367,1359893766266
DC:eu-west
LOAD:2.46247802646E11
/d.6
RPC_ADDRESS:0.0.0.0
SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
RELEASE_VERSION:1.1.6
RACK:1b
STATUS:NORMAL,42535295865117307932921825928971026432
DC:eu-west
LOAD:2.85042986093E11
Since I removed this 3 nodes (marked as "left") at least 3 weeks ago,
shouldn't gossip have them totally removed for a while ?
The "c.164", node that doesn't show nodes that left the ring in
gosssipinfo, is logging every minute the following:
...
INFO [GossipStage:1] 2013-02-25 10:18:56,269 Gossiper.java (line 830)
InetAddress /10.64.167.32 is now dead.
INFO [GossipStage:1] 2013-02-25 10:18:56,283 Gossiper.java (line 830)
InetAddress /10.250.202.154 is now dead.
INFO [GossipStage:1] 2013-02-25 10:18:56,297 Gossiper.java (line 830)
InetAddress /10.64.103.228 is now dead.
INFO [GossipStage:1] 2013-02-25 10:19:57,700 Gossiper.java (line 830)
InetAddress /10.64.167.32 is now dead.
INFO [GossipStage:1] 2013-02-25 10:19:57,721 Gossiper.java (line 830)
InetAddress /10.250.202.154 is now dead.
INFO [GossipStage:1] 2013-02-25 10:19:57,742 Gossiper.java (line 830)
InetAddress /10.64.103.228 is now dead.
INFO [GossipStage:1] 2013-02-25 10:20:58,722 Gossiper.java (line 830)
InetAddress /10.64.167.32 is now dead.
INFO [GossipStage:1] 2013-02-25 10:20:58,739 Gossiper.java (line 830)
InetAddress /10.250.202.154 is now dead.
INFO [GossipStage:1] 2013-02-25 10:20:58,754 Gossiper.java (line 830)
InetAddress /10.64.103.228 is now dead.
...
All this look a bit weird to me, is this normal ?
Alain
Re: Ghost nodes
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
"try assasinate from the jmx?
http://nartax.com/2012/09/assassinate-cassandra-node/"
I finally used this solution... It always solves the problems of ghost
nodes :D. Last time I had unreachable nodes while describing cluster in CLI
(as described in the link) and I used the jmx unsafeAssassinateEnpoint.
This time it was a bit different since the schema was good an sync between
all the nodes, but this function solved this new issue.
Thanks for the answer even if I was also hopping to understand what
happened and not just "Assassinate" the problem, at least my prod is now ok.
Alain
2013/3/5 Jason Wee <pe...@gmail.com>
> try assasinate from the jmx?
> http://nartax.com/2012/09/assassinate-cassandra-node/
>
> or try cassandra -Dcassandra.load_ring_state=false
> http://www.datastax.com/docs/1.0/references/cassandra#options
>
>
> On Tue, Mar 5, 2013 at 6:54 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> Any clue on this ?
>>
>>
>> 2013/2/25 Alain RODRIGUEZ <ar...@gmail.com>
>>
>>> Hi,
>>>
>>> I am having issues after decommissioning 3 nodes one by one of my 1.1.6
>>> C* cluster (RF=3):
>>>
>>> On the "c.164" node, which was added a week after removing the 3 nodes,
>>> with gossipinfo I have:
>>>
>>> /a.135
>>> RPC_ADDRESS:0.0.0.0
>>> STATUS:NORMAL,127605887595351923798765477786913079296
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> DC:eu-west
>>> LOAD:3.40954135223E11
>>> /b.173
>>> RPC_ADDRESS:0.0.0.0
>>> STATUS:NORMAL,0
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> DC:eu-west
>>> LOAD:3.32757832183E11
>>> /c.164
>>> RPC_ADDRESS:0.0.0.0
>>> STATUS:NORMAL,85070591730234615865843651857942052864
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> DC:eu-west
>>> LOAD:2.93726484252E11
>>> /d.6
>>> RPC_ADDRESS:0.0.0.0
>>> STATUS:NORMAL,42535295865117307932921825928971026432
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> DC:eu-west
>>> LOAD:2.85020693654E11
>>>
>>>
>>> On the 3 other nodes I see this:
>>>
>>>
>>> /a.135
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:NORMAL,127605887595351923798765477786913079296
>>> DC:eu-west
>>> LOAD:3.40974023487E11
>>> /10.64.167.32
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:LEFT,28356863910078203714492389662765613056,1359823927010
>>> DC:eu-west
>>> LOAD:1.47947624544E11
>>> /10.250.202.154
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:LEFT,85070591730234615865843651857942052863,1359808901882
>>> DC:eu-west
>>> LOAD:1.45049060742E11
>>> /b.173
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:NORMAL,0
>>> DC:eu-west
>>> LOAD:3.32760540235E11
>>> /c.164
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> LOAD:2.93751485625E11
>>> DC:eu-west
>>> STATUS:NORMAL,85070591730234615865843651857942052864
>>> /10.64.103.228
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:LEFT,141784319550391032739561396922763706367,1359893766266
>>> DC:eu-west
>>> LOAD:2.46247802646E11
>>> /d.6
>>> RPC_ADDRESS:0.0.0.0
>>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>>> RELEASE_VERSION:1.1.6
>>> RACK:1b
>>> STATUS:NORMAL,42535295865117307932921825928971026432
>>> DC:eu-west
>>> LOAD:2.85042986093E11
>>>
>>>
>>> Since I removed this 3 nodes (marked as "left") at least 3 weeks ago,
>>> shouldn't gossip have them totally removed for a while ?
>>>
>>> The "c.164", node that doesn't show nodes that left the ring in
>>> gosssipinfo, is logging every minute the following:
>>>
>>> ...
>>> INFO [GossipStage:1] 2013-02-25 10:18:56,269 Gossiper.java (line 830)
>>> InetAddress /10.64.167.32 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:18:56,283 Gossiper.java (line 830)
>>> InetAddress /10.250.202.154 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:18:56,297 Gossiper.java (line 830)
>>> InetAddress /10.64.103.228 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:19:57,700 Gossiper.java (line 830)
>>> InetAddress /10.64.167.32 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:19:57,721 Gossiper.java (line 830)
>>> InetAddress /10.250.202.154 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:19:57,742 Gossiper.java (line 830)
>>> InetAddress /10.64.103.228 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:20:58,722 Gossiper.java (line 830)
>>> InetAddress /10.64.167.32 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:20:58,739 Gossiper.java (line 830)
>>> InetAddress /10.250.202.154 is now dead.
>>> INFO [GossipStage:1] 2013-02-25 10:20:58,754 Gossiper.java (line 830)
>>> InetAddress /10.64.103.228 is now dead.
>>> ...
>>>
>>> All this look a bit weird to me, is this normal ?
>>>
>>> Alain
>>>
>>>
>>>
>>>
>>
>
Re: Ghost nodes
Posted by Jason Wee <pe...@gmail.com>.
try assasinate from the jmx?
http://nartax.com/2012/09/assassinate-cassandra-node/
or try cassandra -Dcassandra.load_ring_state=false
http://www.datastax.com/docs/1.0/references/cassandra#options
On Tue, Mar 5, 2013 at 6:54 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Any clue on this ?
>
>
> 2013/2/25 Alain RODRIGUEZ <ar...@gmail.com>
>
>> Hi,
>>
>> I am having issues after decommissioning 3 nodes one by one of my 1.1.6
>> C* cluster (RF=3):
>>
>> On the "c.164" node, which was added a week after removing the 3 nodes,
>> with gossipinfo I have:
>>
>> /a.135
>> RPC_ADDRESS:0.0.0.0
>> STATUS:NORMAL,127605887595351923798765477786913079296
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> DC:eu-west
>> LOAD:3.40954135223E11
>> /b.173
>> RPC_ADDRESS:0.0.0.0
>> STATUS:NORMAL,0
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> DC:eu-west
>> LOAD:3.32757832183E11
>> /c.164
>> RPC_ADDRESS:0.0.0.0
>> STATUS:NORMAL,85070591730234615865843651857942052864
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> DC:eu-west
>> LOAD:2.93726484252E11
>> /d.6
>> RPC_ADDRESS:0.0.0.0
>> STATUS:NORMAL,42535295865117307932921825928971026432
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> DC:eu-west
>> LOAD:2.85020693654E11
>>
>>
>> On the 3 other nodes I see this:
>>
>>
>> /a.135
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:NORMAL,127605887595351923798765477786913079296
>> DC:eu-west
>> LOAD:3.40974023487E11
>> /10.64.167.32
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:LEFT,28356863910078203714492389662765613056,1359823927010
>> DC:eu-west
>> LOAD:1.47947624544E11
>> /10.250.202.154
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:LEFT,85070591730234615865843651857942052863,1359808901882
>> DC:eu-west
>> LOAD:1.45049060742E11
>> /b.173
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:NORMAL,0
>> DC:eu-west
>> LOAD:3.32760540235E11
>> /c.164
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> LOAD:2.93751485625E11
>> DC:eu-west
>> STATUS:NORMAL,85070591730234615865843651857942052864
>> /10.64.103.228
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:LEFT,141784319550391032739561396922763706367,1359893766266
>> DC:eu-west
>> LOAD:2.46247802646E11
>> /d.6
>> RPC_ADDRESS:0.0.0.0
>> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
>> RELEASE_VERSION:1.1.6
>> RACK:1b
>> STATUS:NORMAL,42535295865117307932921825928971026432
>> DC:eu-west
>> LOAD:2.85042986093E11
>>
>>
>> Since I removed this 3 nodes (marked as "left") at least 3 weeks ago,
>> shouldn't gossip have them totally removed for a while ?
>>
>> The "c.164", node that doesn't show nodes that left the ring in
>> gosssipinfo, is logging every minute the following:
>>
>> ...
>> INFO [GossipStage:1] 2013-02-25 10:18:56,269 Gossiper.java (line 830)
>> InetAddress /10.64.167.32 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:18:56,283 Gossiper.java (line 830)
>> InetAddress /10.250.202.154 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:18:56,297 Gossiper.java (line 830)
>> InetAddress /10.64.103.228 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:19:57,700 Gossiper.java (line 830)
>> InetAddress /10.64.167.32 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:19:57,721 Gossiper.java (line 830)
>> InetAddress /10.250.202.154 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:19:57,742 Gossiper.java (line 830)
>> InetAddress /10.64.103.228 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:20:58,722 Gossiper.java (line 830)
>> InetAddress /10.64.167.32 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:20:58,739 Gossiper.java (line 830)
>> InetAddress /10.250.202.154 is now dead.
>> INFO [GossipStage:1] 2013-02-25 10:20:58,754 Gossiper.java (line 830)
>> InetAddress /10.64.103.228 is now dead.
>> ...
>>
>> All this look a bit weird to me, is this normal ?
>>
>> Alain
>>
>>
>>
>>
>
Re: Ghost nodes
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Any clue on this ?
2013/2/25 Alain RODRIGUEZ <ar...@gmail.com>
> Hi,
>
> I am having issues after decommissioning 3 nodes one by one of my 1.1.6 C*
> cluster (RF=3):
>
> On the "c.164" node, which was added a week after removing the 3 nodes,
> with gossipinfo I have:
>
> /a.135
> RPC_ADDRESS:0.0.0.0
> STATUS:NORMAL,127605887595351923798765477786913079296
> RELEASE_VERSION:1.1.6
> RACK:1b
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> DC:eu-west
> LOAD:3.40954135223E11
> /b.173
> RPC_ADDRESS:0.0.0.0
> STATUS:NORMAL,0
> RELEASE_VERSION:1.1.6
> RACK:1b
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> DC:eu-west
> LOAD:3.32757832183E11
> /c.164
> RPC_ADDRESS:0.0.0.0
> STATUS:NORMAL,85070591730234615865843651857942052864
> RELEASE_VERSION:1.1.6
> RACK:1b
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> DC:eu-west
> LOAD:2.93726484252E11
> /d.6
> RPC_ADDRESS:0.0.0.0
> STATUS:NORMAL,42535295865117307932921825928971026432
> RELEASE_VERSION:1.1.6
> RACK:1b
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> DC:eu-west
> LOAD:2.85020693654E11
>
>
> On the 3 other nodes I see this:
>
>
> /a.135
> RPC_ADDRESS:0.0.0.0
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:NORMAL,127605887595351923798765477786913079296
> DC:eu-west
> LOAD:3.40974023487E11
> /10.64.167.32
> RPC_ADDRESS:0.0.0.0
> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:LEFT,28356863910078203714492389662765613056,1359823927010
> DC:eu-west
> LOAD:1.47947624544E11
> /10.250.202.154
> RPC_ADDRESS:0.0.0.0
> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:LEFT,85070591730234615865843651857942052863,1359808901882
> DC:eu-west
> LOAD:1.45049060742E11
> /b.173
> RPC_ADDRESS:0.0.0.0
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:NORMAL,0
> DC:eu-west
> LOAD:3.32760540235E11
> /c.164
> RPC_ADDRESS:0.0.0.0
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> RELEASE_VERSION:1.1.6
> RACK:1b
> LOAD:2.93751485625E11
> DC:eu-west
> STATUS:NORMAL,85070591730234615865843651857942052864
> /10.64.103.228
> RPC_ADDRESS:0.0.0.0
> SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:LEFT,141784319550391032739561396922763706367,1359893766266
> DC:eu-west
> LOAD:2.46247802646E11
> /d.6
> RPC_ADDRESS:0.0.0.0
> SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c
> RELEASE_VERSION:1.1.6
> RACK:1b
> STATUS:NORMAL,42535295865117307932921825928971026432
> DC:eu-west
> LOAD:2.85042986093E11
>
>
> Since I removed this 3 nodes (marked as "left") at least 3 weeks ago,
> shouldn't gossip have them totally removed for a while ?
>
> The "c.164", node that doesn't show nodes that left the ring in
> gosssipinfo, is logging every minute the following:
>
> ...
> INFO [GossipStage:1] 2013-02-25 10:18:56,269 Gossiper.java (line 830)
> InetAddress /10.64.167.32 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:18:56,283 Gossiper.java (line 830)
> InetAddress /10.250.202.154 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:18:56,297 Gossiper.java (line 830)
> InetAddress /10.64.103.228 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:19:57,700 Gossiper.java (line 830)
> InetAddress /10.64.167.32 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:19:57,721 Gossiper.java (line 830)
> InetAddress /10.250.202.154 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:19:57,742 Gossiper.java (line 830)
> InetAddress /10.64.103.228 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:20:58,722 Gossiper.java (line 830)
> InetAddress /10.64.167.32 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:20:58,739 Gossiper.java (line 830)
> InetAddress /10.250.202.154 is now dead.
> INFO [GossipStage:1] 2013-02-25 10:20:58,754 Gossiper.java (line 830)
> InetAddress /10.64.103.228 is now dead.
> ...
>
> All this look a bit weird to me, is this normal ?
>
> Alain
>
>
>
>