You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Cassa L <lc...@gmail.com> on 2011/05/18 06:28:55 UTC

Gossiper question

Hi,
  I have 9 node cluster with RF-3 and using Cassandra0.70/Hector26. Recently
we are seeing lot of "UnavailableException" at the client side. Whenever
this happens, I found following pattern in Cassandra node's log file at that
given time,

* INFO [ScheduledTasks:1] 2011-05-13 02:59:55,365 Gossiper.java (line 195)
InetAddress /**.**.***.54 is now dead.*
* INFO [ScheduledTasks:1] 2011-05-13 02:59:57,369 Gossiper.java (line 195)
InetAddress /****.**.***.**59 is now dead.*
 INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 HintedHandOffManager.java
(line 192) Started hinted handoff for endpoint /***.**.***.*54
* INFO [GossipStage:1] 2011-05-13 03:00:04,706 Gossiper.java (line 569)
InetAddress /****.**.*****.54 is now UP*
 INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 HintedHandOffManager.java
(line 248) Finished hinted handoff of 0 rows to endpoint /***.**.****.54
 INFO [HintedHandoff:1] 2011-05-13 03:00:20,601 HintedHandOffManager.java
(line 192) Started hinted handoff for endpoint /***.**.****.59
* INFO [GossipStage:1] 2011-05-13 03:00:20,601 Gossiper.java (line 569)
InetAddress /****.**.*****.59 is now UP*

The exception occurred at "2011-05-13 03:00:00,664". I am wondering what why
this dead/up pattern is occurring at Gossip.

Thanks in advance,
Cassa L.

Re: Gossiper question

Posted by Ali Ahsan <al...@panasiangroup.com>.
On 05/18/2011 09:28 AM, Cassa L wrote:
> Hi,
>   I have 9 node cluster with RF-3 and using Cassandra0.70/Hector26. 
> Recently we are seeing lot of "UnavailableException" at the client 
> side. Whenever this happens, I found following pattern in Cassandra 
> node's log file at that given time,
>
> * INFO [ScheduledTasks:1] 2011-05-13 02:59:55,365 Gossiper.java (line 
> 195) InetAddress /**.**.***.54 is now dead.*
> * INFO [ScheduledTasks:1] 2011-05-13 02:59:57,369 Gossiper.java (line 
> 195) InetAddress /****.**.***.**59 is now dead.*
>  INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 
> HintedHandOffManager.java (line 192) Started hinted handoff for 
> endpoint /***.**.***.*54
> * INFO [GossipStage:1] 2011-05-13 03:00:04,706 Gossiper.java (line 
> 569) InetAddress /****.**.*****.54 is now UP*
>  INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 
> HintedHandOffManager.java (line 248) Finished hinted handoff of 0 rows 
> to endpoint /***.**.****.54
>  INFO [HintedHandoff:1] 2011-05-13 03:00:20,601 
> HintedHandOffManager.java (line 192) Started hinted handoff for 
> endpoint /***.**.****.59
> * INFO [GossipStage:1] 2011-05-13 03:00:20,601 Gossiper.java (line 
> 569) InetAddress /****.**.*****.59 is now UP*
>
> The exception occurred at "2011-05-13 03:00:00,664". I am wondering 
> what why this dead/up pattern is occurring at Gossip.
>
> Thanks in advance,
> Cassa L.

I am not sure do check system Load when you see this happening.If load 
is to High then it means CPU was not able to give time to Network I/O.

-- 
S.Ali Ahsan




Re: Gossiper question

Posted by Peter Schuller <pe...@infidyne.com>.
>   I have 9 node cluster with RF-3 and using Cassandra0.70/Hector26. Recently
> we are seeing lot of "UnavailableException" at the client side. Whenever
> this happens, I found following pattern in Cassandra node's log file at that
> given time,

UnavailableException is the expected error if an insufficient number
of nodes are up in the replicate set for the row you're
reading/writing. "Up" in this case means according to gossip (what you
see with nodetool ring).

The question is really why nodes are being marked as down and whether
it's legitimate or not. There is
https://issues.apache.org/jira/browse/CASSANDRA-2554 which may be
relevant, but simple overload is also a possible reason.

-- 
/ Peter Schuller