Rebooted cassandra node timing out all requests but recovers after a while

[Anand Somani <me...@gmail.com>]

Hi,

We have a 3 node cluster (on VM). Eg. host1, host2, host3. One of the VM
rebooted (host1) and when host1 came up it would see the others as down and
the others (host2 and host3) see it as down. So we restarted host2 and now
the ring seems fine(everybody sees everybody as up).

But now the clients timeout talking to host1. Have not figured out what is
causing it. There is nothing in the logs that indicates a problem. Looking
for indicators/help on what debug/tracing to turn on to find out what could
be causing it.

Now this happens only when a VM reboots (not otherwise), also it seems to
have recovered itself after some hours!!( or restarts) not sure which one.

This is 1.2.15, we are using ssl and cassandra authorizers.

Thanks
Anand

Re: Rebooted cassandra node timing out all requests but recovers after a while

[Anand Somani <me...@gmail.com>]

I will keep an eye for that if it happens again. Times at this point are
synchronized

On Wed, Jan 7, 2015 at 10:37 PM, Duncan Sands <du...@gmail.com>
wrote:

> Hi Anand,
>
>
> On 08/01/15 02:02, Anand Somani wrote:
>
>> Hi,
>>
>> We have a 3 node cluster (on VM). Eg. host1, host2, host3. One of the VM
>> rebooted (host1) and when host1 came up it would see the others as down
>> and the
>> others (host2 and host3) see it as down. So we restarted host2 and now
>> the ring
>> seems fine(everybody sees everybody as up).
>>
>> But now the clients timeout talking to host1. Have not figured out what is
>> causing it. There is nothing in the logs that indicates a problem.
>> Looking for
>> indicators/help on what debug/tracing to turn on to find out what could be
>> causing it.
>>
>> Now this happens only when a VM reboots (not otherwise), also it seems to
>> have
>> recovered itself after some hours!!( or restarts) not sure which one.
>>
>> This is 1.2.15, we are using ssl and cassandra authorizers.
>>
>
> perhaps time is not synchronized between the nodes to begin with, and
> eventually becomes synchronized.
>
> Ciao, Duncan.
>

Re: Rebooted cassandra node timing out all requests but recovers after a while

[Duncan Sands <du...@gmail.com>]

Hi Anand,

On 08/01/15 02:02, Anand Somani wrote:
> Hi,
>
> We have a 3 node cluster (on VM). Eg. host1, host2, host3. One of the VM
> rebooted (host1) and when host1 came up it would see the others as down and the
> others (host2 and host3) see it as down. So we restarted host2 and now the ring
> seems fine(everybody sees everybody as up).
>
> But now the clients timeout talking to host1. Have not figured out what is
> causing it. There is nothing in the logs that indicates a problem. Looking for
> indicators/help on what debug/tracing to turn on to find out what could be
> causing it.
>
> Now this happens only when a VM reboots (not otherwise), also it seems to have
> recovered itself after some hours!!( or restarts) not sure which one.
>
> This is 1.2.15, we are using ssl and cassandra authorizers.

perhaps time is not synchronized between the nodes to begin with, and eventually 
becomes synchronized.

Ciao, Duncan.