unreachable nodes mystery in describecluster output

[Aleksandr Ivanov <al...@gmail.com>]

Hello,

I'm running v3.0.8 in multi-data center deployment (6 DCs, 6 nodes per DC,
maximum latency between some nodes ~200ms).
After clean cluster start I run into issue when "nodetool descibecluster"
shows that some random nodes from deployment are UNREACHABLE however in
"nodetool status" or "nodetool gossipinfo" all nodes are UP and in NORMAL
state.
Nothing suspicious in log file also...

As result ALTER USER failing with following message:
WriteTimeout: code=1100 [Coordinator node timed out waiting for replica
nodes' responses] message="Operation timed out - received only 8
responses." info={'received_responses': 8, 'required_responses': 10,
'consistency': 'QUORUM'}

Any hints or guidance how to troubleshoot this issue?

I have seen advices to restart affected nodes and also tried it but it
helps only till first query to cassandra.
Related question: how production ready is 3.0.x version? Should I switch to
more stable 2.x even if support ends in Nov 2016?

ps. date/time is same on all nodes; synced with NTP

Re: unreachable nodes mystery in describecluster output

[Aleksandr Ivanov <al...@gmail.com>]

No VPN involved and no limitations, which can affects internode
communication, on network level. I'm curious why "nodetool status" shows
that all is OK and no suspicious messages in log file is such problem
exists.
I'm looking for hints how to troubleshoot such problem or maybe anyone have
seen such situation and can give some recommendations how to solve it.


On Wed, Aug 3, 2016 at 4:39 PM Romain Hardouin <ro...@yahoo.fr> wrote:

> That's a good news if describecluster shows the same version on each node.
> Try with a high timeout like 120 seconds to see if it works.
> Is there a VPN between DCs? Is there room for improvement at the network
> level? TCP tuning, etc. I'm not saying you won't have unreachable nodes but
> it's worth it if you can.
>
> Romain
>
> Le Mercredi 3 août 2016 15h02, Aleksandr Ivanov <al...@gmail.com> a
> écrit :
>
>
> The latency is high...
>
> It is but is it really causing the problem? Latency is high but constant
> and not higher than ~200ms.
>
> Regarding the ALTER, did you try to increase the timeout with "cqlsh
> --request-timeout=REQUEST_TIMEOUT"? Because the default is 10 seconds.
>
> I use 25sec timeout (--request-timeout 25)
>
> Apart the unreachable nodes, do you know if all nodes have the same schema
> version?
>
> "nodetool gossipinfo" shows same scheme version on all nodes.
>
> Best,
>
> Romain
>
>
>
>

Re: unreachable nodes mystery in describecluster output

[Romain Hardouin <ro...@yahoo.fr>]

That's a good news if describecluster shows the same version on each node. Try with a high timeout like 120 seconds to see if it works. Is there a VPN between DCs? Is there room for improvement at the network level? TCP tuning, etc. I'm not saying you won't have unreachable nodes but it's worth it if you can.
Romain
    Le Mercredi 3 août 2016 15h02, Aleksandr Ivanov <al...@gmail.com> a écrit :
 

 
The latency is high...
It is but is it really causing the problem? Latency is high but constant and not higher than ~200ms.

Regarding the ALTER, did you try to increase the timeout with "cqlsh --request-timeout=REQUEST_TIMEOUT"? Because the default is 10 seconds.

I use 25sec timeout (--request-timeout 25)

 Apart the unreachable nodes, do you know if all nodes have the same schema version? 
"nodetool gossipinfo" shows same scheme version on all nodes.

Best,
Romain


  

Re: unreachable nodes mystery in describecluster output

[Aleksandr Ivanov <al...@gmail.com>]

>
> The latency is high...
>
It is but is it really causing the problem? Latency is high but constant
and not higher than ~200ms.

Regarding the ALTER, did you try to increase the timeout with "cqlsh
> --request-timeout=REQUEST_TIMEOUT"? Because the default is 10 seconds.
>
I use 25sec timeout (--request-timeout 25)

Apart the unreachable nodes, do you know if all nodes have the same schema
> version?
>
"nodetool gossipinfo" shows same scheme version on all nodes.

Best,
>
> Romain
>

Re: unreachable nodes mystery in describecluster output

[Romain Hardouin <ro...@yahoo.fr>]

Hi,
The latency is high...
Regarding the ALTER, did you try to increase the timeout with "cqlsh --request-timeout=REQUEST_TIMEOUT"? Because the default is 10 seconds. Apart the unreachable nodes, do you know if all nodes have the same schema version? 
Best,
Romain