Node doesn't join to the ring

[Aleksandr Ivanov <al...@gmail.com>]

I'm trying to join node to the ring with nodetool join command but it fails
with error message about inconsistent replica.
My steps:
1. run cassandra with -Dcassandra.join_ring=false
2. wait couple of minutes
3. ensure that all nodes are in UN state
4. run nodetool join

Joining fails with message:
error: A node required to move the data consistently is down
(/XX.XX.XX.XX). If you wish to move the data from a potentially
inconsistent replica, restart the node with
-Dcassandra.consistent.rangemovement
=false
-- StackTrace --
java.lang.RuntimeException: A node required to move the data consistently
is down (/XX.XX.X.XX). If you wish to move the data from a potentially
inconsistent replica, restart the node with -Dcassandra.con
sistent.rangemovement=false
        at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:275)
        at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:158)
        at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
        ...
I'm pretty sure that node XX.XX.XX.XX is up and reachable over network.
Also no references in log that XX.XX.XX.XX node is went down.
Different tries shows different nodes are "down", but all nodes are from
remote data centers.

Log from XX.XX.XX.XX node:
DEBUG [GossipStage:1] 2016-12-09 16:37:41,557 StorageService.java:1893 -
Node /YY.YY.YY.YY state bootstrapping, token [-1853578429238772836,
-155879016060746037, 8986645362194256101, 7352444016819915166,
-7093258492559265403, ..., -5513577076972264694, 411791028346144716,
-7405822878444068495]
DEBUG [PendingRangeCalculator:1] 2016-12-09 16:37:47,225
PendingRangeCalculatorService.java:66 - finished calculation for 8
keyspaces in 5667ms

C* v3.0.9

Any clues how to troubleshoot it?

Re: Node doesn't join to the ring

[Aleksandr Ivanov <al...@gmail.com>]

Anomaly solved.
Reason was same TCP timeouts on OS level and hardware firewall.
OS TCP timeout changed to smaller value than timeout on firewall and it
fixed problem.

On Fri, Dec 9, 2016 at 7:10 PM Aleksandr Ivanov <al...@gmail.com> wrote:

> I'm trying to join node to the ring with nodetool join command but it
> fails with error message about inconsistent replica.
> My steps:
> 1. run cassandra with -Dcassandra.join_ring=false
> 2. wait couple of minutes
> 3. ensure that all nodes are in UN state
> 4. run nodetool join
>
> Joining fails with message:
> error: A node required to move the data consistently is down
> (/XX.XX.XX.XX). If you wish to move the data from a potentially
> inconsistent replica, restart the node with
> -Dcassandra.consistent.rangemovement
> =false
> -- StackTrace --
> java.lang.RuntimeException: A node required to move the data consistently
> is down (/XX.XX.X.XX). If you wish to move the data from a potentially
> inconsistent replica, restart the node with -Dcassandra.con
> sistent.rangemovement=false
>         at
> org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:275)
>         at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:158)
>         at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
>         ...
> I'm pretty sure that node XX.XX.XX.XX is up and reachable over network.
> Also no references in log that XX.XX.XX.XX node is went down.
> Different tries shows different nodes are "down", but all nodes are from
> remote data centers.
>
> Log from XX.XX.XX.XX node:
> DEBUG [GossipStage:1] 2016-12-09 16:37:41,557 StorageService.java:1893 -
> Node /YY.YY.YY.YY state bootstrapping, token [-1853578429238772836,
> -155879016060746037, 8986645362194256101, 7352444016819915166,
> -7093258492559265403, ..., -5513577076972264694, 411791028346144716,
> -7405822878444068495]
> DEBUG [PendingRangeCalculator:1] 2016-12-09 16:37:47,225
> PendingRangeCalculatorService.java:66 - finished calculation for 8
> keyspaces in 5667ms
>
> C* v3.0.9
>
> Any clues how to troubleshoot it?
>