You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gene Robichaux <Ge...@match.com> on 2014/09/24 20:01:39 UTC

Node Joining, Not Streaming

I just added two nodes, one in DC-A and one in DC-B.

The node in DC-A started and immediately started to stream files from its piers. The node in DC-B has been in the JOINING state for nearly 24 hours and I have not seen any streams started.

This cluster is in-house spanning 2 DCs. Using DataStax 4.0.3 (Cassandra 2.0.7). There are no errors reported, it just sits at JOINING. Here is the GossipInfo for the cluster.

Any ideas?

/10.32.1.158
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra"}
  RPC_ADDRESS:0.0.0.0
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  LOAD:2.8806352661E10
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.0
  HOST_ID:c1b4b919-e26e-4d34-b629-b531f9878fce
  DC:DPT
  STATUS:BOOT,-6892935543732432942
/10.32.1.157
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:2.1943389849E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.12536564469337463
  DC:DPT
  STATUS:NORMAL,4990283037218436151
  HOST_ID:472131ef-80ca-46bb-a9bd-6a04da3ceb30
/10.32.1.156
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:2.280831696114E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.12552301585674286
  DC:DPT
  STATUS:NORMAL,378597018791048251
  HOST_ID:2b1fc7a6-316a-4235-bfa9-2ff0f4f4de33
/10.32.1.155
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:2.141134386354E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.08361203968524933
  DC:DPT
  STATUS:NORMAL,-8844775018063727558
  HOST_ID:33ca33d6-384d-45d7-8ce9-08be7292f487
/10.1.1.152
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:1.396792762048E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.0
  DC:CLV
  STATUS:NORMAL,6140487838204514978
  HOST_ID:69868030-4c2a-4261-9586-1cc0c4423959
/10.1.1.153
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra"}
  LOAD:7.66529213407E11
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.0
  DC:CLV
  STATUS:BOOT,-5381045770254015948
  HOST_ID:90fd4296-90a2-44d3-8f5d-eda3479feddb
/10.1.1.150
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:1.776464445614E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.12552301585674286
  DC:CLV
  STATUS:NORMAL,1528801819777127078
  HOST_ID:e2a074fa-59c7-449f-b87d-d17f303f8774
/10.1.1.151
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:2.153245271245E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.0
  DC:CLV
  STATUS:NORMAL,-3082884198650260828
  HOST_ID:459aa667-39a5-46e7-9374-62f90108cb65
/10.1.1.154
  NET_VERSION:7
  RACK:RAC1
  X_11_PADDING:{"workload":"Cassandra","active":"true"}
  LOAD:1.304807703954E12
  SCHEMA:90809305-a44e-3351-862d-f80dc7903ae1
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:2.0.7.31
  SEVERITY:0.0
  DC:CLV
  STATUS:NORMAL,-5609228167613478907
  HOST_ID:f48c9bde-501f-496d-8f0f-fa92a03ebe7e

Gene Robichaux
Manager, Database Operations
8300 Douglas Avenue I Suite 800 I Dallas, TX  75225
Phone: 214-576-3273

Re: Node Joining, Not Streaming

Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Sep 24, 2014 at 11:01 AM, Gene Robichaux <Ge...@match.com>
wrote:

>  I just added two nodes, one in DC-A and one in DC-B.
>
>
>
> The node in DC-A started and immediately started to stream files from its
> piers. The node in DC-B has been in the JOINING state for nearly 24 hours
> and I have not seen any streams started.
>

Adding more than one node at a time is not really supported, and you can
end up in bad cases.

Future versions of Cassandra will Strongly Discourage you from doing this.

https://issues.apache.org/jira/browse/CASSANDRA-7069

If I were you, I would :

1) stop the DC-B node's bootstrap by stopping it and wiping its partially
bootstrapped state
2) wait for DC-A to finish bootstrapping
3) re-bootstrap DC-B node.

=Rob
http://twitter.com/rcolidba