You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Troels Arvin <tr...@arvin.dk> on 2019/02/18 10:53:54 UTC
Bootstrap stuck in JOINING state?
Hello,
Nine days ago, I bootstrapped an Cassandra node, let's call it
10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster
spread across two datacenters. The bootstrapping was conducted the
following way on the node:
1. Shut down Cassandra.
2. Deleted its data.
3. Temporarily adjusted its configuration to be in bootstrapping node:
- In cassandra-env.sh:
- JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode"
- In cassandra.yaml:
- auto_bootstrap: true
- Made sure the nodes' IP address was not listed in seed_provider
4. Started Cassandra again.
That seemed to go well: In system.log the following was seen:
INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING:
Starting to bootstrap...
No errors seen in system.log.
A graph of the server's data filesystem shows fine growth during the
next ~24 hours.
However, it seems the joining is stuck without completing.
Parts of output from "nodetool netstats", note the "JOINING" mode:
================================================================
[cas@nodename ~]$ nodetool netstats
Mode: JOINING
Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701
/10.1.2.10
/10.1.1.10
/10.1.1.12
/10.1.2.12
Receiving 225 files, 461265115573 bytes total. Already received
225 files, 461265115573 bytes total
/data/cassandra/ksname/...db 43392372/43392372 bytes(100%)
received from idx:0/10.1.2.12
[... lots of lines like this ...]
/data/cassandra/ksname/...db 131825151/131825151 bytes(100%)
received from idx:0/10.1.2.12
/10.1.1.11
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 5
Responses n/a 0 3153171
================================================================
All the "...received from..." lines show 100% bytes received.
Parts of output from "nodetool status":
================================================================
[cas@nodename ~]$ nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1
UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1
UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1
UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1
UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1
================================================================
How come "nodetool netstats" is still showing "Mode: JOINING"? Is there
a way I can push the node to complete its joining?
--
Kind regards,
Troels Arvin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Bootstrap stuck in JOINING state?
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo
ining-operation
https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j
oining-the-cluster
-----Original Message-----
From: Kenneth Brotman [mailto:kenbrotman@yahoo.com.INVALID]
Sent: Monday, February 18, 2019 7:43 AM
To: user@cassandra.apache.org
Subject: RE: Bootstrap stuck in JOINING state?
Hi Troels,
Stackoverflow.com is a good resource for such situations. Have you seen
these posts? I think they are same Cassandra version:
https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo
ining-operation
https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j
oining-the-cluster
Kenneth Brotman
-----Original Message-----
From: Troels Arvin [mailto:troels@arvin.dk]
Sent: Monday, February 18, 2019 2:54 AM
To: user@cassandra.apache.org
Subject: Bootstrap stuck in JOINING state?
Hello,
Nine days ago, I bootstrapped an Cassandra node, let's call it
10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster
spread across two datacenters. The bootstrapping was conducted the
following way on the node:
1. Shut down Cassandra.
2. Deleted its data.
3. Temporarily adjusted its configuration to be in bootstrapping node:
- In cassandra-env.sh:
- JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode"
- In cassandra.yaml:
- auto_bootstrap: true
- Made sure the nodes' IP address was not listed in seed_provider
4. Started Cassandra again.
That seemed to go well: In system.log the following was seen:
INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING:
Starting to bootstrap...
No errors seen in system.log.
A graph of the server's data filesystem shows fine growth during the
next ~24 hours.
However, it seems the joining is stuck without completing.
Parts of output from "nodetool netstats", note the "JOINING" mode:
================================================================
[cas@nodename ~]$ nodetool netstats
Mode: JOINING
Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701
/10.1.2.10
/10.1.1.10
/10.1.1.12
/10.1.2.12
Receiving 225 files, 461265115573 bytes total. Already received
225 files, 461265115573 bytes total
/data/cassandra/ksname/...db 43392372/43392372 bytes(100%)
received from idx:0/10.1.2.12
[... lots of lines like this ...]
/data/cassandra/ksname/...db 131825151/131825151 bytes(100%)
received from idx:0/10.1.2.12
/10.1.1.11
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 5
Responses n/a 0 3153171
================================================================
All the "...received from..." lines show 100% bytes received.
Parts of output from "nodetool status":
================================================================
[cas@nodename ~]$ nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1
UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1
UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1
UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1
UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1
================================================================
How come "nodetool netstats" is still showing "Mode: JOINING"? Is there
a way I can push the node to complete its joining?
--
Kind regards,
Troels Arvin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Bootstrap stuck in JOINING state?
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Hi Troels,
Stackoverflow.com is a good resource for such situations. Have you seen
these posts? I think they are same Cassandra version:
https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo
ining-operation
https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j
oining-the-cluster
Kenneth Brotman
-----Original Message-----
From: Troels Arvin [mailto:troels@arvin.dk]
Sent: Monday, February 18, 2019 2:54 AM
To: user@cassandra.apache.org
Subject: Bootstrap stuck in JOINING state?
Hello,
Nine days ago, I bootstrapped an Cassandra node, let's call it
10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster
spread across two datacenters. The bootstrapping was conducted the
following way on the node:
1. Shut down Cassandra.
2. Deleted its data.
3. Temporarily adjusted its configuration to be in bootstrapping node:
- In cassandra-env.sh:
- JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode"
- In cassandra.yaml:
- auto_bootstrap: true
- Made sure the nodes' IP address was not listed in seed_provider
4. Started Cassandra again.
That seemed to go well: In system.log the following was seen:
INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING:
Starting to bootstrap...
No errors seen in system.log.
A graph of the server's data filesystem shows fine growth during the
next ~24 hours.
However, it seems the joining is stuck without completing.
Parts of output from "nodetool netstats", note the "JOINING" mode:
================================================================
[cas@nodename ~]$ nodetool netstats
Mode: JOINING
Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701
/10.1.2.10
/10.1.1.10
/10.1.1.12
/10.1.2.12
Receiving 225 files, 461265115573 bytes total. Already received
225 files, 461265115573 bytes total
/data/cassandra/ksname/...db 43392372/43392372 bytes(100%)
received from idx:0/10.1.2.12
[... lots of lines like this ...]
/data/cassandra/ksname/...db 131825151/131825151 bytes(100%)
received from idx:0/10.1.2.12
/10.1.1.11
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 5
Responses n/a 0 3153171
================================================================
All the "...received from..." lines show 100% bytes received.
Parts of output from "nodetool status":
================================================================
[cas@nodename ~]$ nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1
UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1
UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1
UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1
UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1
================================================================
How come "nodetool netstats" is still showing "Mode: JOINING"? Is there
a way I can push the node to complete its joining?
--
Kind regards,
Troels Arvin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org