You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Narendra Sharma <na...@gmail.com> on 2013/10/29 19:45:16 UTC
Cassandra 1.1.6 - New node bootstrap not completing
We had a cluster of 4 nodes in AWS. The average load on each node was
approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
node is still in JOINING mode.
Specifically I am analyzing the one with IP 10.3.1.29. There is no
compaction or streaming or index building happening.
$ ./nodetool ring
Note: Ownership information does not include topology, please specify a
keyspace.
Address DC Rack Status State Load Owns
Token
148873535527910577765226390751398592512
10.3.1.179 datacenter1 rack1 Up Normal 740.41 GB
25.00% 0
10.3.1.29 datacenter1 rack1 Up Joining 562.49 GB
0.00% 21267647932558653966460912964485513215
10.3.1.175 datacenter1 rack1 Up Normal 755.7 GB
25.00% 42535295865117307932921825928971026431
10.3.1.30 datacenter1 rack1 Up Joining 565.68 GB
0.00% 63802943797675961899382738893456539648
10.3.1.177 datacenter1 rack1 Up Normal 754.18 GB
25.00% 85070591730234615865843651857942052863
10.3.1.135 datacenter1 rack1 Up Normal 95.97 GB
20.87% 120580289963820081458352857409882669785
10.3.1.178 datacenter1 rack1 Up Normal 747.53 GB
4.13% 127605887595351923798765477786913079295
10.3.1.24 datacenter1 rack1 Up Joining 522.09 GB
0.00% 148873535527910577765226390751398592512
$ ./nodetool netstats
Mode: JOINING
Not sending any streams.
Nothing streaming from /10.3.1.177
Nothing streaming from /10.3.1.179
Pool Name Active Pending Completed
Commands n/a 0 82
Responses n/a 0 40135123
$ ./nodetool compactionStats
pending tasks: 0
Active compaction remaining time : n/a
$ ./nodetool info
Token : 21267647932558653966460912964485513215
Gossip active : true
Thrift active : false
Load : 562.49 GB
Generation No : 1382981644
Uptime (seconds) : 90340
Heap Memory (MB) : 9298.59 / 13272.00
Data Center : datacenter1
Rack : rack1
Exceptions : 2
Key Cache : size 104857584 (bytes), capacity 104857584 (bytes),
187373 hits, 94709046 requests, 0.002 recent hit rate, 14400 save period in
seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
The 2 Exceptions in info output are the ones that were logged when I
stopped index build to let bootstrap complete faster.
Any clue whats wrong and where should I look for to further analyze the
issue? I haven't restarted the Cassandra process. I am afraid the node will
start bootstrap again if I restart the node.
Thanks,
Naren
--
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*
Re: Cassandra 1.1.6 - New node bootstrap not completing
Posted by Chris Burroughs <ch...@gmail.com>.
On 11/01/2013 03:03 PM, Robert Coli wrote:
> On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma
> <na...@gmail.com>wrote:
>
>> I was successfully able to bootstrap the node. The issue was RF > 2.
>> Thanks again Robert.
>>
>
> For the record, I'm not entirely clear why bootstrapping two nodes into the
> same range should have caused your specific bootstrap problem, but I am
> glad to hear that bootstrapping one node at a time was a usable workaround.
>
> =Rob
>
(A) If it can't work shouldn't a node refuse to bootstrap if it sees
another node already in that state?
(B) It would be nice if nodes in independent DCs could at least be
bootstrapped at the same time.
Re: Cassandra 1.1.6 - New node bootstrap not completing
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma
<na...@gmail.com>wrote:
> I was successfully able to bootstrap the node. The issue was RF > 2.
> Thanks again Robert.
>
For the record, I'm not entirely clear why bootstrapping two nodes into the
same range should have caused your specific bootstrap problem, but I am
glad to hear that bootstrapping one node at a time was a usable workaround.
=Rob
Re: Cassandra 1.1.6 - New node bootstrap not completing
Posted by Narendra Sharma <na...@gmail.com>.
I was successfully able to bootstrap the node. The issue was RF > 2. Thanks
again Robert.
On Wed, Oct 30, 2013 at 10:29 AM, Narendra Sharma <narendra.sharma@gmail.com
> wrote:
> Thanks Robert.
>
> I didn't realize that some of the keyspaces (not all and esp. the biggest
> one I was focusing on) had RF > 2. I wasted 3 days on it. Thanks again for
> the pointers. I will try again and share the results.
>
>
> On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli <rc...@eventbrite.com>wrote:
>
>> On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <
>> narendra.sharma@gmail.com> wrote:
>>
>>> We had a cluster of 4 nodes in AWS. The average load on each node was
>>> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
>>> node is still in JOINING mode.
>>> Specifically I am analyzing the one with IP 10.3.1.29. There is no
>>> compaction or streaming or index building happening.
>>>
>>
>> If your cluster has RF>2, you are bootstrapping two nodes into the same
>> range simultaneously. That is not supported. [1,2] The node you are having
>> the problem with is in the range that is probably overlapping.
>>
>> If I were you I would :
>>
>> 1) stop all "Joining" nodes and wipe their state including system keyspace
>> 2) optionally "removetoken" any nodes which remain in cluster gossip
>> state after stopping
>> 3) re-start/bootstrap them one at a time, waiting for each to complete
>> bootstrapping before starting the next one
>> 4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.
>>
>> =Rob
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-2434
>> [2]
>> https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851
>>
>
>
>
> --
> Narendra Sharma
> Software Engineer
> *http://www.aeris.com*
> *http://narendrasharma.blogspot.com/*
>
>
--
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*
Re: Cassandra 1.1.6 - New node bootstrap not completing
Posted by Narendra Sharma <na...@gmail.com>.
Thanks Robert.
I didn't realize that some of the keyspaces (not all and esp. the biggest
one I was focusing on) had RF > 2. I wasted 3 days on it. Thanks again for
the pointers. I will try again and share the results.
On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <
> narendra.sharma@gmail.com> wrote:
>
>> We had a cluster of 4 nodes in AWS. The average load on each node was
>> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
>> node is still in JOINING mode.
>> Specifically I am analyzing the one with IP 10.3.1.29. There is no
>> compaction or streaming or index building happening.
>>
>
> If your cluster has RF>2, you are bootstrapping two nodes into the same
> range simultaneously. That is not supported. [1,2] The node you are having
> the problem with is in the range that is probably overlapping.
>
> If I were you I would :
>
> 1) stop all "Joining" nodes and wipe their state including system keyspace
> 2) optionally "removetoken" any nodes which remain in cluster gossip state
> after stopping
> 3) re-start/bootstrap them one at a time, waiting for each to complete
> bootstrapping before starting the next one
> 4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.
>
> =Rob
> [1] https://issues.apache.org/jira/browse/CASSANDRA-2434
> [2]
> https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851
>
--
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*
Re: Cassandra 1.1.6 - New node bootstrap not completing
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <narendra.sharma@gmail.com
> wrote:
> We had a cluster of 4 nodes in AWS. The average load on each node was
> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
> node is still in JOINING mode.
> Specifically I am analyzing the one with IP 10.3.1.29. There is no
> compaction or streaming or index building happening.
>
If your cluster has RF>2, you are bootstrapping two nodes into the same
range simultaneously. That is not supported. [1,2] The node you are having
the problem with is in the range that is probably overlapping.
If I were you I would :
1) stop all "Joining" nodes and wipe their state including system keyspace
2) optionally "removetoken" any nodes which remain in cluster gossip state
after stopping
3) re-start/bootstrap them one at a time, waiting for each to complete
bootstrapping before starting the next one
4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.
=Rob
[1] https://issues.apache.org/jira/browse/CASSANDRA-2434
[2]
https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851