Re: Cassandra 1.1.6 - New node bootstrap not completing

[Narendra Sharma <na...@gmail.com>]

I was successfully able to bootstrap the node. The issue was RF > 2. Thanks
again Robert.


On Wed, Oct 30, 2013 at 10:29 AM, Narendra Sharma <narendra.sharma@gmail.com
> wrote:

> Thanks Robert.
>
> I didn't realize that some of the keyspaces (not all and esp. the biggest
> one I was focusing on) had RF > 2. I wasted 3 days on it. Thanks again for
> the pointers. I will try again and share the results.
>
>
> On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli <rc...@eventbrite.com>wrote:
>
>> On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <
>> narendra.sharma@gmail.com> wrote:
>>
>>> We had a cluster of 4 nodes in AWS. The average load on each node was
>>> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
>>> node is still in JOINING mode.
>>> Specifically I am analyzing the one with IP 10.3.1.29. There is no
>>> compaction or streaming or index building happening.
>>>
>>
>> If your cluster has RF>2, you are bootstrapping two nodes into the same
>> range simultaneously. That is not supported. [1,2] The node you are having
>> the problem with is in the range that is probably overlapping.
>>
>> If I were you I would :
>>
>> 1) stop all "Joining" nodes and wipe their state including system keyspace
>> 2) optionally "removetoken" any nodes which remain in cluster gossip
>> state after stopping
>> 3) re-start/bootstrap them one at a time, waiting for each to complete
>> bootstrapping before starting the next  one
>> 4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.
>>
>> =Rob
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-2434
>> [2]
>> https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851
>>
>
>
>
> --
> Narendra Sharma
> Software Engineer
> *http://www.aeris.com*
> *http://narendrasharma.blogspot.com/*
>
>


-- 
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*

Re: Cassandra 1.1.6 - New node bootstrap not completing

[Chris Burroughs <ch...@gmail.com>]

On 11/01/2013 03:03 PM, Robert Coli wrote:
> On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma
> <na...@gmail.com>wrote:
>
>> I was successfully able to bootstrap the node. The issue was RF > 2.
>> Thanks again Robert.
>>
>
> For the record, I'm not entirely clear why bootstrapping two nodes into the
> same range should have caused your specific bootstrap problem, but I am
> glad to hear that bootstrapping one node at a time was a usable workaround.
>
> =Rob
>

(A) If it can't work shouldn't a node refuse to bootstrap if it sees 
another node already in that state?

(B) It would be nice if nodes in independent DCs could at least be 
bootstrapped at the same time.

Re: Cassandra 1.1.6 - New node bootstrap not completing

[Robert Coli <rc...@eventbrite.com>]

On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma
<na...@gmail.com>wrote:

> I was successfully able to bootstrap the node. The issue was RF > 2.
> Thanks again Robert.
>

For the record, I'm not entirely clear why bootstrapping two nodes into the
same range should have caused your specific bootstrap problem, but I am
glad to hear that bootstrapping one node at a time was a usable workaround.

=Rob