Re: Bootstrap failure on C* 1.2.13

[Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>]

Hello,

After about 3 months I was able to solve this issue, which happened again
after another node died.

The problem is the datastax 1.2 node replacement docs [1] said that "This
procedure applies to clusters using vnodes. If not using vnodes, use the
instructions in the Cassandra 1.1 documentation".

However, the 1.1 docs did not mention the property
"-Dcassandra.replace_address=address_of_dead_node", which was only
introduced in 1.2. So, what happens without this flag is that the
replacement node tries to stream data from the dead node, failing the
bootstrap process. Adding this flag solves the problem.

Big thanks to driftx from #cassandra who helped troubleshoot the issue. The
docs were already updated to mention the property even for non-vnodes
cluster.

[1]
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_replace_node_t.html

Cheers,

On Sat, Feb 15, 2014 at 3:31 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:

> Hi Rob,
>
> I don't understand how setting those "initial_token" might solve this
> issue. Even more since we cannot set them before bootstrapping...
>
> Plus, once those tokens set, we would have to modify them after any new
> bootstrap / decommission. Which would also imply to run a rolling restart
> for the new configuration (cassandra.yaml)  to be taken into account. This
> is quite a heavy process to perform a "NOOP"...
>
> What did I miss ?
>
> Thanks for getting involved and trying to help anyway :).
>
> Alain
>
>
> 2014-02-15 1:13 GMT+01:00 Robert Coli <rc...@eventbrite.com>:
>
> On Fri, Feb 14, 2014 at 10:08 AM, Paulo Ricardo Motta Gomes <
>> paulo.motta@chaordicsystems.com> wrote:
>>
>>> But in our case, our cluster was not using VNodes, so this workaround
>>> will probably not work with VNodes, since you cannot specify the 256 tokens
>>> from the old node.
>>>
>>
>> Sure you can, in a comma delimited list. I plan to write a short blog
>> post about this, but...
>>
>> I recommend that anyone using Cassandra, vnodes or not, always explicitly
>> populate their initial_token line in cassandra.yaml. There are a number of
>> cases where you will lose if you do not do so, and AFAICT no cases where
>> you lose by doing so.
>>
>> If one is using vnodes and wants to do this, the process goes like :
>>
>> 1) set num_tokens to the desired number of vnodes
>> 2) start node/bootstrap
>> 3) use a one liner like jeffj's :
>> "
>> nodetool info -T | grep ^Token | awk '{ print $3 }' | tr \\n , | sed -e
>> 's/,$/\n/'
>> "
>> to get a comma delimited list of the vnode tokens
>> 4) insert this comma delimited list in initial_token, and comment out
>> num_tokens (though it is a NOOP)
>>
>> =Rob
>>
>
>


-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200