a simple solution to replace dead node

[Yang <te...@gmail.com>]

when we setup our new Cassandra Cluster recently, we looked at the problem
of what to do
when a node dies and needs to be replaced.

we could either let it be dead and have ops team spin up a new node and
manually assign a token to it;
we could also use Priam.

but our company uses Puppet to manage nodes, instead of EC2 autoscaler, so
Priam was more advanced than
we need or could integrate.

So we came up with a simple script that integrates with Puppet and
"nventory" (Nventory.sourceforge.net),


the idea is very simple, and similar to Priam. for example, we have 16
nodes, nventory makes sure that
they are always given dns names like host1.mycompany.comhost2.mycompany.......
host16.mycompany.com .
if host10 dies and needs to be replaced, Puppet spins up a new box, and the
new box acquires the hostname
"host10.mycompany.com". When the new box boots up, it runs a script, which
queries nventory to figure out its own hostname, and the hostname of its
siblings in the nventory group (kind of similar to the asg idea), it uses
its order in the 16 nodes to calculate its token. and does a
"-Dcassandra.replace_token=......" when starting
cassandra.


the drawback is that you can do automatic launching of the new host. but
that limitation is due to nventor/Puppet.

Just want to share with the community here, cuz many people probably run
Puppet+nventory, so this simple solution might work well


Thanks
Yang