Can't replace failed node

[Ian Spence <Ia...@globalrelay.net>]

Hello,

ENV: Cassandra: 2.2.9, JRE: 1.8.0_74, CentOS 6/7

We've run into a weird issue as a result of a node dying. A specific server had
a catastrophic failure and lost all data on disk. It's since been replaced but
during that time we've brought down the cluster a couple of times (as this is a
part of our deployment process). I've attempted to bring that node back up
twice, once using the -Dcassandra.replace_address option, and second just
brining the node back up.

The first attempt (with -replace_address) didn't work because of
CASSANDRA-8138, but the second attempt also failed because it tried to
bootstrap, but refused to start because
"Cannot service reads while bootstrapping!"

We can't see the node in nodetool status, ring, or gossipinfo, but we can still
see it in the system.peers table (although I've learned that this table can be
wrong)

How can we re-add this node to the ring?

- Ian