streaming got stuck at bootstrapping process

[Analia Lorenzatto <an...@gmail.com>]

Hello,

I have a cassandra cluster comprised of 5 nodes (version 1.2.13) configured
with vnodes. One of these is down.  So I've followed these steps to replace
it:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_replace_node_t.html

It was nicely being bootstrapped, but after one day teh streaming got
stuck.  I do not see timeout, heapsize, exceptions errors.  I just see
messages like this:

10.x.y.1 = dead node
10.x.y.2 = node to be bootstrapped

INFO 21:59:46,875 InetAddress /10.x.y.1 is now DOWN
 INFO 21:59:46,875 FatClient /10.x.y.1 has been silent for 30000ms,
removing from gossip
 INFO 22:00:47,118 Node /10.x.y.1 is now part of the cluster
 INFO 22:00:47,119 InetAddress /10.x.y.1 is now UP
 WARN 22:00:47,121 Not updating token metadata for /10.x.y.1 because I am
replacing it
 INFO 22:00:47,121 Nodes /10.x.y.1 and server.dom.net/10.x.y.2 have the
same token -1067333895391196053.  Ignoring /10.x.y.1

I am not sure if they are ok.

The output of "nodetool netstats" shows me a big list of files to be
streamed with 0%  progress.

I am out of ideas.. the node got stuck streaming but it is not already
bootstrapped.  I tried this process more than once with the same result.
Any suggestion?

Thanks in advance!


-- 
Saludos / Regards.

Analía Lorenzatto.

“It's possible to commit no errors and still lose. That is not weakness.
That is life".  By Captain Jean-Luc Picard.

Re: streaming got stuck at bootstrapping process

[Ryan Svihla <rs...@datastax.com>]

Analia,

I believe your issues is most likely related to a couple of known issues in
1.2.13

https://issues.apache.org/jira/browse/CASSANDRA-6648
https://issues.apache.org/jira/browse/CASSANDRA-6615

I'm not sure of the internals myself, but I've observed similar behavior on
a number of 1.2.13 clusters. I suggest an upgrade.

If you'd like to respond I suggest doing so on the user's mailing list and
not the developers as this is for commiters.


On Tue, Dec 9, 2014 at 10:58 AM, Analia Lorenzatto <
analialorenzatto@gmail.com> wrote:
>
> Hello,
>
> I have a cassandra cluster comprised of 5 nodes (version 1.2.13) configured
> with vnodes. One of these is down.  So I've followed these steps to replace
> it:
>
> http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_replace_node_t.html
>
> It was nicely being bootstrapped, but after one day teh streaming got
> stuck.  I do not see timeout, heapsize, exceptions errors.  I just see
> messages like this:
>
> 10.x.y.1 = dead node
> 10.x.y.2 = node to be bootstrapped
>
> INFO 21:59:46,875 InetAddress /10.x.y.1 is now DOWN
>  INFO 21:59:46,875 FatClient /10.x.y.1 has been silent for 30000ms,
> removing from gossip
>  INFO 22:00:47,118 Node /10.x.y.1 is now part of the cluster
>  INFO 22:00:47,119 InetAddress /10.x.y.1 is now UP
>  WARN 22:00:47,121 Not updating token metadata for /10.x.y.1 because I am
> replacing it
>  INFO 22:00:47,121 Nodes /10.x.y.1 and server.dom.net/10.x.y.2 have the
> same token -1067333895391196053.  Ignoring /10.x.y.1
>
> I am not sure if they are ok.
>
> The output of "nodetool netstats" shows me a big list of files to be
> streamed with 0%  progress.
>
> I am out of ideas.. the node got stuck streaming but it is not already
> bootstrapped.  I tried this process more than once with the same result.
> Any suggestion?
>
> Thanks in advance!
>
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>


-- 

[image: datastax_logo.png] <http://www.datastax.com/>

Ryan Svihla

Solution Architect

[image: twitter.png] <https://twitter.com/foundev> [image: linkedin.png]
<http://www.linkedin.com/pub/ryan-svihla/12/621/727/>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.