Cassandra 2.1 bootstrap - No streaming progress from one node

["Steinmaurer, Thomas" <th...@dynatrace.com>]

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting as source for streaming got restarted unfortunately. Since then, from nodetool netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
        Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not doing any resume here, in case there have been e.g. connectivity troubles or in our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313

Re: Cassandra 2.1 bootstrap - No streaming progress from one node

[Alain RODRIGUEZ <ar...@gmail.com>]

Hello,

+1 with Sean above.
In Cassandra 2.2 you got the new 'nodetool resume' command to resume a
bootstrap. For C*2.1, nothing equivalent sadly. Thus old school technics
apply and the possible alternatives are

Option 1 - safe and slow- Stop Cassandra from the stuck joining node.
Remove everything (comitlog/data), restart bootstrap, mentioned by Sean
above.
Option 2 - More wild / hopefully quicker - Stop the joining node and start
it with 'auto_bootstrap: false'. The node joins with missing data, that you
can repair after. Yet this can be done without inconsistencies only using a
strong consistency (CL.R+CL.W > RF). This node will be read from but is
then not enough to induce a stale read.

Unless you're confident or have a huge amount of data and most of it made
it to the new node already, I would stick with the option 1, safe and slow
(and upgrade soon ;-))

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le mer. 7 nov. 2018 à 19:22, Durity, Sean R <SE...@homedepot.com> a
écrit :

> I would wipe the new node and bootstrap again. I do not know of any way to
> resume the streaming that was previously in progress.
>
>
>
>
>
> Sean Durity
>
> *From:* Steinmaurer, Thomas <th...@dynatrace.com>
> *Sent:* Wednesday, November 07, 2018 5:13 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress
> from one node
>
>
>
> Hello,
>
>
>
> while bootstrapping a new node into an existing cluster, a node which is
> acting as source for streaming got restarted unfortunately. Since then,
> from nodetool netstats I don’t see any progress for this particular node
> anymore.
>
>
>
> E.g.:
>
>
>
> /X.X.X.X
>
>         Receiving 94 files, 260.09 GB total. Already received 26 files,
> 69.33 GB total
>
>
>
> Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our
> case) not doing any resume here, in case there have been e.g. connectivity
> troubles or in our case, Cassandra on the node acting as stream source got
> restarted?
>
>
>
> Can I force the joining node to recover connection to X.X.X.X or do I need
> to restart the bootstrap via restart on the new node from scratch?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: Cassandra 2.1 bootstrap - No streaming progress from one node

["Durity, Sean R" <SE...@homedepot.com>]

I would wipe the new node and bootstrap again. I do not know of any way to resume the streaming that was previously in progress.


Sean Durity
From: Steinmaurer, Thomas <th...@dynatrace.com>
Sent: Wednesday, November 07, 2018 5:13 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress from one node

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting as source for streaming got restarted unfortunately. Since then, from nodetool netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
        Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not doing any resume here, in case there have been e.g. connectivity troubles or in our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.