You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Vlad <qa...@yahoo.com.INVALID> on 2018/09/06 19:10:29 UTC

Re: URGENT: disable reads from node

Hi,
this node isn't in system.peers on both nodes.

On Wednesday, August 29, 2018 4:22 PM, Vlad <qa...@yahoo.com.INVALID> wrote:

Hi,
>You'll need to disable the native transportWell, this is what I did already, it seems repair is running

I'm not sure whether repair will finish within 3 hours, but I can run it again (as it's incremental repair by default, right?)

I'm not sure about RF=3 and QUORUM reads because of load/disk space constrains we have, but we'll definitely consider this.

Thanks to all for help!

On Wednesday, August 29, 2018 4:13 PM, Alexander Dejanovski <al...@thelastpickle.com> wrote:

Kurt is right.
So here are the options I can think of : - use the join_ring false technique and rely on hints. You'll need to disable the native transport on the node as well to prevent direct connections to be made to it. Hopefully, you can run repair in less than 3 hours which is the hint window (hints will be collected while the node hasn't joined the ring). Otherwise you'll have more consistency issues after the node joins the ring again. Maybe incremental repair could help fixing this quickly afterwards if you've been running full repairs that involved anticompaction (if you're running at least Cassandra 2.2).- Fully re-bootstrap the node by replacing itself, using the replace_address_first_boot technique (but since you have RF=2, that would most probably mean some data loss since you read/write at ONE)- Try to cheat the dynamic snitch to take the node out of reads. You would then have the node join the ring normally, disable native transport and raise Severity (in org.apache.cassandra.db:type=DynamicEndpointSnitch) to something like 50 so the node won't be selected by the dynamic snitch. I guess the value will reset itself over time so you may need to set it to 50 on a regular basis while repair is happening.
I would then strongly consider moving to RF=3 because RF=2 will lead you to this type of situation again in the future and does not allow quorum reads with fault tolerance. Good luck,
On Wed, Aug 29, 2018 at 1:56 PM Vlad <qa...@yahoo.com.invalid> wrote:

I restarted with cassandra.join_ring=falsenodetool status on other nodes shows this node as DN, while it see itself as UN.

>I'd say best to just query at QUORUM until you can finish repairs.We have RH 2, so I guess QUORUM queries will fail. Also different application should be changed for this.

On Wednesday, August 29, 2018 2:41 PM, kurt greaves <ku...@instaclustr.com> wrote:

Note that you'll miss incoming writes if you do that, so you'll be inconsistent even after the repair. I'd say best to just query at QUORUM until you can finish repairs.
On 29 August 2018 at 21:22, Alexander Dejanovski <al...@thelastpickle.com> wrote:

Hi Vlad, you must restart the node but first disable joining the cluster, as described in the second part of this blog post :
http://thelastpickle.com/blog/ 2018/08/02/Re-Bootstrapping- Without-Bootstrapping.html
Once repaired, you'll have to run "nodetool join" to start serving reads.

Le mer. 29 août 2018 à 12:40, Vlad <qa...@yahoo.com.invalid> a écrit :

Will it help to set read_repair_chance to 1 (compaction is SizeTieredCompactionStrategy)?

On Wednesday, August 29, 2018 1:34 PM, Vlad <qa...@yahoo.com.INVALID> wrote:

Hi,
quite urgent questions:due to disk and C* start problem we were forced to delete commit logs from one of nodes.
Now repair is running, but meanwhile some reads bring no data (RF=2)

Can this node be excluded from reads queries? And that all reads will be redirected to other node in the ring?

Thanks to All for help.

--
-----------------Alexander DejanovskiFrance@alexanderdeja
ConsultantApache Cassandra Consultinghttp://www.thelastpickle.com