Rebuilding a node without clients hitting it

[Cyril Scetbon <cy...@free.fr>]

Hey guys,

Can you confirm that disabling the native transport (nodetool disablebinary) is enough with Cassandra 3.11+ to avoid clients hitting inconsistent data on that node when they use LOCAL_ONE  consistency ? (Particularly when the node is rebuilding …)
I'd like to avoid any fancy client configuration like blacklisting nodes.

Thanks 
—
Cyril Scetbon

Re: Rebuilding a node without clients hitting it

[Cyril Scetbon <cy...@free.fr>]

Thanks Jeff, that’s the type of parameter I was looking for but I missed it when I first read it. We’ll ensure that dynamic snitch is enabled.
—
Cyril Scetbon

> On Aug 5, 2019, at 11:23 PM, Jeff Jirsa <jj...@gmail.com> wrote:
> 
> You can make THAT less likely with some snitch trickery (setting the badness for the rebuilding host) via jmx 

Re: Rebuilding a node without clients hitting it

[Alexander Dejanovski <al...@thelastpickle.com>]

Hi Cyril,

it will depend on the load balancing policy that is used in the client code.

If you're only accessing DC1, with the node being rebuilt living in DC2,
then you need your clients to be using the DCAwareRoundRobinPolicy to
restrict connections to DC1 and avoid all kind of queries hitting DC2.
If clients are accessing both datacenters, and you're not using the
TokenAwarePolicy, even with LOCAL_ONE, the coordinator could pick the node
being rebuilt to process the query.

If you're not spinning up a new datacenter in an existing cluster,
rebuilding a node is not the best way to achieve this without compromising
consistency.
The node should be replaced, which will make it bootstrap safely (he can
replace himself, using the
"-Dcassandra.replace_address_first_boot=<address_of_replaced_node>" flag.
Bootstrap lets the node stream the data it needs faster than repair would,
while keeping it out of read requests.
The procedure is to stop Cassandra, wipe data, commit log and saved caches,
and then restart it with the JVM flag set in cassandra-env.sh. The node
will appear as joining or down while bootstrapping (it depends if it
replaces itself or another node, can't remember the specifics).
If it shows up as down, it will rely on hints to get the writes. If it
shows as joining, it will get the writes while streaming is ongoing.

Cheers,

-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Tue, Aug 6, 2019 at 12:03 PM Cyril Scetbon <cy...@free.fr> wrote:

> Can you elaborate on that ? We use GPFS
> without cassandra-topology.properties.
> —
> Cyril Scetbon
>
> On Aug 5, 2019, at 11:23 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> some snitch trickery (setting the badness for the rebuilding host) via jmx
>
>
>

Re: Rebuilding a node without clients hitting it

[Cyril Scetbon <cy...@free.fr>]

Can you elaborate on that ? We use GPFS without cassandra-topology.properties.
—
Cyril Scetbon

> On Aug 5, 2019, at 11:23 PM, Jeff Jirsa <jj...@gmail.com> wrote:
> 
> some snitch trickery (setting the badness for the rebuilding host) via jmx 

Re: Rebuilding a node without clients hitting it

[Cyril Scetbon <cy...@free.fr>]

We have clients in all our DCs.

Rebuild has always been much faster for us than repairs. It operates as bootstrap by streaming data from only one source replica for each token range (need to do a cleanup if run multiple times). Repair is a different operation and is not supposed to be run on an empty node. It does more processing, merkle trees comparison, delete tombstones etc…

We use repairs when we add a DC as a new replication source to a Keyspace.
—
Cyril Scetbon

> Assuming the rebuild is happening on a node in another DC, then there should not be an issue if you are using LOCAL_ONE. If the node is in the local DC (i.e., same DC as the client), I am inclined to think repair would be more appropriate than rebuild but I am not 100% certain.
> -- 
> 
> - John

Re: Rebuilding a node without clients hitting it

[John Sanda <jo...@gmail.com>]

Assuming the rebuild is happening on a node in another DC, then there
should not be an issue if you are using LOCAL_ONE. If the node is in the
local DC (i.e., same DC as the client), I am inclined to think repair would
be more appropriate than rebuild but I am not 100% certain.

On Mon, Aug 5, 2019 at 11:23 PM Jeff Jirsa <jj...@gmail.com> wrote:

> No, not strictly sufficient - makes it much less likely though
>
> A client may connect to another node and still send the request to that
> host if the snitch picks it. You can make THAT less likely with some snitch
> trickery (setting the badness for the rebuilding host) via jmx
>
> On Aug 5, 2019, at 8:17 PM, Cyril Scetbon <cy...@free.fr> wrote:
>
> Hey guys,
>
> Can you confirm that disabling the native transport (nodetool
> disablebinary) is enough with Cassandra 3.11+ to avoid clients hitting
> inconsistent data on that node when they use LOCAL_ONE  consistency ?
> (Particularly when the node is rebuilding …)
> I'd like to avoid any fancy client configuration like blacklisting nodes.
>
> Thanks
> —
> Cyril Scetbon
>
>

-- 

- John

Re: Rebuilding a node without clients hitting it

[Jeff Jirsa <jj...@gmail.com>]

No, not strictly sufficient - makes it much less likely though

A client may connect to another node and still send the request to that host if the snitch picks it. You can make THAT less likely with some snitch trickery (setting the badness for the rebuilding host) via jmx 

> On Aug 5, 2019, at 8:17 PM, Cyril Scetbon <cy...@free.fr> wrote:
> 
> Hey guys,
> 
> Can you confirm that disabling the native transport (nodetool disablebinary) is enough with Cassandra 3.11+ to avoid clients hitting inconsistent data on that node when they use LOCAL_ONE  consistency ? (Particularly when the node is rebuilding …)
> I'd like to avoid any fancy client configuration like blacklisting nodes.
> 
> Thanks 
> —
> Cyril Scetbon
>