Adding New Nodes to a Production Cluster

[Eran Chinthaka Withana <er...@gmail.com>]

Hi,

We have a production cluster with few nodes in each data center. Each node
is being contacted in each data center to serve front end requests. I have
a question about the method adding new nodes to the cluster (say, to
improve RF or scalability). AFAIK, there are two methods to do this.

1. Bring up the node, with no data in it, and let it get the data from the
peers. But the issue with this problem is all the requests coming into this
node can not be served until it gets all the data. This is a big impact for
front end.
2. Copy all the data into the new node from other nodes (we can be little
bit smart here selecting which nodes to copy from), bring it up and run
cleanup and repair. The issue with this approach is all the data will be
around 1TB and it will take a consider amount of time to copy them all and
also because of the rate of updates happening its sometimes hard to shadow
the time to copy the data in to this node.

Is there an option in Cassandra, where I can bring a node up, just like in
#1, but ask peers not to send any read request to it? Or may be apart from
all these, are there better option to handle new nodes?

Also, I think I can use a similar method when a node goes down (say, disk
or RAID failure) for a longer period of time.

Please help me to find an answer to this.

Thanks,
Eran Chinthaka Withana

Re: Adding New Nodes to a Production Cluster

[Eran Chinthaka Withana <er...@gmail.com>]

Thanks Robin for the insight. I thought the node will start serving
requests before the bootstrap process completes. Seems like I'm wrong. I
got this verified thru #cassandra too.

Thanks,
Eran Chinthaka Withana


On Wed, Jun 27, 2012 at 2:44 PM, Robin Verlangen <ro...@us2.nl> wrote:

> Hi Eran,
>
> As far as I'm aware of a node will not serve requests until the bootstrap
> (starts automatically these days) has been completed. So the problem of #1
> is not really there Solution #2 is not straight forward and easy to make
> mistakes.
>
> When you're concerned about read consistency use a proper consistency
> level, like QUORUM or LOCAL_QUORUM.
>
> Cheers!
>
>
> 2012/6/27 Eran Chinthaka Withana <er...@gmail.com>
>
>> Hi,
>>
>> We have a production cluster with few nodes in each data center. Each
>> node is being contacted in each data center to serve front end requests. I
>> have a question about the method adding new nodes to the cluster (say, to
>> improve RF or scalability). AFAIK, there are two methods to do this.
>>
>> 1. Bring up the node, with no data in it, and let it get the data from
>> the peers. But the issue with this problem is all the requests coming into
>> this node can not be served until it gets all the data. This is a big
>> impact for front end.
>> 2. Copy all the data into the new node from other nodes (we can be little
>> bit smart here selecting which nodes to copy from), bring it up and run
>> cleanup and repair. The issue with this approach is all the data will be
>> around 1TB and it will take a consider amount of time to copy them all and
>> also because of the rate of updates happening its sometimes hard to shadow
>> the time to copy the data in to this node.
>>
>> Is there an option in Cassandra, where I can bring a node up, just like
>> in #1, but ask peers not to send any read request to it? Or may be apart
>> from all these, are there better option to handle new nodes?
>>
>> Also, I think I can use a similar method when a node goes down (say, disk
>> or RAID failure) for a longer period of time.
>>
>> Please help me to find an answer to this.
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>
>
>
> --
> With kind regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>

Re: Adding New Nodes to a Production Cluster

[Robin Verlangen <ro...@us2.nl>]

Hi Eran,

As far as I'm aware of a node will not serve requests until the bootstrap
(starts automatically these days) has been completed. So the problem of #1
is not really there Solution #2 is not straight forward and easy to make
mistakes.

When you're concerned about read consistency use a proper consistency
level, like QUORUM or LOCAL_QUORUM.

Cheers!

2012/6/27 Eran Chinthaka Withana <er...@gmail.com>

> Hi,
>
> We have a production cluster with few nodes in each data center. Each node
> is being contacted in each data center to serve front end requests. I have
> a question about the method adding new nodes to the cluster (say, to
> improve RF or scalability). AFAIK, there are two methods to do this.
>
> 1. Bring up the node, with no data in it, and let it get the data from the
> peers. But the issue with this problem is all the requests coming into this
> node can not be served until it gets all the data. This is a big impact for
> front end.
> 2. Copy all the data into the new node from other nodes (we can be little
> bit smart here selecting which nodes to copy from), bring it up and run
> cleanup and repair. The issue with this approach is all the data will be
> around 1TB and it will take a consider amount of time to copy them all and
> also because of the rate of updates happening its sometimes hard to shadow
> the time to copy the data in to this node.
>
> Is there an option in Cassandra, where I can bring a node up, just like in
> #1, but ask peers not to send any read request to it? Or may be apart from
> all these, are there better option to handle new nodes?
>
> Also, I think I can use a similar method when a node goes down (say, disk
> or RAID failure) for a longer period of time.
>
> Please help me to find an answer to this.
>
> Thanks,
> Eran Chinthaka Withana
>



-- 
With kind regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.