You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anshu Vajpayee <an...@gmail.com> on 2017/11/17 18:02:10 UTC
How quickly we can bootstrap
Cassandra supports elastic scalability - meaning on demand we can increase
or decrease #of nodes as per scaling demand from the application.
Let's consider we have 5 node cluster and each node has data pressure of
about 3 TB.
Now as per sudden load, we want to add 1 node in the cluster as quick as
possible.
Please suggest what would be the fastest method to add the new node on
cluster? Normal bootstrapping will definitely take time because it needs to
stream at least 2.5 TB ( 5*3TB/6 nodes) from 5 nodes. Please
consider multi-core machines & 10 Gpbs card .
Streaming throughput can help but not much.
The similar requirement can come when we want to replace the failed node
due to any hardware.
Please suggest any best practice or scenarios to deal with above
situations.
Scaling is good but how quickly we can scale is another thing to consider.
--
*C*heers,*
*Anshu V*
Re: How quickly we can bootstrap
Posted by Justin Cameron <ju...@instaclustr.com>.
If fast bootstrapping is a high priority then using less dense nodes is the
tradeoff.
For example, rather than having 5 nodes with 3TB you could deploy 15 (less
powerful) nodes with 1TB each. Your cluster will have higher overall disk
throughput which would allow you to increase the stream throughput, and
your new node will not need to stream as much data to bootstrap. The
process will still take some time for a production cluster under load, but
should be significantly faster.
On Mon, 20 Nov 2017 at 16:11 Jon Haddad <jo...@jonhaddad.com> wrote:
> It sounds like you’re asking how to bootstrap without paying the cost of
> bootstrapping :)
>
> If you want to scale out, you’ll need to deal with the time it takes. You
> can’t add a node and have it up in 15 minutes, if you’re running 3 TB it’ll
> take a while. The exact amount of time depends largely on your network and
> disk performance.
>
> Jon
>
>
>
> On Nov 19, 2017, at 11:42 PM, Anshu Vajpayee <an...@gmail.com>
> wrote:
>
> Adding more compute power means again vertical scaling. I understand this
> is one method to handle the load in case of increasing demand. But it
> doesn't match with philosophy of Cassandra for horizontal scaling. Hitting
> capacity cannot be restricted to only compute power.
>
>
> Also in case of node failure, this vertical scaling is not going help.
> Where we need to bootstrap a new or same node faster and quicker in ring.
>
>
> On Sat, Nov 18, 2017 at 4:20 AM, Ben Slater <be...@instaclustr.com>
> wrote:
>
>> Hi Anshu
>>
>> For quick scaling, we’ve had success with an approach of scaling up the
>> compute capacity (attached to EBS) rather than scaling out with more nodes
>> in order to provide relatively quick scale up/down capability. The approach
>> is implemented as part of our managed service but the concept is generic
>> enough to work in any virtualised environment. You can find more detail
>> here if interested:
>> https://www.instaclustr.com/instaclustr-dynamic-resizing-for-apache-cassandra/
>>
>> Cheers
>> Ben
>>
>> On Sat, 18 Nov 2017 at 05:02 Anshu Vajpayee <an...@gmail.com>
>> wrote:
>>
>>> Cassandra supports elastic scalability - meaning on demand we can
>>> increase or decrease #of nodes as per scaling demand from the application.
>>>
>>> Let's consider we have 5 node cluster and each node has data pressure of
>>> about 3 TB.
>>>
>>> Now as per sudden load, we want to add 1 node in the cluster as quick
>>> as possible.
>>>
>>> Please suggest what would be the fastest method to add the new node on
>>> cluster? Normal bootstrapping will definitely take time because it needs to
>>> stream at least 2.5 TB ( 5*3TB/6 nodes) from 5 nodes. Please
>>> consider multi-core machines & 10 Gpbs card .
>>>
>>> Streaming throughput can help but not much.
>>>
>>> The similar requirement can come when we want to replace the failed node
>>> due to any hardware.
>>>
>>> Please suggest any best practice or scenarios to deal with above
>>> situations.
>>>
>>> Scaling is good but how quickly we can scale is another thing to
>>> consider.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> *C*heers,*
>>> *Anshu V*
>>>
>>>
>>> --
>>
>>
>> *Ben Slater*
>>
>> *Chief Product Officer <https://www.instaclustr.com/>*
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information. If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
>
> --
*Justin Cameron*Senior Software Engineer
<https://www.instaclustr.com/>
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
Re: How quickly we can bootstrap
Posted by Jon Haddad <jo...@jonhaddad.com>.
It sounds like you’re asking how to bootstrap without paying the cost of bootstrapping :)
If you want to scale out, you’ll need to deal with the time it takes. You can’t add a node and have it up in 15 minutes, if you’re running 3 TB it’ll take a while. The exact amount of time depends largely on your network and disk performance.
Jon
> On Nov 19, 2017, at 11:42 PM, Anshu Vajpayee <an...@gmail.com> wrote:
>
> Adding more compute power means again vertical scaling. I understand this is one method to handle the load in case of increasing demand. But it doesn't match with philosophy of Cassandra for horizontal scaling. Hitting capacity cannot be restricted to only compute power.
>
> Also in case of node failure, this vertical scaling is not going help. Where we need to bootstrap a new or same node faster and quicker in ring.
>
>
> On Sat, Nov 18, 2017 at 4:20 AM, Ben Slater <ben.slater@instaclustr.com <ma...@instaclustr.com>> wrote:
> Hi Anshu
>
> For quick scaling, we’ve had success with an approach of scaling up the compute capacity (attached to EBS) rather than scaling out with more nodes in order to provide relatively quick scale up/down capability. The approach is implemented as part of our managed service but the concept is generic enough to work in any virtualised environment. You can find more detail here if interested: https://www.instaclustr.com/instaclustr-dynamic-resizing-for-apache-cassandra/ <https://www.instaclustr.com/instaclustr-dynamic-resizing-for-apache-cassandra/>
>
> Cheers
> Ben
>
> On Sat, 18 Nov 2017 at 05:02 Anshu Vajpayee <anshu.vajpayee@gmail.com <ma...@gmail.com>> wrote:
> Cassandra supports elastic scalability - meaning on demand we can increase or decrease #of nodes as per scaling demand from the application.
>
> Let's consider we have 5 node cluster and each node has data pressure of about 3 TB.
>
> Now as per sudden load, we want to add 1 node in the cluster as quick as possible.
>
> Please suggest what would be the fastest method to add the new node on cluster? Normal bootstrapping will definitely take time because it needs to stream at least 2.5 TB ( 5*3TB/6 nodes) from 5 nodes. Please consider multi-core machines & 10 Gpbs card .
>
> Streaming throughput can help but not much.
>
> The similar requirement can come when we want to replace the failed node due to any hardware.
>
> Please suggest any best practice or scenarios to deal with above situations.
>
> Scaling is good but how quickly we can scale is another thing to consider.
>
>
>
>
>
>
>
>
> --
> C*heers,
> Anshu V
>
>
> --
> Ben Slater
> Chief Product Officer
> <https://www.instaclustr.com/>
> <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> <https://www.linkedin.com/company/instaclustr>
> Read our latest technical blog posts here <https://www.instaclustr.com/blog/>.
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA).
> This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
>
>
>
> --
> C*heers,
> Anshu V
>
>
Re: How quickly we can bootstrap
Posted by Anshu Vajpayee <an...@gmail.com>.
Adding more compute power means again vertical scaling. I understand this
is one method to handle the load in case of increasing demand. But it
doesn't match with philosophy of Cassandra for horizontal scaling. Hitting
capacity cannot be restricted to only compute power.
Also in case of node failure, this vertical scaling is not going help.
Where we need to bootstrap a new or same node faster and quicker in ring.
On Sat, Nov 18, 2017 at 4:20 AM, Ben Slater <be...@instaclustr.com>
wrote:
> Hi Anshu
>
> For quick scaling, we’ve had success with an approach of scaling up the
> compute capacity (attached to EBS) rather than scaling out with more nodes
> in order to provide relatively quick scale up/down capability. The approach
> is implemented as part of our managed service but the concept is generic
> enough to work in any virtualised environment. You can find more detail
> here if interested: https://www.instaclustr.com/instaclustr-
> dynamic-resizing-for-apache-cassandra/
>
> Cheers
> Ben
>
> On Sat, 18 Nov 2017 at 05:02 Anshu Vajpayee <an...@gmail.com>
> wrote:
>
>> Cassandra supports elastic scalability - meaning on demand we can
>> increase or decrease #of nodes as per scaling demand from the application.
>>
>> Let's consider we have 5 node cluster and each node has data pressure of
>> about 3 TB.
>>
>> Now as per sudden load, we want to add 1 node in the cluster as quick as
>> possible.
>>
>> Please suggest what would be the fastest method to add the new node on
>> cluster? Normal bootstrapping will definitely take time because it needs to
>> stream at least 2.5 TB ( 5*3TB/6 nodes) from 5 nodes. Please
>> consider multi-core machines & 10 Gpbs card .
>>
>> Streaming throughput can help but not much.
>>
>> The similar requirement can come when we want to replace the failed node
>> due to any hardware.
>>
>> Please suggest any best practice or scenarios to deal with above
>> situations.
>>
>> Scaling is good but how quickly we can scale is another thing to
>> consider.
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> *C*heers,*
>> *Anshu V*
>>
>>
>> --
>
>
> *Ben Slater*
>
> *Chief Product Officer <https://www.instaclustr.com/>*
>
> <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
--
*C*heers,*
*Anshu V*
Re: How quickly we can bootstrap
Posted by Ben Slater <be...@instaclustr.com>.
Hi Anshu
For quick scaling, we’ve had success with an approach of scaling up the
compute capacity (attached to EBS) rather than scaling out with more nodes
in order to provide relatively quick scale up/down capability. The approach
is implemented as part of our managed service but the concept is generic
enough to work in any virtualised environment. You can find more detail
here if interested:
https://www.instaclustr.com/instaclustr-dynamic-resizing-for-apache-cassandra/
Cheers
Ben
On Sat, 18 Nov 2017 at 05:02 Anshu Vajpayee <an...@gmail.com>
wrote:
> Cassandra supports elastic scalability - meaning on demand we can
> increase or decrease #of nodes as per scaling demand from the application.
>
> Let's consider we have 5 node cluster and each node has data pressure of
> about 3 TB.
>
> Now as per sudden load, we want to add 1 node in the cluster as quick as
> possible.
>
> Please suggest what would be the fastest method to add the new node on
> cluster? Normal bootstrapping will definitely take time because it needs to
> stream at least 2.5 TB ( 5*3TB/6 nodes) from 5 nodes. Please
> consider multi-core machines & 10 Gpbs card .
>
> Streaming throughput can help but not much.
>
> The similar requirement can come when we want to replace the failed node
> due to any hardware.
>
> Please suggest any best practice or scenarios to deal with above
> situations.
>
> Scaling is good but how quickly we can scale is another thing to consider.
>
>
>
>
>
>
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
> --
*Ben Slater*
*Chief Product Officer <https://www.instaclustr.com/>*
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.