You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Peng Xiao <25...@qq.com> on 2017/10/31 11:18:47 UTC
Tuning bootstrap new node
Dear All,
Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
Could you please advise how to tune this ?
Many Thanks,
Peng Xiao
Re: Tuning bootstrap new node
Posted by Jon Haddad <jo...@jonhaddad.com>.
Of all the settings you could change, why one that’s related to memtables? Streaming doesn’t go through the write path, memtables aren’t involved unless you’re using materialized views or CDC.
> On Oct 31, 2017, at 11:44 AM, Anubhav Kale <An...@microsoft.com.INVALID> wrote:
>
> You can change YAML setting of memtable_cleanup_threshold to 0.7 (from the default of 0.3). This will push SSTables to disk less often and will reduce the compaction time.
>
> While this won’t change the streaming time, it will reduce the overall time for your node to be healthy.
>
> From: Harikrishnan Pillai [mailto:HPillai@walmartlabs.com]
> Sent: Tuesday, October 31, 2017 11:28 AM
> To: user@cassandra.apache.org
> Subject: Re: Re: Tuning bootstrap new node
>
> There is no magic in speeding up the node addition other than increasing stream throughput and compaction throughput.
>
> it has been noticed that with heavy compactions the latency may go up if the node also start serving data.
>
> if you really don't want this node to service traffic till all compactions settle down, you can disable gossip and binary protocol using the nodetool command. This will allow compactions to continue but requires a repair to fix the stale data later.
>
> Regards
>
> Hari
>
>
>
> From: Nitan Kainth <nitankainth@gmail.com <ma...@gmail.com>>
> Sent: Tuesday, October 31, 2017 5:47 AM
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: EXT: Re: Tuning bootstrap new node
>
> Do not stop compaction, you will end up with thousands of sstables.
>
> You increase stream throughput from default 200 to a heifer value if your network can handle it.
>
> Sent from my iPhone
>
> On Oct 31, 2017, at 6:35 AM, Peng Xiao <2535053@qq.com <ma...@qq.com>> wrote:
>
> Can we stop the compaction during the new node bootstraping and enable it after the new node joined?
>
> Thanks
> ------------------ Original ------------------
> From: "我自己的邮箱";<2535053@qq.com <ma...@qq.com>>;
> Date: Tue, Oct 31, 2017 07:18 PM
> To: "user"<user@cassandra.apache.org <ma...@cassandra.apache.org>>;
> Subject: Tuning bootstrap new node
>
> Dear All,
>
> Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
> Could you please advise how to tune this ?
>
> Many Thanks,
> Peng Xiao
RE: Re: Tuning bootstrap new node
Posted by Anubhav Kale <An...@microsoft.com.INVALID>.
You can change YAML setting of memtable_cleanup_threshold to 0.7 (from the default of 0.3). This will push SSTables to disk less often and will reduce the compaction time.
While this won’t change the streaming time, it will reduce the overall time for your node to be healthy.
From: Harikrishnan Pillai [mailto:HPillai@walmartlabs.com]
Sent: Tuesday, October 31, 2017 11:28 AM
To: user@cassandra.apache.org
Subject: Re: Re: Tuning bootstrap new node
There is no magic in speeding up the node addition other than increasing stream throughput and compaction throughput.
it has been noticed that with heavy compactions the latency may go up if the node also start serving data.
if you really don't want this node to service traffic till all compactions settle down, you can disable gossip and binary protocol using the nodetool command. This will allow compactions to continue but requires a repair to fix the stale data later.
Regards
Hari
________________________________
From: Nitan Kainth <ni...@gmail.com>>
Sent: Tuesday, October 31, 2017 5:47 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: EXT: Re: Tuning bootstrap new node
Do not stop compaction, you will end up with thousands of sstables.
You increase stream throughput from default 200 to a heifer value if your network can handle it.
Sent from my iPhone
On Oct 31, 2017, at 6:35 AM, Peng Xiao <25...@qq.com>> wrote:
Can we stop the compaction during the new node bootstraping and enable it after the new node joined?
Thanks
------------------ Original ------------------
From: "我自己的邮箱";<25...@qq.com>>;
Date: Tue, Oct 31, 2017 07:18 PM
To: "user"<us...@cassandra.apache.org>>;
Subject: Tuning bootstrap new node
Dear All,
Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
Could you please advise how to tune this ?
Many Thanks,
Peng Xiao
Re: Re: Tuning bootstrap new node
Posted by Harikrishnan Pillai <HP...@walmartlabs.com>.
There is no magic in speeding up the node addition other than increasing stream throughput and compaction throughput.
it has been noticed that with heavy compactions the latency may go up if the node also start serving data.
if you really don't want this node to service traffic till all compactions settle down, you can disable gossip and binary protocol using the nodetool command. This will allow compactions to continue but requires a repair to fix the stale data later.
Regards
Hari
________________________________
From: Nitan Kainth <ni...@gmail.com>
Sent: Tuesday, October 31, 2017 5:47 AM
To: user@cassandra.apache.org
Subject: EXT: Re: Tuning bootstrap new node
Do not stop compaction, you will end up with thousands of sstables.
You increase stream throughput from default 200 to a heifer value if your network can handle it.
Sent from my iPhone
On Oct 31, 2017, at 6:35 AM, Peng Xiao <25...@qq.com>> wrote:
Can we stop the compaction during the new node bootstraping and enable it after the new node joined?
Thanks
------------------ Original ------------------
From: "我自己的邮箱";<25...@qq.com>>;
Date: Tue, Oct 31, 2017 07:18 PM
To: "user"<us...@cassandra.apache.org>>;
Subject: Tuning bootstrap new node
Dear All,
Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
Could you please advise how to tune this ?
Many Thanks,
Peng Xiao
Re: Tuning bootstrap new node
Posted by Nitan Kainth <ni...@gmail.com>.
Do not stop compaction, you will end up with thousands of sstables.
You increase stream throughput from default 200 to a heifer value if your network can handle it.
Sent from my iPhone
> On Oct 31, 2017, at 6:35 AM, Peng Xiao <25...@qq.com> wrote:
>
> Can we stop the compaction during the new node bootstraping and enable it after the new node joined?
>
> Thanks
> ------------------ Original ------------------
> From: "我自己的邮箱";<25...@qq.com>;
> Date: Tue, Oct 31, 2017 07:18 PM
> To: "user"<us...@cassandra.apache.org>;
> Subject: Tuning bootstrap new node
>
> Dear All,
>
> Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
> Could you please advise how to tune this ?
>
> Many Thanks,
> Peng Xiao
Re: Tuning bootstrap new node
Posted by Peng Xiao <25...@qq.com>.
Can we stop the compaction during the new node bootstraping and enable it after the new node joined?
Thanks
------------------ Original ------------------
From: "我自己的邮箱";<25...@qq.com>;
Date: Tue, Oct 31, 2017 07:18 PM
To: "user"<us...@cassandra.apache.org>;
Subject: Tuning bootstrap new node
Dear All,
Can we make some tuning to make bootstrap new node more quick?We have a three DC cluster(RF=3 in two DCs,RF=1 in another ,48 nodes in the DC with RF=3).As the Cluster is becoming larger and larger,we need to spend more than 24 hours to bootstrap a new node.
Could you please advise how to tune this ?
Many Thanks,
Peng Xiao