You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jonathan Ballet <jb...@edgelab.ch> on 2015/12/03 16:51:43 UTC
Rebuilding a new Cassandra node at 100Mb/s
Hi,
I added a new node to my cluster but in a new datacenter. After updating
the keyspace replication factor values (using the
NetworkTopologyStrategy strategy), I'm now running a "nodetool rebuild"
on the new node.
I noticed it's not really fast and my monitoring system shows that the
traffic incoming on this node is exactly at 100Mb/s (12.6MB/s). I know
it can be much more than that (I just tested sending a file through SSH
between the two machines and it goes up to 1Gb/s), is there a limitation
of some sort on Cassandra which limit the transfer rate to 100Mb/s?
I was thinking that "nodetool setstreamthroughput 0" (or even 999) would
help, but it doesn't seem to change anything on the current rebuilding
process.
Any idea?
Thanks!
Jonathan
Re: Rebuilding a new Cassandra node at 100Mb/s
Posted by Sebastian Estevez <se...@datastax.com>.
If you change stream throughput it won't affect currently running streams
but it should affect new ones.
all the best,
Sebastián
On Dec 4, 2015 5:39 AM, "Jonathan Ballet" <jb...@edgelab.ch> wrote:
> Thanks for your answer Rob,
>
> On 12/03/2015 08:32 PM, Robert Coli wrote:
>
>> On Thu, Dec 3, 2015 at 7:51 AM, Jonathan Ballet <jballet@edgelab.ch
>> <ma...@edgelab.ch>> wrote:
>>
>> I noticed it's not really fast and my monitoring system shows that
>> the traffic incoming on this node is exactly at 100Mb/s (12.6MB/s).
>> I know it can be much more than that (I just tested sending a file
>> through SSH between the two machines and it goes up to 1Gb/s), is
>> there a limitation of some sort on Cassandra which limit the
>> transfer rate to 100Mb/s?
>>
>>
>> Probably limited by number of simultaneous parallel streams. Many people
>> do not want streams to go "as fast as possible" because their priority
>> is maintaining baseline service times while rebuilding/bootstrapping.
>>
>> Not sure there's a way to tune it, but this is definitely on the "large
>> node" radar..
>>
>
> I was actually a bit surprised that the limit seems to really be capped at
> 100 Mb/s, not more not less. So I was thinking there was something else
> playing here...
>
> Jonathan
>
Re: Rebuilding a new Cassandra node at 100Mb/s
Posted by Jonathan Ballet <jb...@edgelab.ch>.
Thanks for your answer Rob,
On 12/03/2015 08:32 PM, Robert Coli wrote:
> On Thu, Dec 3, 2015 at 7:51 AM, Jonathan Ballet <jballet@edgelab.ch
> <ma...@edgelab.ch>> wrote:
>
> I noticed it's not really fast and my monitoring system shows that
> the traffic incoming on this node is exactly at 100Mb/s (12.6MB/s).
> I know it can be much more than that (I just tested sending a file
> through SSH between the two machines and it goes up to 1Gb/s), is
> there a limitation of some sort on Cassandra which limit the
> transfer rate to 100Mb/s?
>
>
> Probably limited by number of simultaneous parallel streams. Many people
> do not want streams to go "as fast as possible" because their priority
> is maintaining baseline service times while rebuilding/bootstrapping.
>
> Not sure there's a way to tune it, but this is definitely on the "large
> node" radar..
I was actually a bit surprised that the limit seems to really be capped
at 100 Mb/s, not more not less. So I was thinking there was something
else playing here...
Jonathan
Re: Rebuilding a new Cassandra node at 100Mb/s
Posted by Jonathan Ballet <jb...@edgelab.ch>.
Hey Bryan,
I haven't change this setting, but it looks like this is the same
setting that can be changed with "nodetool setstreamthroughput"?
It sounds pretty interesting at a first glance, but FWIW, the limit was
12.6 MB/s, not 25 MB/s (so effectively 100 Mb/s).
On 12/03/2015 11:40 PM, Bryan Cheng wrote:
> Jonathan: Have you changed stream_throughput_outbound_megabits_per_sec
> in cassandra.yaml?
>
> # Throttles all outbound streaming file transfers on this node to the
> # given total throughput in Mbps. This is necessary because Cassandra does
> # mostly sequential IO when streaming data during bootstrap or repair, which
> # can lead to saturating the network connection and degrading rpc
> performance.
> # When unset, the default is 200 Mbps or 25 MB/s.
> # stream_throughput_outbound_megabits_per_sec: 200
>
>
> On Thu, Dec 3, 2015 at 11:32 AM, Robert Coli <rcoli@eventbrite.com
> <ma...@eventbrite.com>> wrote:
>
> On Thu, Dec 3, 2015 at 7:51 AM, Jonathan Ballet <jballet@edgelab.ch
> <ma...@edgelab.ch>> wrote:
>
> I noticed it's not really fast and my monitoring system shows
> that the traffic incoming on this node is exactly at 100Mb/s
> (12.6MB/s). I know it can be much more than that (I just tested
> sending a file through SSH between the two machines and it goes
> up to 1Gb/s), is there a limitation of some sort on Cassandra
> which limit the transfer rate to 100Mb/s?
>
>
> Probably limited by number of simultaneous parallel streams. Many
> people do not want streams to go "as fast as possible" because their
> priority is maintaining baseline service times while
> rebuilding/bootstrapping.
>
> Not sure there's a way to tune it, but this is definitely on the
> "large node" radar..
>
> =Rob
>
>
Re: Rebuilding a new Cassandra node at 100Mb/s
Posted by Bryan Cheng <br...@blockcypher.com>.
Jonathan: Have you changed stream_throughput_outbound_megabits_per_sec in
cassandra.yaml?
# Throttles all outbound streaming file transfers on this node to the
# given total throughput in Mbps. This is necessary because Cassandra does
# mostly sequential IO when streaming data during bootstrap or repair, which
# can lead to saturating the network connection and degrading rpc
performance.
# When unset, the default is 200 Mbps or 25 MB/s.
# stream_throughput_outbound_megabits_per_sec: 200
On Thu, Dec 3, 2015 at 11:32 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Dec 3, 2015 at 7:51 AM, Jonathan Ballet <jb...@edgelab.ch>
> wrote:
>
>> I noticed it's not really fast and my monitoring system shows that the
>> traffic incoming on this node is exactly at 100Mb/s (12.6MB/s). I know it
>> can be much more than that (I just tested sending a file through SSH
>> between the two machines and it goes up to 1Gb/s), is there a limitation of
>> some sort on Cassandra which limit the transfer rate to 100Mb/s?
>>
>
> Probably limited by number of simultaneous parallel streams. Many people
> do not want streams to go "as fast as possible" because their priority is
> maintaining baseline service times while rebuilding/bootstrapping.
>
> Not sure there's a way to tune it, but this is definitely on the "large
> node" radar..
>
> =Rob
>
>
Re: Rebuilding a new Cassandra node at 100Mb/s
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Dec 3, 2015 at 7:51 AM, Jonathan Ballet <jb...@edgelab.ch> wrote:
> I noticed it's not really fast and my monitoring system shows that the
> traffic incoming on this node is exactly at 100Mb/s (12.6MB/s). I know it
> can be much more than that (I just tested sending a file through SSH
> between the two machines and it goes up to 1Gb/s), is there a limitation of
> some sort on Cassandra which limit the transfer rate to 100Mb/s?
>
Probably limited by number of simultaneous parallel streams. Many people do
not want streams to go "as fast as possible" because their priority is
maintaining baseline service times while rebuilding/bootstrapping.
Not sure there's a way to tune it, but this is definitely on the "large
node" radar..
=Rob