You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2015/04/21 03:08:54 UTC
Bootstrap performance.
Hi guys,
We have a 100+ nodes cluster, each node has about 400G data, and is running
on a flash disk. We are running 2.1.2.
When I bring in a new node into the cluster, it introduces significant load
to the cluster. For the new node, the cpu usage is 100%, but disk write io
is only around 50MB/s, while we have 10G network.
Does it sound normal to you?
Here are some iostat and vmstat metrics:
==== iostat ====
avg-cpu: %user %nice %system %iowait %steal %idle
88.52 3.99 4.11 0.00 0.00 3.38
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 1.00 0.00 0.04 0 0
sdb 156.50 0.00 55.62 0 1
==== vmstat =====
138 0 0 86781912 438780 101523368 0 0 0 31893 264496 247316
95 4 1 0 0 2015-04-21 01:04:01 UTC
147 0 0 86562400 438780 101607248 0 0 0 90510 456635 245849
91 5 4 0 0 2015-04-21 01:04:03 UTC
143 0 0 86341168 438780 101692224 0 0 0 32392 284495 273656
92 4 4 0 0 2015-04-21 01:04:05 UTC
Thanks.
--
Dikang
Re: Bootstrap performance.
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Apr 20, 2015 at 8:09 PM, Dikang Gu <di...@gmail.com> wrote:
> Why do you say steaming is single threaded? I see a lot of background
> streaming threads running, for example:
>
Imprecise :
"Each stream is a single thread".
As I said, first place to look is throttles... but I would not be surprised
if the overall number of threads available to streaming is a meaningful
bound.
=Rob
Re: Bootstrap performance.
Posted by Dikang Gu <di...@gmail.com>.
Hi Rob,
Why do you say steaming is single threaded? I see a lot of background
streaming threads running, for example:
"STREAM-IN-/10.210.165.49" daemon prio=10 tid=0x00007f81fc001000
nid=0x107075 runnable [0x00007f836b256000]
"STREAM-IN-/10.213.51.57" daemon prio=10 tid=0x00007f81f0002000
nid=0x107073 runnable [0x00007f836b1d4000]
"STREAM-IN-/10.213.51.61" daemon prio=10 tid=0x00007f81e8001000
nid=0x107070 runnable [0x00007f836b110000]
"STREAM-IN-/10.213.51.63" daemon prio=10 tid=0x00007f81dc001800
nid=0x10706f runnable [0x00007f836b0cf000]
Thanks
Dikang.
On Mon, Apr 20, 2015 at 6:48 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu <di...@gmail.com> wrote:
>
>> When I bring in a new node into the cluster, it introduces significant
>> load to the cluster. For the new node, the cpu usage is 100%, but disk
>> write io is only around 50MB/s, while we have 10G network.
>>
>> Does it sound normal to you?
>>
>
> Have you unthrottled both compaction and streaming via JMX/nodetool?
>
> Streaming is single threaded and can (?) be CPU bound, I would not be
> surprised if JIRA contains a ticket on the upper bounds of streaming
> performance in current implementation.
>
> =Rob
>
>
>
>
--
Dikang
Re: Bootstrap performance.
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu <di...@gmail.com> wrote:
> When I bring in a new node into the cluster, it introduces significant
> load to the cluster. For the new node, the cpu usage is 100%, but disk
> write io is only around 50MB/s, while we have 10G network.
>
> Does it sound normal to you?
>
Have you unthrottled both compaction and streaming via JMX/nodetool?
Streaming is single threaded and can (?) be CPU bound, I would not be
surprised if JIRA contains a ticket on the upper bounds of streaming
performance in current implementation.
=Rob