You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rohit bhatia <ro...@gmail.com> on 2012/07/05 08:49:23 UTC
Finding bottleneck of a cluster
Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
threads). The replication factor is 2 with 10 column families and we
service Counter incrementing write intensive tasks(CL=ONE).
I am trying to figure out the bottleneck,
1) Is using JDK 1.7 any way detrimental to cassandra?
2) What is the max write operation qps that should be expected. Is the
netflix benchmark also applicable for counter incrmenting tasks?
http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
idle time is around 30%, cassandra is not disk bound(insignificant
read operations and cpu's iowait is around 0.05%) and is not swapping
its memory(around 15 gb RAM is free or inactive). The average gc pause
time for parnew are 100ms occuring every second. So cassandra spends
10% of its time stuck in "Stop the world" collector.
The os load is around 16-20 and the average write latency is 3ms.
tpstats do not show any significant pending tasks.
At this point suddenly, Several nodes start dropping several
"Mutation" messages. There are also lots of pending
MutationStage,replicateOnWriteStage tasks in tpstats.
The number of threads in the java process increase to around 25,000
from the usual 300-400. Almost all the new threads seem to be named
"pool-2-thread-*".
The OS load jumps to around 30-40, the "write request latency" starts
spiking to more than 500ms (even to several tens of seconds sometime).
Even the "Local write latency" increases fourfolds to 200 microseconds
from 50 microseconds. This happens across all the nodes and in around
2-3 minutes.
My guess is that this might be due to the 128 Writer threads not being
able to perform more writes.(though with average local write latency
of 100-150 micro seconds, each thread should be able to serve 10,000
qps and with 128 writer threads, should be able to serve 1,280,000 qps
per node)
Could there be any other reason for this? What else should I monitor
since system.log do not seem to say anything conclusive before
dropping messages.
Thanks
Rohit
Re: Finding bottleneck of a cluster
Posted by rohit bhatia <ro...@gmail.com>.
Also,
Looking at gc log. I see messages like this across different servers
before they start dropping messages
"2012-07-04T10:48:20.336+0000: 96771.117: [GC 96771.118: [ParNew:
1367297K->57371K(1474560K), 0.0617350 secs]
6641571K->5340088K(12419072K), 0.0634460 secs] [Times: user=0.56
sys=0.01, real=0.06 secs]
Total time for which application threads were stopped: 0.0850010 seconds
Total time for which application threads were stopped: 16.7663710 seconds"
The 16 second pause doesnt seem to be caused by the minor/major gc
which are quite fast and are also logged. "Total time for which ..."
messages are caused by PrintGCApplicationStoppedTime paramater which
is supposed to be logged whenever threads reach a safepoint. Is there
any way I can figure out what caused the java threads to pause.
Thanks
Rohit
On Thu, Jul 5, 2012 at 12:19 PM, rohit bhatia <ro...@gmail.com> wrote:
> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
> threads). The replication factor is 2 with 10 column families and we
> service Counter incrementing write intensive tasks(CL=ONE).
>
> I am trying to figure out the bottleneck,
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>
> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%) and is not swapping
> its memory(around 15 gb RAM is free or inactive). The average gc pause
> time for parnew are 100ms occuring every second. So cassandra spends
> 10% of its time stuck in "Stop the world" collector.
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
> At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
> MutationStage,replicateOnWriteStage tasks in tpstats.
> The number of threads in the java process increase to around 25,000
> from the usual 300-400. Almost all the new threads seem to be named
> "pool-2-thread-*".
> The OS load jumps to around 30-40, the "write request latency" starts
> spiking to more than 500ms (even to several tens of seconds sometime).
> Even the "Local write latency" increases fourfolds to 200 microseconds
> from 50 microseconds. This happens across all the nodes and in around
> 2-3 minutes.
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(though with average local write latency
> of 100-150 micro seconds, each thread should be able to serve 10,000
> qps and with 128 writer threads, should be able to serve 1,280,000 qps
> per node)
> Could there be any other reason for this? What else should I monitor
> since system.log do not seem to say anything conclusive before
> dropping messages.
>
>
>
> Thanks
> Rohit
Re: Finding bottleneck of a cluster
Posted by rohit bhatia <ro...@gmail.com>.
On Fri, Jul 6, 2012 at 9:44 AM, rohit bhatia <ro...@gmail.com> wrote:
> On Fri, Jul 6, 2012 at 4:47 AM, aaron morton <aa...@thelastpickle.com> wrote:
>> 12G Heap,
>> 1600Mb Young gen,
>>
>> Is a bit higher than the normal recommendation. 1600MB young gen can cause
>> some extra ParNew pauses.
> Thanks for heads up, i'll try tinkering on this
>
>>
>> 128 Concurrent writer
>> threads
>>
>> Unless you are on SSD this is too many.
>>
> I mean http://www.datastax.com/docs/0.8/configuration/node_configuration#concurrent-writes
> , this is not memtable flush queue writers.
> Suggested value is 8*number of cores(16) = 128 itself.
>>
>> 1) Is using JDK 1.7 any way detrimental to cassandra?
>>
>> as far as I know it's not fully certified, thanks for trying it :)
>>
>> 2) What is the max write operation qps that should be expected. Is the
>> netflix benchmark also applicable for counter incrmenting tasks?
>>
>> Counters use a different write path than normal writes and are a bit slower.
>>
>> To benchmark, get a single node and work out the max throughput. Then
>> multiply by the number of nodes and divide by the RF to get a rough idea.
>>
>> the cpu
>> idle time is around 30%, cassandra is not disk bound(insignificant
>> read operations and cpu's iowait is around 0.05%)
>>
>> Wait until compaction kicks in and handle all your inserts.
>>
>> The os load is around 16-20 and the average write latency is 3ms.
>> tpstats do not show any significant pending tasks.
>>
>> The node is overloaded. What is the write latency for a single thread doing
>> as single increment against a node that has not other traffic ? The latency
>> for a request is the time spent working and the time spent waiting, once you
>> read the max throughput the time spent waiting increases. The SEDA
>> architecture is designed to limit the time spent working.
The write latency I reported is as reported by datastax opscenter for
the total latency of a client's request. This is minimum at .5ms.
In contrast, the "local write request latency" as reported by cfstats
are around 50 micro seconds but jump to 150 microseconds during the
crash.
>>
>> At this point suddenly, Several nodes start dropping several
>> "Mutation" messages. There are also lots of pending
>>
>> The cluster is overwhelmed.
>>
>> Almost all the new threads seem to be named
>> "pool-2-thread-*".
>>
>> These are client connection threads.
>>
>> My guess is that this might be due to the 128 Writer threads not being
>> able to perform more writes.(
>>
>> Yes.
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L214
>>
>> Work out the latency for a single client single node, then start adding
>> replication, nodes and load. When the latency increases you are getting to
>> the max throughput for that config.
>
> Also, as mentioned in my second mail, seeing messages like this "Total
> time for which application threads were stopped: 16.7663710 seconds",
> if something pauses for this long, it might be overwhelmed by the
> hints stored at other nodes. This can further cause the node to wait
> on/drop a lot of client connection threads. I'll look into what is
> causing these non-gc pauses. Thanks for the help.
>
>>
>> Hope that helps
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/07/2012, at 6:49 PM, rohit bhatia wrote:
>>
>> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
>> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
>> threads). The replication factor is 2 with 10 column families and we
>> service Counter incrementing write intensive tasks(CL=ONE).
>>
>> I am trying to figure out the bottleneck,
>>
>> 1) Is using JDK 1.7 any way detrimental to cassandra?
>>
>> 2) What is the max write operation qps that should be expected. Is the
>> netflix benchmark also applicable for counter incrmenting tasks?
>>
>> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>>
>> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
>> idle time is around 30%, cassandra is not disk bound(insignificant
>> read operations and cpu's iowait is around 0.05%) and is not swapping
>> its memory(around 15 gb RAM is free or inactive). The average gc pause
>> time for parnew are 100ms occuring every second. So cassandra spends
>> 10% of its time stuck in "Stop the world" collector.
>> The os load is around 16-20 and the average write latency is 3ms.
>> tpstats do not show any significant pending tasks.
>>
>> At this point suddenly, Several nodes start dropping several
>> "Mutation" messages. There are also lots of pending
>> MutationStage,replicateOnWriteStage tasks in tpstats.
>> The number of threads in the java process increase to around 25,000
>> from the usual 300-400. Almost all the new threads seem to be named
>> "pool-2-thread-*".
>> The OS load jumps to around 30-40, the "write request latency" starts
>> spiking to more than 500ms (even to several tens of seconds sometime).
>> Even the "Local write latency" increases fourfolds to 200 microseconds
>> from 50 microseconds. This happens across all the nodes and in around
>> 2-3 minutes.
>> My guess is that this might be due to the 128 Writer threads not being
>> able to perform more writes.(though with average local write latency
>> of 100-150 micro seconds, each thread should be able to serve 10,000
>> qps and with 128 writer threads, should be able to serve 1,280,000 qps
>> per node)
>> Could there be any other reason for this? What else should I monitor
>> since system.log do not seem to say anything conclusive before
>> dropping messages.
>>
>>
>>
>> Thanks
>> Rohit
>>
>>
Re: Finding bottleneck of a cluster
Posted by rohit bhatia <ro...@gmail.com>.
On Fri, Jul 6, 2012 at 4:47 AM, aaron morton <aa...@thelastpickle.com> wrote:
> 12G Heap,
> 1600Mb Young gen,
>
> Is a bit higher than the normal recommendation. 1600MB young gen can cause
> some extra ParNew pauses.
Thanks for heads up, i'll try tinkering on this
>
> 128 Concurrent writer
> threads
>
> Unless you are on SSD this is too many.
>
I mean http://www.datastax.com/docs/0.8/configuration/node_configuration#concurrent-writes
, this is not memtable flush queue writers.
Suggested value is 8*number of cores(16) = 128 itself.
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> as far as I know it's not fully certified, thanks for trying it :)
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
>
> Counters use a different write path than normal writes and are a bit slower.
>
> To benchmark, get a single node and work out the max throughput. Then
> multiply by the number of nodes and divide by the RF to get a rough idea.
>
> the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%)
>
> Wait until compaction kicks in and handle all your inserts.
>
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
> The node is overloaded. What is the write latency for a single thread doing
> as single increment against a node that has not other traffic ? The latency
> for a request is the time spent working and the time spent waiting, once you
> read the max throughput the time spent waiting increases. The SEDA
> architecture is designed to limit the time spent working.
>
> At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
>
> The cluster is overwhelmed.
>
> Almost all the new threads seem to be named
> "pool-2-thread-*".
>
> These are client connection threads.
>
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(
>
> Yes.
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L214
>
> Work out the latency for a single client single node, then start adding
> replication, nodes and load. When the latency increases you are getting to
> the max throughput for that config.
Also, as mentioned in my second mail, seeing messages like this "Total
time for which application threads were stopped: 16.7663710 seconds",
if something pauses for this long, it might be overwhelmed by the
hints stored at other nodes. This can further cause the node to wait
on/drop a lot of client connection threads. I'll look into what is
causing these non-gc pauses. Thanks for the help.
>
> Hope that helps
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/07/2012, at 6:49 PM, rohit bhatia wrote:
>
> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
> threads). The replication factor is 2 with 10 column families and we
> service Counter incrementing write intensive tasks(CL=ONE).
>
> I am trying to figure out the bottleneck,
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
>
> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>
> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%) and is not swapping
> its memory(around 15 gb RAM is free or inactive). The average gc pause
> time for parnew are 100ms occuring every second. So cassandra spends
> 10% of its time stuck in "Stop the world" collector.
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
> At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
> MutationStage,replicateOnWriteStage tasks in tpstats.
> The number of threads in the java process increase to around 25,000
> from the usual 300-400. Almost all the new threads seem to be named
> "pool-2-thread-*".
> The OS load jumps to around 30-40, the "write request latency" starts
> spiking to more than 500ms (even to several tens of seconds sometime).
> Even the "Local write latency" increases fourfolds to 200 microseconds
> from 50 microseconds. This happens across all the nodes and in around
> 2-3 minutes.
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(though with average local write latency
> of 100-150 micro seconds, each thread should be able to serve 10,000
> qps and with 128 writer threads, should be able to serve 1,280,000 qps
> per node)
> Could there be any other reason for this? What else should I monitor
> since system.log do not seem to say anything conclusive before
> dropping messages.
>
>
>
> Thanks
> Rohit
>
>
Re: Finding bottleneck of a cluster
Posted by aaron morton <aa...@thelastpickle.com>.
> 12G Heap,
> 1600Mb Young gen,
Is a bit higher than the normal recommendation. 1600MB young gen can cause some extra ParNew pauses.
> 128 Concurrent writer
> threads
Unless you are on SSD this is too many.
> 1) Is using JDK 1.7 any way detrimental to cassandra?
as far as I know it's not fully certified, thanks for trying it :)
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
Counters use a different write path than normal writes and are a bit slower.
To benchmark, get a single node and work out the max throughput. Then multiply by the number of nodes and divide by the RF to get a rough idea.
> the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%)
Wait until compaction kicks in and handle all your inserts.
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
The node is overloaded. What is the write latency for a single thread doing as single increment against a node that has not other traffic ? The latency for a request is the time spent working and the time spent waiting, once you read the max throughput the time spent waiting increases. The SEDA architecture is designed to limit the time spent working.
> At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
The cluster is overwhelmed.
> Almost all the new threads seem to be named
> "pool-2-thread-*".
These are client connection threads.
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(
Yes.
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L214
Work out the latency for a single client single node, then start adding replication, nodes and load. When the latency increases you are getting to the max throughput for that config.
Hope that helps
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 5/07/2012, at 6:49 PM, rohit bhatia wrote:
> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
> threads). The replication factor is 2 with 10 column families and we
> service Counter incrementing write intensive tasks(CL=ONE).
>
> I am trying to figure out the bottleneck,
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>
> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%) and is not swapping
> its memory(around 15 gb RAM is free or inactive). The average gc pause
> time for parnew are 100ms occuring every second. So cassandra spends
> 10% of its time stuck in "Stop the world" collector.
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
> At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
> MutationStage,replicateOnWriteStage tasks in tpstats.
> The number of threads in the java process increase to around 25,000
> from the usual 300-400. Almost all the new threads seem to be named
> "pool-2-thread-*".
> The OS load jumps to around 30-40, the "write request latency" starts
> spiking to more than 500ms (even to several tens of seconds sometime).
> Even the "Local write latency" increases fourfolds to 200 microseconds
> from 50 microseconds. This happens across all the nodes and in around
> 2-3 minutes.
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(though with average local write latency
> of 100-150 micro seconds, each thread should be able to serve 10,000
> qps and with 128 writer threads, should be able to serve 1,280,000 qps
> per node)
> Could there be any other reason for this? What else should I monitor
> since system.log do not seem to say anything conclusive before
> dropping messages.
>
>
>
> Thanks
> Rohit