You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "R. Verlangen" <ro...@us2.nl> on 2012/01/05 11:57:14 UTC
Writes slower then reads
Hi there,
I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
When I'm running my writes against the cluster with cl = ONE all reads
appear to be faster then the writes.
Average write speed = 1600us/operation
Average read speed = 200us/operation
I'm really wondering why this is the case. Anyone got a clue?
With kind regards,
Robin
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
I turned off 1 large cronjob which caused the CPU not to get used for ~ 60%
once every 10 minutes. Both write and read are fast now. Just think I was
overloading the node.
Weird though that shutting down the node did not improve the speed.
Thank you all for your time!
Robin
2012/1/5 aaron morton <aa...@thelastpickle.com>
> What happens when you turn off the cron jobs ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/01/2012, at 6:57 AM, Philippe wrote:
>
> Unless you are doing huge batches no... don't have any other idea for
> now...
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
>> The write and read load is very minimal the moment. Roughly 10 writes +
>> 10 reads / second. So 20 operations per second. Don't think that overloads
>> my cluster, does it?
>>
>>
>> 2012/1/5 Philippe <wa...@gmail.com>
>>
>>> You may be overloading the cluster though...
>>>
>>> My hypothesis is that your traffic is being spread across your node and
>>> that one slow node is slowing down the fraction of traffic that goes to
>>> that node (when it's acting as coordinator).
>>> So what I would do is reduce the read load a lot to make sure I don't
>>> overload the cluster and measure if I see a 1/RF improvement in response
>>> time which would validate my hypothesis.
>>>
>>>
>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>
>>> It does not appear to affect the response time, certainly not in a
>>>> positive way.
>>>>
>>>>
>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>
>>>>> What if you shutdown the cassandra service on the slow node, does that
>>>>> improve your read performance ?
>>>>> If it does then that sole node is responsible for the slow down
>>>>> because it can't act as a coordinator fast enough.
>>>>>
>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>
>>>>> I'm also reading with CL = ONE
>>>>>>
>>>>>>
>>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>>
>>>>>>> Depending on the CL you're reading at it will yes : if the CL
>>>>>>> requires that the "slow" node create a digest of the data and send it to
>>>>>>> the coordinator then it might explain the poor performance on reads. What
>>>>>>> is your read CL ?
>>>>>>>
>>>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>>>
>>>>>>> As I posted this I noticed that the other node's CPU is running high
>>>>>>>> on some other cronjobs (every couple of minutes to 60% usage). Is the lack
>>>>>>>> of more CPU cycles a problem in this case?
>>>>>>>>
>>>>>>>> Robin
>>>>>>>>
>>>>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>>>>
>>>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over
>>>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in
>>>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>>>>>
>>>>>>>>>> What can you see in vmstat/dstat ?
>>>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>>>>>>
>>>>>>>>>> Hi there,
>>>>>>>>>>>
>>>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's),
>>>>>>>>>>> RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>>>>>>> ram).
>>>>>>>>>>>
>>>>>>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>>>>
>>>>>>>>>>> Average write speed = 1600us/operation
>>>>>>>>>>> Average read speed = 200us/operation
>>>>>>>>>>>
>>>>>>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>>>>>>
>>>>>>>>>>> With kind regards,
>>>>>>>>>>> Robin
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
Re: Writes slower then reads
Posted by aaron morton <aa...@thelastpickle.com>.
What happens when you turn off the cron jobs ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 6/01/2012, at 6:57 AM, Philippe wrote:
> Unless you are doing huge batches no... don't have any other idea for now...
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
> The write and read load is very minimal the moment. Roughly 10 writes + 10 reads / second. So 20 operations per second. Don't think that overloads my cluster, does it?
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
> You may be overloading the cluster though...
>
> My hypothesis is that your traffic is being spread across your node and that one slow node is slowing down the fraction of traffic that goes to that node (when it's acting as coordinator).
> So what I would do is reduce the read load a lot to make sure I don't overload the cluster and measure if I see a 1/RF improvement in response time which would validate my hypothesis.
>
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> It does not appear to affect the response time, certainly not in a positive way.
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
> What if you shutdown the cassandra service on the slow node, does that improve your read performance ?
> If it does then that sole node is responsible for the slow down because it can't act as a coordinator fast enough.
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> I'm also reading with CL = ONE
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
> Depending on the CL you're reading at it will yes : if the CL requires that the "slow" node create a digest of the data and send it to the coordinator then it might explain the poor performance on reads. What is your read CL ?
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> As I posted this I noticed that the other node's CPU is running high on some other cronjobs (every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case?
>
> Robin
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. System int ~ 1300, csw ~ 2500.
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
> What can you see in vmstat/dstat ?
>
> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>
> Hi there,
>
> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
>
> When I'm running my writes against the cluster with cl = ONE all reads appear to be faster then the writes.
>
> Average write speed = 1600us/operation
> Average read speed = 200us/operation
>
> I'm really wondering why this is the case. Anyone got a clue?
>
> With kind regards,
> Robin
>
>
>
>
>
>
>
>
>
Re: Writes slower then reads
Posted by Philippe <wa...@gmail.com>.
Unless you are doing huge batches no... don't have any other idea for now...
2012/1/5 R. Verlangen <ro...@us2.nl>
> The write and read load is very minimal the moment. Roughly 10 writes + 10
> reads / second. So 20 operations per second. Don't think that overloads my
> cluster, does it?
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
>
>> You may be overloading the cluster though...
>>
>> My hypothesis is that your traffic is being spread across your node and
>> that one slow node is slowing down the fraction of traffic that goes to
>> that node (when it's acting as coordinator).
>> So what I would do is reduce the read load a lot to make sure I don't
>> overload the cluster and measure if I see a 1/RF improvement in response
>> time which would validate my hypothesis.
>>
>>
>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>
>> It does not appear to affect the response time, certainly not in a
>>> positive way.
>>>
>>>
>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>
>>>> What if you shutdown the cassandra service on the slow node, does that
>>>> improve your read performance ?
>>>> If it does then that sole node is responsible for the slow down because
>>>> it can't act as a coordinator fast enough.
>>>>
>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>
>>>> I'm also reading with CL = ONE
>>>>>
>>>>>
>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>
>>>>>> Depending on the CL you're reading at it will yes : if the CL
>>>>>> requires that the "slow" node create a digest of the data and send it to
>>>>>> the coordinator then it might explain the poor performance on reads. What
>>>>>> is your read CL ?
>>>>>>
>>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>>
>>>>>> As I posted this I noticed that the other node's CPU is running high
>>>>>>> on some other cronjobs (every couple of minutes to 60% usage). Is the lack
>>>>>>> of more CPU cycles a problem in this case?
>>>>>>>
>>>>>>> Robin
>>>>>>>
>>>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>>>
>>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over
>>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in
>>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>>>>
>>>>>>>>> What can you see in vmstat/dstat ?
>>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>>>>>
>>>>>>>>> Hi there,
>>>>>>>>>>
>>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's),
>>>>>>>>>> RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>>>>>> ram).
>>>>>>>>>>
>>>>>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>>>
>>>>>>>>>> Average write speed = 1600us/operation
>>>>>>>>>> Average read speed = 200us/operation
>>>>>>>>>>
>>>>>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>>>>>
>>>>>>>>>> With kind regards,
>>>>>>>>>> Robin
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
The write and read load is very minimal the moment. Roughly 10 writes + 10
reads / second. So 20 operations per second. Don't think that overloads my
cluster, does it?
2012/1/5 Philippe <wa...@gmail.com>
> You may be overloading the cluster though...
>
> My hypothesis is that your traffic is being spread across your node and
> that one slow node is slowing down the fraction of traffic that goes to
> that node (when it's acting as coordinator).
> So what I would do is reduce the read load a lot to make sure I don't
> overload the cluster and measure if I see a 1/RF improvement in response
> time which would validate my hypothesis.
>
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> It does not appear to affect the response time, certainly not in a
>> positive way.
>>
>>
>> 2012/1/5 Philippe <wa...@gmail.com>
>>
>>> What if you shutdown the cassandra service on the slow node, does that
>>> improve your read performance ?
>>> If it does then that sole node is responsible for the slow down because
>>> it can't act as a coordinator fast enough.
>>>
>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>
>>> I'm also reading with CL = ONE
>>>>
>>>>
>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>
>>>>> Depending on the CL you're reading at it will yes : if the CL requires
>>>>> that the "slow" node create a digest of the data and send it to the
>>>>> coordinator then it might explain the poor performance on reads. What is
>>>>> your read CL ?
>>>>>
>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>
>>>>> As I posted this I noticed that the other node's CPU is running high
>>>>>> on some other cronjobs (every couple of minutes to 60% usage). Is the lack
>>>>>> of more CPU cycles a problem in this case?
>>>>>>
>>>>>> Robin
>>>>>>
>>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>>
>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over
>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in
>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>>
>>>>>>>
>>>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>>>
>>>>>>>> What can you see in vmstat/dstat ?
>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>>>>
>>>>>>>> Hi there,
>>>>>>>>>
>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF
>>>>>>>>> = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>>>>> ram).
>>>>>>>>>
>>>>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>>
>>>>>>>>> Average write speed = 1600us/operation
>>>>>>>>> Average read speed = 200us/operation
>>>>>>>>>
>>>>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>>>>
>>>>>>>>> With kind regards,
>>>>>>>>> Robin
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by Philippe <wa...@gmail.com>.
You may be overloading the cluster though...
My hypothesis is that your traffic is being spread across your node and
that one slow node is slowing down the fraction of traffic that goes to
that node (when it's acting as coordinator).
So what I would do is reduce the read load a lot to make sure I don't
overload the cluster and measure if I see a 1/RF improvement in response
time which would validate my hypothesis.
2012/1/5 R. Verlangen <ro...@us2.nl>
> It does not appear to affect the response time, certainly not in a
> positive way.
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
>
>> What if you shutdown the cassandra service on the slow node, does that
>> improve your read performance ?
>> If it does then that sole node is responsible for the slow down because
>> it can't act as a coordinator fast enough.
>>
>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>
>> I'm also reading with CL = ONE
>>>
>>>
>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>
>>>> Depending on the CL you're reading at it will yes : if the CL requires
>>>> that the "slow" node create a digest of the data and send it to the
>>>> coordinator then it might explain the poor performance on reads. What is
>>>> your read CL ?
>>>>
>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>
>>>> As I posted this I noticed that the other node's CPU is running high on
>>>>> some other cronjobs (every couple of minutes to 60% usage). Is the lack of
>>>>> more CPU cycles a problem in this case?
>>>>>
>>>>> Robin
>>>>>
>>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>>
>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
>>>>>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
>>>>>> Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>
>>>>>>
>>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>>
>>>>>>> What can you see in vmstat/dstat ?
>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>>>
>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF
>>>>>>>> = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>>>> ram).
>>>>>>>>
>>>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>
>>>>>>>> Average write speed = 1600us/operation
>>>>>>>> Average read speed = 200us/operation
>>>>>>>>
>>>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>>>
>>>>>>>> With kind regards,
>>>>>>>> Robin
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
It does not appear to affect the response time, certainly not in a positive
way.
2012/1/5 Philippe <wa...@gmail.com>
> What if you shutdown the cassandra service on the slow node, does that
> improve your read performance ?
> If it does then that sole node is responsible for the slow down because it
> can't act as a coordinator fast enough.
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> I'm also reading with CL = ONE
>>
>>
>> 2012/1/5 Philippe <wa...@gmail.com>
>>
>>> Depending on the CL you're reading at it will yes : if the CL requires
>>> that the "slow" node create a digest of the data and send it to the
>>> coordinator then it might explain the poor performance on reads. What is
>>> your read CL ?
>>>
>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>
>>> As I posted this I noticed that the other node's CPU is running high on
>>>> some other cronjobs (every couple of minutes to 60% usage). Is the lack of
>>>> more CPU cycles a problem in this case?
>>>>
>>>> Robin
>>>>
>>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>>
>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
>>>>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
>>>>> Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>
>>>>>
>>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>>
>>>>>> What can you see in vmstat/dstat ?
>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>>
>>>>>> Hi there,
>>>>>>>
>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF =
>>>>>>> 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>>> ram).
>>>>>>>
>>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>>> reads appear to be faster then the writes.
>>>>>>>
>>>>>>> Average write speed = 1600us/operation
>>>>>>> Average read speed = 200us/operation
>>>>>>>
>>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>>
>>>>>>> With kind regards,
>>>>>>> Robin
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by Philippe <wa...@gmail.com>.
What if you shutdown the cassandra service on the slow node, does that
improve your read performance ?
If it does then that sole node is responsible for the slow down because it
can't act as a coordinator fast enough.
2012/1/5 R. Verlangen <ro...@us2.nl>
> I'm also reading with CL = ONE
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
>
>> Depending on the CL you're reading at it will yes : if the CL requires
>> that the "slow" node create a digest of the data and send it to the
>> coordinator then it might explain the poor performance on reads. What is
>> your read CL ?
>>
>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>
>> As I posted this I noticed that the other node's CPU is running high on
>>> some other cronjobs (every couple of minutes to 60% usage). Is the lack of
>>> more CPU cycles a problem in this case?
>>>
>>> Robin
>>>
>>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>>
>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
>>>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
>>>> Paging 0. System int ~ 1300, csw ~ 2500.
>>>>
>>>>
>>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>>
>>>>> What can you see in vmstat/dstat ?
>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>>
>>>>> Hi there,
>>>>>>
>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF =
>>>>>> 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>>> ram).
>>>>>>
>>>>>> When I'm running my writes against the cluster with cl = ONE all
>>>>>> reads appear to be faster then the writes.
>>>>>>
>>>>>> Average write speed = 1600us/operation
>>>>>> Average read speed = 200us/operation
>>>>>>
>>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>>
>>>>>> With kind regards,
>>>>>> Robin
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
I'm also reading with CL = ONE
2012/1/5 Philippe <wa...@gmail.com>
> Depending on the CL you're reading at it will yes : if the CL requires
> that the "slow" node create a digest of the data and send it to the
> coordinator then it might explain the poor performance on reads. What is
> your read CL ?
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> As I posted this I noticed that the other node's CPU is running high on
>> some other cronjobs (every couple of minutes to 60% usage). Is the lack of
>> more CPU cycles a problem in this case?
>>
>> Robin
>>
>> 2012/1/5 R. Verlangen <ro...@us2.nl>
>>
>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
>>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
>>> Paging 0. System int ~ 1300, csw ~ 2500.
>>>
>>>
>>> 2012/1/5 Philippe <wa...@gmail.com>
>>>
>>>> What can you see in vmstat/dstat ?
>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>>
>>>> Hi there,
>>>>>
>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF =
>>>>> 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk
>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB
>>>>> ram).
>>>>>
>>>>> When I'm running my writes against the cluster with cl = ONE all reads
>>>>> appear to be faster then the writes.
>>>>>
>>>>> Average write speed = 1600us/operation
>>>>> Average read speed = 200us/operation
>>>>>
>>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>>
>>>>> With kind regards,
>>>>> Robin
>>>>>
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by Philippe <wa...@gmail.com>.
Depending on the CL you're reading at it will yes : if the CL requires that
the "slow" node create a digest of the data and send it to the coordinator
then it might explain the poor performance on reads. What is your read CL ?
2012/1/5 R. Verlangen <ro...@us2.nl>
> As I posted this I noticed that the other node's CPU is running high on
> some other cronjobs (every couple of minutes to 60% usage). Is the lack of
> more CPU cycles a problem in this case?
>
> Robin
>
> 2012/1/5 R. Verlangen <ro...@us2.nl>
>
> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
>> Paging 0. System int ~ 1300, csw ~ 2500.
>>
>>
>> 2012/1/5 Philippe <wa...@gmail.com>
>>
>>> What can you see in vmstat/dstat ?
>>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>>
>>> Hi there,
>>>>
>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
>>>> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
>>>> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
>>>>
>>>> When I'm running my writes against the cluster with cl = ONE all reads
>>>> appear to be faster then the writes.
>>>>
>>>> Average write speed = 1600us/operation
>>>> Average read speed = 200us/operation
>>>>
>>>> I'm really wondering why this is the case. Anyone got a clue?
>>>>
>>>> With kind regards,
>>>> Robin
>>>>
>>>
>>
>
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
As I posted this I noticed that the other node's CPU is running high on
some other cronjobs (every couple of minutes to 60% usage). Is the lack of
more CPU cycles a problem in this case?
Robin
2012/1/5 R. Verlangen <ro...@us2.nl>
> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
> Paging 0. System int ~ 1300, csw ~ 2500.
>
>
> 2012/1/5 Philippe <wa...@gmail.com>
>
>> What can you see in vmstat/dstat ?
>> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>>
>> Hi there,
>>>
>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
>>> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
>>> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
>>>
>>> When I'm running my writes against the cluster with cl = ONE all reads
>>> appear to be faster then the writes.
>>>
>>> Average write speed = 1600us/operation
>>> Average read speed = 200us/operation
>>>
>>> I'm really wondering why this is the case. Anyone got a clue?
>>>
>>> With kind regards,
>>> Robin
>>>
>>
>
Re: Writes slower then reads
Posted by "R. Verlangen" <ro...@us2.nl>.
CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
Paging 0. System int ~ 1300, csw ~ 2500.
2012/1/5 Philippe <wa...@gmail.com>
> What can you see in vmstat/dstat ?
> Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
>
> Hi there,
>>
>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
>> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
>> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
>>
>> When I'm running my writes against the cluster with cl = ONE all reads
>> appear to be faster then the writes.
>>
>> Average write speed = 1600us/operation
>> Average read speed = 200us/operation
>>
>> I'm really wondering why this is the case. Anyone got a clue?
>>
>> With kind regards,
>> Robin
>>
>
Re: Writes slower then reads
Posted by Philippe <wa...@gmail.com>.
What can you see in vmstat/dstat ?
Le 5 janv. 2012 11:58, "R. Verlangen" <ro...@us2.nl> a écrit :
> Hi there,
>
> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
>
> When I'm running my writes against the cluster with cl = ONE all reads
> appear to be faster then the writes.
>
> Average write speed = 1600us/operation
> Average read speed = 200us/operation
>
> I'm really wondering why this is the case. Anyone got a clue?
>
> With kind regards,
> Robin
>