how to reduce latency?

[Yan Chunlu <sp...@gmail.com>]

I have three nodes running cassandra 0.7.4 about two years, as showed below:
10.x.x.x      Up     Normal  138.07 GB       33.33%  0

10.x.x.x     Up     Normal  143.97 GB       33.33%
 56713727820156410577229101238628035242
10.x.x.x      Up     Normal  137.33 GB       33.33%
 113427455640312821154458202477256070484

the commitlog and data directory are separated on different disk(Western
Digital WD RE3 WD1002FBYS 1TB).


as the data size grow the reads and write time are keep increasing, slow
down the website frequently.

Based on the experience that every time using nodetool to maintain the
nodes, could cost a very long time, consume a lot system resource(always
ended nowhere),  and cause my web service very unstable, I really have no
idea what to do, upgrade seems not solving this either, I have a newer
cluster with have the same configuration but version is 1.0.2, also got
increasing latency, and the new system also suffering unstable problem...

just wondering does that means I must add more nodes(which is also a
painful and slow path)?



[image: Inline image 1]

Re: how to reduce latency?

[Safdar Kureishy <sa...@gmail.com>]

Hi Yan,
Did you manage to figure out what was causing the increasing latency on
your cluster? Was the resolution just to add more nodes, or something else?
Thanks,
Safdar
On Jun 13, 2012 2:40 PM, "Yan Chunlu" <sp...@gmail.com> wrote:

> I have three nodes running cassandra 0.7.4 about two years, as showed
> below:
> 10.x.x.x      Up     Normal  138.07 GB       33.33%  0
>
> 10.x.x.x     Up     Normal  143.97 GB       33.33%
>  56713727820156410577229101238628035242
> 10.x.x.x      Up     Normal  137.33 GB       33.33%
>  113427455640312821154458202477256070484
>
> the commitlog and data directory are separated on different disk(Western
> Digital WD RE3 WD1002FBYS 1TB).
>
>
> as the data size grow the reads and write time are keep increasing, slow
> down the website frequently.
>
> Based on the experience that every time using nodetool to maintain the
> nodes, could cost a very long time, consume a lot system resource(always
> ended nowhere),  and cause my web service very unstable, I really have no
> idea what to do, upgrade seems not solving this either, I have a newer
> cluster with have the same configuration but version is 1.0.2, also got
> increasing latency, and the new system also suffering unstable problem...
>
> just wondering does that means I must add more nodes(which is also a
> painful and slow path)?
>
>
>
> [image: Inline image 1]
>