Temporarily slow nodes on Cassandra

[Michael Theroux <mt...@yahoo.com>]

Hello,

We are experiencing an issue where nodes a temporarily slow due to I/O contention anywhere from 10 minutes to 2 hours.  I don't believe this slowdown is Cassandra related, but factors outside of Cassandra.  We run Cassandra 1.1.9.  We run a 12 node cluster, with a replication factor of 3, and all queries use LOCAL_QUORUM consistency.

Our problem is (other than the contention issue, which we are working on), when this one node slows down, the whole system performance appears to slow down.  Is there a way in Cassandra to accommodate or mitigate slower nodes?  Shutting down the node in question during the period of contention does "resolve" the performance problem, but is there anything in cassandra that can assist this situation while we resolve the hardware problem?

Thanks,
-Mike

Re: Temporarily slow nodes on Cassandra

[Mohit Anchlia <mo...@gmail.com>]

In general with LOCAL_QUORUM you should not see such an issue when one node
is slow. However, it could be because Client's are still sending requests
to that node. Depending on what client library you are using , you could
try to take that node out of your connection pool. Not knowing exact issue
you are facing this is just a hunch at this point.

On Mon, Sep 2, 2013 at 12:33 PM, Michael Theroux <mt...@yahoo.com>wrote:

> Hello,
>
> We are experiencing an issue where nodes a temporarily slow due to I/O
> contention anywhere from 10 minutes to 2 hours.  I don't believe this
> slowdown is Cassandra related, but factors outside of Cassandra.  We run
> Cassandra 1.1.9.  We run a 12 node cluster, with a replication factor of 3,
> and all queries use LOCAL_QUORUM consistency.
>
> Our problem is (other than the contention issue, which we are working on),
> when this one node slows down, the whole system performance appears to slow
> down.  Is there a way in Cassandra to accommodate or mitigate slower nodes?
>  Shutting down the node in question during the period of contention does
> "resolve" the performance problem, but is there anything in cassandra that
> can assist this situation while we resolve the hardware problem?
>
> Thanks,
> -Mike