Client Read Latency is too high during repair

[Benyi Wang <be...@gmail.com>]

I'm using cassandra java driver to access a small cassandra cluster

* The cluster have 3 nodes in DC1 and 3 nodes in DC2
* The keyspace is originally created in DC1 only with RF=2
* The client had good read latency about 40 ms of 99 percentile under 100
requests/sec (measured at the client side)
* Then keyspace is updated with 2-DC and RF=3 for each DC
* After the repair started (DBA started it, I don't exactly the command),
the client's read latency reached to 2 secs.
* The metric ClientRequest.read.latency.99percentile is still about 4ms
* There were two nodes having 3MB/sec outgoing streaming.

I'm using Cassandra 2.1.8 and the read consistency is LOCAL_ONE.

Can you point me some metrics to see what's the bottleneck?

Thanks

Re: Client Read Latency is too high during repair

[Benyi Wang <be...@gmail.com>]

Never mind. I found the root cause. This has nothing to do with Cassandra
and repair. Some web services called by the client caused the problem.

On Fri, Aug 19, 2016 at 11:53 AM, Benyi Wang <be...@gmail.com> wrote:

> I'm using cassandra java driver to access a small cassandra cluster
>
> * The cluster have 3 nodes in DC1 and 3 nodes in DC2
> * The keyspace is originally created in DC1 only with RF=2
> * The client had good read latency about 40 ms of 99 percentile under 100
> requests/sec (measured at the client side)
> * Then keyspace is updated with 2-DC and RF=3 for each DC
> * After the repair started (DBA started it, I don't exactly the command),
> the client's read latency reached to 2 secs.
> * The metric ClientRequest.read.latency.99percentile is still about 4ms
> * There were two nodes having 3MB/sec outgoing streaming.
>
> I'm using Cassandra 2.1.8 and the read consistency is LOCAL_ONE.
>
> Can you point me some metrics to see what's the bottleneck?
>
> Thanks
>