Throttle Heavy Read / Write Loads

[Anuj Wadehra <an...@yahoo.co.in>]

We are using Cassandra 2.0.14 with Hector as client ( will be gradually moving to CQL Driver ). 


Often we see that heavy read and write loads lead to Cassandra timeouts and unpredictable results due to gc pauses and request timeouts. We need to know the best way to throttle read and write load on Cassandra such that even if heavy operations are slower they complete gracefully. This will also shield us against misbehaving clients.


I was thinking of limiting rpc connections via rpc_max_threads property and implementing connection pool at client side. 


I would appreciate if you could please share your suggestions on the above mentioned approach or share any alternatives to the approach.


Thanks

Anuj Wadehra

Re: Throttle Heavy Read / Write Loads

[Graham Sanderson <gr...@vast.com>]

Are you doing large batch inserts via thrift - you need to be careful there

Sent from my iPhone

> On Jun 4, 2015, at 11:37 PM, Anishek Agarwal <an...@gmail.com> wrote:
> 
> may be just increase the read and write timeouts at cassandra currently at 5 sec i think. i think the datastax java client driver provides ability to say how many max requests per connection are to be sent, you can try and lower that to limit excessive requests along with limiting the number of connections a client can do. 
> 
> just out of curiosity how long are GC pauses for you both ParNew and CMS and at what intervals are you seeing the GC happening. I just recently spent time to tune it and would be good to know if its working well.
> 
> thanks
> anishek
> 
>> On Fri, Jun 5, 2015 at 12:03 AM, Anuj Wadehra <an...@yahoo.co.in> wrote:
>> 
>> We are using Cassandra 2.0.14 with Hector as client ( will be gradually moving to CQL Driver ). 
>> 
>> Often we see that heavy read and write loads lead to Cassandra timeouts and unpredictable results due to gc pauses and request timeouts. We need to know the best way to throttle read and write load on Cassandra such that even if heavy operations are slower they complete gracefully. This will also shield us against misbehaving clients.
>> 
>> I was thinking of limiting rpc connections via rpc_max_threads property and implementing connection pool at client side. 
>> 
>> I would appreciate if you could please share your suggestions on the above mentioned approach or share any alternatives to the approach.
>> 
>> Thanks
>> Anuj Wadehra
> 

Re: Throttle Heavy Read / Write Loads

[Anishek Agarwal <an...@gmail.com>]

may be just increase the read and write timeouts at cassandra currently at
5 sec i think. i think the datastax java client driver provides ability to
say how many max requests per connection are to be sent, you can try and
lower that to limit excessive requests along with limiting the number of
connections a client can do.

just out of curiosity how long are GC pauses for you both ParNew and CMS
and at what intervals are you seeing the GC happening. I just recently
spent time to tune it and would be good to know if its working well.

thanks
anishek

On Fri, Jun 5, 2015 at 12:03 AM, Anuj Wadehra <an...@yahoo.co.in>
wrote:

> We are using Cassandra 2.0.14 with Hector as client ( will be gradually
> moving to CQL Driver ).
>
> Often we see that heavy read and write loads lead to Cassandra timeouts
> and unpredictable results due to gc pauses and request timeouts. We need to
> know the best way to throttle read and write load on Cassandra such that
> even if heavy operations are slower they complete gracefully. This will
> also shield us against misbehaving clients.
>
> I was thinking of limiting rpc connections via rpc_max_threads property
> and implementing connection pool at client side.
>
> I would appreciate if you could please share your suggestions on the above
> mentioned approach or share any alternatives to the approach.
>
> Thanks
> Anuj Wadehra
>
>