You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by laxmikanth sadula <la...@gmail.com> on 2016/09/17 13:16:03 UTC
Throughout of hints delivery
Hi,
Is there any way to monitor hints delivery throughout/performance/issue
delivering hints?
We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
connectivity issues/some other issues hints shoot upto GBs/node.
So I would like to monitor hints throughput/pin point the reason for hints
growth on nodes.
Kindly let us know if any one of you have such thing to monitor hints.
Thanks
Laxmikanth
--
Regards,
Laxmikanth
99621 38051
Re: Throughout of hints delivery
Posted by daemeon reiydelle <da...@gmail.com>.
timeouts indicate network or equivalent throughput delays, from the
physical box's network card out and to the other dc's card. If you are
using VM's add that layer. Your network team needs to be looking for ANY
timeouts, retries, packets delivered in retry window > 0, etc. ANY value
other than zero forever is your problem
*.......*
*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Sat, Sep 17, 2016 at 12:02 PM, laxmikanth sadula <laxmikanth524@gmail.com
> wrote:
> Hi Matija,
>
> All nodes are UP & running and even GC patterns are all well. But I see
> lot of "Timed out replaying hints" in HintedHandOff Manger, I suspect this
> might be the reason why GBs of hints getting piled up instead of proper
> delivery.
> So this clearly indicates some network related issues , so just wanted to
> know the way to monitor hints delivery throughput and also tcp
> throughput(packets sent, received ,dropped etc., on an interface).
>
> If anyone monitoring such stats, please let me know.
>
> On Sat, Sep 17, 2016 at 11:26 PM, Matija Gobec <ma...@gmail.com>
> wrote:
>
>> Hi,
>>
>> You should first figure out why you have so many hints and then think
>> about throughput of hints delivery.
>> Hints are generated for dead nodes and in a healthy cluster are not
>> present.
>> Are all your nodes alive and running? What is the issue of inter DC
>> connectivity?
>>
>> Matija
>>
>> --
>>
>> *Matija Gobec*
>> *Co-Founder & Senior Consultant*
>> www.smartcat.io
>>
>>
>>
>> *Data <https://www.smartcat.io/expertise/> --> Knowledge
>> <https://www.smartcat.io/blog/> --> Power <https://www.smartcat.io/news/> *
>>
>> On Sat, Sep 17, 2016 at 3:16 PM, laxmikanth sadula <
>> laxmikanth524@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is there any way to monitor hints delivery throughout/performance/issue
>>> delivering hints?
>>>
>>> We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
>>> connectivity issues/some other issues hints shoot upto GBs/node.
>>>
>>> So I would like to monitor hints throughput/pin point the reason for
>>> hints growth on nodes.
>>>
>>> Kindly let us know if any one of you have such thing to monitor hints.
>>>
>>>
>>> Thanks
>>> Laxmikanth
>>>
>>>
>>> --
>>> Regards,
>>> Laxmikanth
>>> 99621 38051
>>>
>>>
>>>
>>
>
>
> --
> Regards,
> Laxmikanth
> 99621 38051
>
>
Re: Throughout of hints delivery
Posted by laxmikanth sadula <la...@gmail.com>.
Hi Matija,
All nodes are UP & running and even GC patterns are all well. But I see lot
of "Timed out replaying hints" in HintedHandOff Manger, I suspect this
might be the reason why GBs of hints getting piled up instead of proper
delivery.
So this clearly indicates some network related issues , so just wanted to
know the way to monitor hints delivery throughput and also tcp
throughput(packets sent, received ,dropped etc., on an interface).
If anyone monitoring such stats, please let me know.
On Sat, Sep 17, 2016 at 11:26 PM, Matija Gobec <ma...@gmail.com> wrote:
> Hi,
>
> You should first figure out why you have so many hints and then think
> about throughput of hints delivery.
> Hints are generated for dead nodes and in a healthy cluster are not
> present.
> Are all your nodes alive and running? What is the issue of inter DC
> connectivity?
>
> Matija
>
> --
>
> *Matija Gobec*
> *Co-Founder & Senior Consultant*
> www.smartcat.io
>
>
>
> *Data <https://www.smartcat.io/expertise/> --> Knowledge
> <https://www.smartcat.io/blog/> --> Power <https://www.smartcat.io/news/> *
>
> On Sat, Sep 17, 2016 at 3:16 PM, laxmikanth sadula <
> laxmikanth524@gmail.com> wrote:
>
>> Hi,
>>
>> Is there any way to monitor hints delivery throughout/performance/issue
>> delivering hints?
>>
>> We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
>> connectivity issues/some other issues hints shoot upto GBs/node.
>>
>> So I would like to monitor hints throughput/pin point the reason for
>> hints growth on nodes.
>>
>> Kindly let us know if any one of you have such thing to monitor hints.
>>
>>
>> Thanks
>> Laxmikanth
>>
>>
>> --
>> Regards,
>> Laxmikanth
>> 99621 38051
>>
>>
>>
>
--
Regards,
Laxmikanth
99621 38051
Re: Throughout of hints delivery
Posted by Matija Gobec <ma...@gmail.com>.
Hi,
You should first figure out why you have so many hints and then think about
throughput of hints delivery.
Hints are generated for dead nodes and in a healthy cluster are not present.
Are all your nodes alive and running? What is the issue of inter DC
connectivity?
Matija
--
*Matija Gobec*
*Co-Founder & Senior Consultant*
www.smartcat.io
*Data <https://www.smartcat.io/expertise/> --> Knowledge
<https://www.smartcat.io/blog/> --> Power <https://www.smartcat.io/news/> *
On Sat, Sep 17, 2016 at 3:16 PM, laxmikanth sadula <la...@gmail.com>
wrote:
> Hi,
>
> Is there any way to monitor hints delivery throughout/performance/issue
> delivering hints?
>
> We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
> connectivity issues/some other issues hints shoot upto GBs/node.
>
> So I would like to monitor hints throughput/pin point the reason for hints
> growth on nodes.
>
> Kindly let us know if any one of you have such thing to monitor hints.
>
>
> Thanks
> Laxmikanth
>
>
> --
> Regards,
> Laxmikanth
> 99621 38051
>
>
>