You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Hannu Kröger <hk...@gmail.com> on 2018/07/02 07:48:38 UTC
Re: Problem with dropped mutations
Yes, there are timeouts sometimes but more on the read side. And yes, there are certain data modeling problems which will be soon addressed but we need to keep things steady before we get there.
I guess many write timeouts go unnoticed due to consistency level != ALL.
Network looks to be working fine.
Hannu
> ZAIDI, ASAD A <az...@att.com> kirjoitti 26.6.2018 kello 21.42:
>
> Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark!
>
> You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks.
>
> Cheers/Asad
>
>
> -----Original Message-----
> From: Hannu Kröger [mailto:hkroger@gmail.com]
> Sent: Tuesday, June 26, 2018 9:49 AM
> To: user <us...@cassandra.apache.org>
> Subject: Problem with dropped mutations
>
> Hello,
>
> We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
>
> Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
>
> Cheers,
> Hannu
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Problem with dropped mutations
Posted by Jeff Jirsa <jj...@gmail.com>.
Dropped mutations are load shedding - somethings not happy.
Are you seeing GC pauses?
What heap size and version?
What memtable settings ?
--
Jeff Jirsa
> On Jul 2, 2018, at 12:48 AM, Hannu Kröger <hk...@gmail.com> wrote:
>
> Yes, there are timeouts sometimes but more on the read side. And yes, there are certain data modeling problems which will be soon addressed but we need to keep things steady before we get there.
>
> I guess many write timeouts go unnoticed due to consistency level != ALL.
>
> Network looks to be working fine.
>
> Hannu
>
>> ZAIDI, ASAD A <az...@att.com> kirjoitti 26.6.2018 kello 21.42:
>>
>> Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark!
>>
>> You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks.
>>
>> Cheers/Asad
>>
>>
>> -----Original Message-----
>> From: Hannu Kröger [mailto:hkroger@gmail.com]
>> Sent: Tuesday, June 26, 2018 9:49 AM
>> To: user <us...@cassandra.apache.org>
>> Subject: Problem with dropped mutations
>>
>> Hello,
>>
>> We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
>>
>> Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
>>
>> Cheers,
>> Hannu
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org