You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Hannu Kröger <hk...@gmail.com> on 2018/06/26 14:48:44 UTC
Problem with dropped mutations
Hello,
We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
Cheers,
Hannu
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Problem with dropped mutations
Posted by Joshua Galbraith <jg...@newrelic.com.INVALID>.
Hannu,
Dropped mutations are often a sign of load-shedding due to an overloaded
node or cluster. Are you seeing resource saturation like high CPU usage
(because the write path is usually CPU-bound) on any of the nodes in your
cluster?
Some potential contributing factors that might be causing you to drop
mutations are long garbage collection (GC) pauses or large partitions. Do
the drops coincide with an increase in requests, a code change, or
compaction activity?
On Tue, Jun 26, 2018 at 7:48 AM, Hannu Kröger <hk...@gmail.com> wrote:
> Hello,
>
> We have a cluster with somewhat heavy load and we are seeing dropped
> mutations (variable amount and not all nodes have those).
>
> Are there some clear trigger which cause those? What would be the best
> pragmatic approach to start debugging those? We have already added more
> memory which seemed to help somewhat but not completely.
>
> Cheers,
> Hannu
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
--
*Joshua Galbraith *| Senior Software Engineer | New Relic
Re: Problem with dropped mutations
Posted by Jeff Jirsa <jj...@gmail.com>.
Dropped mutations are load shedding - somethings not happy.
Are you seeing GC pauses?
What heap size and version?
What memtable settings ?
--
Jeff Jirsa
> On Jul 2, 2018, at 12:48 AM, Hannu Kröger <hk...@gmail.com> wrote:
>
> Yes, there are timeouts sometimes but more on the read side. And yes, there are certain data modeling problems which will be soon addressed but we need to keep things steady before we get there.
>
> I guess many write timeouts go unnoticed due to consistency level != ALL.
>
> Network looks to be working fine.
>
> Hannu
>
>> ZAIDI, ASAD A <az...@att.com> kirjoitti 26.6.2018 kello 21.42:
>>
>> Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark!
>>
>> You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks.
>>
>> Cheers/Asad
>>
>>
>> -----Original Message-----
>> From: Hannu Kröger [mailto:hkroger@gmail.com]
>> Sent: Tuesday, June 26, 2018 9:49 AM
>> To: user <us...@cassandra.apache.org>
>> Subject: Problem with dropped mutations
>>
>> Hello,
>>
>> We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
>>
>> Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
>>
>> Cheers,
>> Hannu
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Problem with dropped mutations
Posted by Hannu Kröger <hk...@gmail.com>.
Yes, there are timeouts sometimes but more on the read side. And yes, there are certain data modeling problems which will be soon addressed but we need to keep things steady before we get there.
I guess many write timeouts go unnoticed due to consistency level != ALL.
Network looks to be working fine.
Hannu
> ZAIDI, ASAD A <az...@att.com> kirjoitti 26.6.2018 kello 21.42:
>
> Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark!
>
> You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks.
>
> Cheers/Asad
>
>
> -----Original Message-----
> From: Hannu Kröger [mailto:hkroger@gmail.com]
> Sent: Tuesday, June 26, 2018 9:49 AM
> To: user <us...@cassandra.apache.org>
> Subject: Problem with dropped mutations
>
> Hello,
>
> We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
>
> Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
>
> Cheers,
> Hannu
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Problem with dropped mutations
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark!
You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks.
Cheers/Asad
-----Original Message-----
From: Hannu Kröger [mailto:hkroger@gmail.com]
Sent: Tuesday, June 26, 2018 9:49 AM
To: user <us...@cassandra.apache.org>
Subject: Problem with dropped mutations
Hello,
We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those).
Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely.
Cheers,
Hannu
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org