RE: Log messages "MUTATION messages were dropped" in system.log

[Michail Kotsiouros via user <us...@cassandra.apache.org>]

Hello community,
I came across the following post:
https://lists.apache.org/thread/t94jjc4j8dfd9sg5of9283mqcy3y3bgq
This gives me a good explanation of the log. I will try to check the storage of the problematic datacenters. I will coma back if I have any good findings.

BR
MK

From: Michail Kotsiouros via user <us...@cassandra.apache.org>
Sent: Tuesday, September 27, 2022 17:36
To: user@cassandra.apache.org
Subject: Log messages "MUTATION messages were dropped" in system.log

Hello Cassandra community,
I see the following log lines in the Cassandra nodes of 2 datacenters that participate in a cluster of 6 datacenters.

INFO  [ScheduledTasks:1] 2022-09-12 17:19:44,422 MessagingService.java:1236 - MUTATION messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 9140 ms<https://protect2.fireeye.com/v1/url?k=31323334-501cfaf3-313273af-454445554331-29833d79f9680915&q=1&e=25113adf-a158-4321-a644-3c79d2792963&u=http%3A%2F%2Fmessagingservice.java%3A1236%2F>

From the information I have found online, I see that those logs indicate a resource issue on the Cassandra nodes. I am trying to troubleshoot what could be the bottleneck.
Can you, please, help me interpret the log line.
So far I know that
MUTATION means that it is write operations for already existing keys.
What internal means? Does it indicate that the write operation is originated from a coordinator node residing in the same datacenter?
What cross node means? Does it indicate that the write operation is originated from a coordinator node residing in another datacenter?
What dropped latency means? Is it the time that the message spent in some queue before dropped?

I would like to know that so as to understand the impact observed on client application.

Thank you in advance!!! I find really useful all the information found in different posts!!!

BR
Michail