You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Azamat Hackimov <az...@gmail.com> on 2022/04/12 09:15:52 UTC
MUTATION_* messages in log
Hello.
I've got two DCs with 8 hardware servers on each. After migration to
Cassandra 4 I have strange messages in system.log:
INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595
MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last
5000 ms: 0 internal and 241 cross node. Mean internal dropped latency:
0 ms and Mean cross-node dropped latency: 2395 ms
INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last
5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0
ms and Mean cross-node dropped latency: 2395 ms
INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in
last 5000 ms: 0 internal and 102 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 2395 ms
Can someone interpret these messages? I suspect that messages indicate
that the cluster has some replication issues.
Re: MUTATION_* messages in log
Posted by Azamat Hackimov <az...@gmail.com>.
Hello again!
Bowen, your answer was right. After forcing clock synchronization, the
count of MUTATION messages radically dropped down. After sync I made
full repair on affected nodes and now I don't have any MUTATION drops
in logs. Thanks for your help!
вт, 12 апр. 2022 г. в 12:31, Bowen Song <bo...@bso.ng>:
>
> First thing to check, do you have NTP client running on all Cassandra
> servers? Are their clock in sync? If you answer "yes" to both, check the
> server load, does any server have high CPU usage or disk utilization?
> Any swapping activity? If not, check the GC logs, and looking for long
> GC pauses.
>
>
> On 12/04/2022 10:15, Azamat Hackimov wrote:
> > Hello.
> >
> > I've got two DCs with 8 hardware servers on each. After migration to
> > Cassandra 4 I have strange messages in system.log:
> >
> > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595
> > MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last
> > 5000 ms: 0 internal and 241 cross node. Mean internal dropped latency:
> > 0 ms and Mean cross-node dropped latency: 2395 ms
> > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
> > MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last
> > 5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0
> > ms and Mean cross-node dropped latency: 2395 ms
> > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
> > MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in
> > last 5000 ms: 0 internal and 102 cross node. Mean internal dropped
> > latency: 0 ms and Mean cross-node dropped latency: 2395 ms
> >
> > Can someone interpret these messages? I suspect that messages indicate
> > that the cluster has some replication issues.
--
From Siberia with Love!
Re: MUTATION_* messages in log
Posted by Bowen Song <bo...@bso.ng>.
First thing to check, do you have NTP client running on all Cassandra
servers? Are their clock in sync? If you answer "yes" to both, check the
server load, does any server have high CPU usage or disk utilization?
Any swapping activity? If not, check the GC logs, and looking for long
GC pauses.
On 12/04/2022 10:15, Azamat Hackimov wrote:
> Hello.
>
> I've got two DCs with 8 hardware servers on each. After migration to
> Cassandra 4 I have strange messages in system.log:
>
> INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595
> MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last
> 5000 ms: 0 internal and 241 cross node. Mean internal dropped latency:
> 0 ms and Mean cross-node dropped latency: 2395 ms
> INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
> MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last
> 5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0
> ms and Mean cross-node dropped latency: 2395 ms
> INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596
> MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in
> last 5000 ms: 0 internal and 102 cross node. Mean internal dropped
> latency: 0 ms and Mean cross-node dropped latency: 2395 ms
>
> Can someone interpret these messages? I suspect that messages indicate
> that the cluster has some replication issues.