You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Lohfink (JIRA)" <ji...@apache.org> on 2018/11/07 00:48:00 UTC
[jira] [Commented] (CASSANDRA-14867) Histogram overflows
potentially leading to writes failing
[ https://issues.apache.org/jira/browse/CASSANDRA-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677468#comment-16677468 ]
Chris Lohfink commented on CASSANDRA-14867:
-------------------------------------------
That dropped messages logging runs at on a scheduled executor. It will not interrupt your writes so it is mostly cosmetic. The fact that you had over 2^32 dropped messages is something that you should be concerned about (something to move to mailing list).
The histogram throwing runtime exceptions has been recurring issue and is not really the right approach. In some of the methods this instead returns max value and I think should just be the result.
If these methods are going to throw exceptions they should not be runtime as the uses of it rarely handle the condition.
> Histogram overflows potentially leading to writes failing
> ---------------------------------------------------------
>
> Key: CASSANDRA-14867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14867
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: cassandra 3.11.1 on ubuntu 16.04
> Reporter: David
> Priority: Major
>
> I observed the following in cassandra logs on 1 host of a 6-node cluster:
> ERROR [ScheduledTasks:1] 2018-11-01 17:26:41,277 CassandraDaemon.java:228 - Exception in thread Thread[ScheduledTasks:1,5,main]
> java.lang.IllegalStateException: Unable to compute when histogram overflowed
> at org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getMean(DecayingEstimatedHistogramReservoir.java:472) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessagingService.getDroppedMessagesLogs(MessagingService.java:1263) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessagingService.logDroppedMessages(MessagingService.java:1236) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessagingService.access$200(MessagingService.java:87) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessagingService$4.run(MessagingService.java:507) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_172]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_172]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_172]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_172]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_172]
>
> At the same time, this node was failing all writes issued to it. Restarting cassandra on the node brought the cluster into a good state and we stopped seeing the histogram overflow errors.
> Has this issue been observed before? Could the histogram overflows cause writes to fail?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org