[jira] [Comment Edited] (CASSANDRA-9625) GraphiteReporter not reporting

["Alexey Kovyrin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695532#comment-14695532 ] 

Alexey Kovyrin edited comment on CASSANDRA-9625 at 8/13/15 4:40 PM:
--------------------------------------------------------------------

In our case the issues have started when the cluster went over some large number of segments in it and we've started seeing a lot of compaction activity all the time. Then our nagios checks using jolokia started timing out because the pages those checks were pulling from cassandra nodes were huge (full of compaction history items) and around the same time our graphite metrics stopped being reported. A node restart helps with the issue for a few hours and then the server stops sending anything again.


was (Author: kovyrin):
In our case the issues have started when the cluster went over some large number of segments in it and we've started seeing a lot of merging all the time. Then our nagios checks using jolokia started timing out because the pages those checks were pulling from cassandra nodes were huge (full of merge history items) and around the same time our graphite metrics stopped being reported. A node restart helps with the issue for a few hours and then the server stops sending anything again.

> GraphiteReporter not reporting
> ------------------------------
>
>                 Key: CASSANDRA-9625
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>            Reporter: Eric Evans
>            Assignee: T Jake Luciani
>         Attachments: metrics.yaml, thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops working.  The usual startup is logged, and one batch of samples is sent, but the reporting interval comes and goes, and no other samples are ever sent.  The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not on a 3 node (otherwise identical) staging cluster (maybe it takes a certain level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)