monitoring read and write problems via log file?

[Robert <ca...@spamgourmet.com>]

I was told by a colleague that read and write problems in Cassandra can be
detected by monitoring a Cassandra log file.

A few questions:
 1) Is this right?  Can read and write problems be detected this way?
 2) If they can be detected, what is/are the right log file(s) to look at?
 3) Is there a text or regular expression pattern I should be looking for in the
log file(s) to detect a problem?
 4) Is there any easy way to constantly monitor log file(s)?

Thanks!

Re: monitoring read and write problems via log file?

[Edward Capriolo <ed...@gmail.com>]

On Wed, Nov 24, 2010 at 3:04 AM, Peter Schuller
<pe...@infidyne.com> wrote:
>> I was told by a colleague that read and write problems in Cassandra can be
>> detected by monitoring a Cassandra log file.
>
> What do you mean by "problem"? If you mean something like a hard I/O
> error or corruption causing an internal error, you should get an
> exception of some kind in the system log (typically
> /var/log/cassandra/output.log or similar, unless otherwise
> configured).
>
> --
> / Peter Schuller
>

At the default log level of info you should look for
DroppedMessageLogger, -- backpressure is causing failures
GCInspector, --garbage collector paused
"optimal bloom filter" -- not sure this is critical but appears at times
"Large row" -- message from compaction about a really large row
"STATE Down" -- message from gossip about node flap
"STATE UP"  -- message from gossip about node flap
"Digest mismatch exception" --Quorum read fixed data (I do not see this much")

I use a log4j syslog appender to send info to our splunk/syslog
station.  I use splunk to count these events based on time buckets.

Re: monitoring read and write problems via log file?

[Peter Schuller <pe...@infidyne.com>]

> I was told by a colleague that read and write problems in Cassandra can be
> detected by monitoring a Cassandra log file.

What do you mean by "problem"? If you mean something like a hard I/O
error or corruption causing an internal error, you should get an
exception of some kind in the system log (typically
/var/log/cassandra/output.log or similar, unless otherwise
configured).

-- 
/ Peter Schuller