Regular NullPointerExceptions from `nodetool compactionstats` on 3.7 node

[Paul Pollack <pa...@klaviyo.com>]

Hi all,

We have a cluster running on Cassandra 3.7 (we already know this is
considered a "bad" version and plan to upgrade to 3.11 in the
not-too-distant future) and we have a few Nagios checks that run `nodetool
compactionstats` to check how many pending compactions there currently are,
as well as bytes remaining for compactions to see if they will push us past
our comfortable disk utilization threshold.

The check regularly fails with an exit code of 2, and then shortly after
will run successfully, resulting in a check that flaps.

When I am able to reproduce the issue, the output looks like this:

ubuntu@statistic-timelines-11:~$ nodetool compactionstats
error: null
-- StackTrace --
java.lang.NullPointerException

ubuntu@statistic-timelines-11:~$ echo $?
2

I've seen this issue <https://issues.apache.org/jira/browse/CASSANDRA-13021>
for 3.0.11 that was fixed and seems slightly different since in this case,
something is swallowing the full stack trace.

So given all this I have a few questions:
- Has anyone seen this before and have an idea as to what might cause it?
- Is it possible that I have something misconfigured that's swallowing the
stack trace?
- Should I file an issue in the Cassandra JIRA for this?

Thanks,
Paul

Re: Regular NullPointerExceptions from `nodetool compactionstats` on 3.7 node

[kurt greaves <ku...@instaclustr.com>]

Typically have seen that in the past when the node is overloaded. Is that a
possibility for you? If it works consistently after restarting C* it's
likely the issue.

On 20 April 2018 at 19:27, Paul Pollack <pa...@klaviyo.com> wrote:

> Hi all,
>
> We have a cluster running on Cassandra 3.7 (we already know this is
> considered a "bad" version and plan to upgrade to 3.11 in the
> not-too-distant future) and we have a few Nagios checks that run `nodetool
> compactionstats` to check how many pending compactions there currently are,
> as well as bytes remaining for compactions to see if they will push us past
> our comfortable disk utilization threshold.
>
> The check regularly fails with an exit code of 2, and then shortly after
> will run successfully, resulting in a check that flaps.
>
> When I am able to reproduce the issue, the output looks like this:
>
> ubuntu@statistic-timelines-11:~$ nodetool compactionstats
> error: null
> -- StackTrace --
> java.lang.NullPointerException
>
> ubuntu@statistic-timelines-11:~$ echo $?
> 2
>
> I've seen this issue
> <https://issues.apache.org/jira/browse/CASSANDRA-13021> for 3.0.11 that
> was fixed and seems slightly different since in this case, something is
> swallowing the full stack trace.
>
> So given all this I have a few questions:
> - Has anyone seen this before and have an idea as to what might cause it?
> - Is it possible that I have something misconfigured that's swallowing the
> stack trace?
> - Should I file an issue in the Cassandra JIRA for this?
>
> Thanks,
> Paul
>