You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Jeff Jirsa <jj...@gmail.com> on 2019/09/09 19:38:08 UTC

Re: Is this something to be concerned about? Compaction Error reported in the log every minute

It's unfortunate you dont have a better stack trace to know what's actually
going on here. 3.7 is pretty old, I'd be inclined to upgrade to the latest
3.11 branch to hope that you either get a better stack or an outright fix,
but that stack doesn't ring any bells for me.


On Mon, Sep 9, 2019 at 10:20 AM Leena Ghatpande <lg...@hotmail.com>
wrote:

> We are on Cassandra 3.7 and have a 8 node cluster , 2DC, with 4 nodes in
> each DC. RF=3
>
> The below Compaction Error message is being logged to the system.log
> exactly every Minute.
>
> ERROR [CompactionExecutor:5751] 2019-06-09 03:24:50,585
> CassandraDaemon.java:217 - Exception in thread
> Thread[CompactionExecutor:5751,1,main]
> java.lang.NullPointerException: null
>
> ERROR [CompactionExecutor:5751] 2019-06-09 03:25:50,592
> CassandraDaemon.java:217 - Exception in thread
> Thread[CompactionExecutor:5751,1,main]
> java.lang.NullPointerException: null
>
> ERROR [CompactionExecutor:5753] 2019-06-09 03:26:50,709
> CassandraDaemon.java:217 - Exception in thread
> Thread[CompactionExecutor:5753,1,main]
> java.lang.NullPointerException: null
>
> ERROR [CompactionExecutor:5753] 2019-06-09 03:27:50,734
> CassandraDaemon.java:217 - Exception in thread
> Thread[CompactionExecutor:5753,1,main]
>
> We have been seeing this error for some time now. But this error is
> showing only on 5 out of the 8 nodes.
>
> It does not provide any more details of the error even if we enable the
> Debug Mode. I am not sure what table that this error is related to either.
> We run repairs with -pr option every alternate day and they have been
> running successfully.
>
> nodetool compactionstats shows 0 pending tasks.
> pending tasks: 0
>
> We are trying to figure out what this error indicates and if its a concern
> that we need to address.
>
> Any other suggestions also will be greatly appreciated.
>
>
>

Re: Is this something to be concerned about? Compaction Error reported in the log every minute

Posted by Leena Ghatpande <lg...@hotmail.com>.

THank you for the response. We have an upgrade planned for beginning of next year.

Just saw this one of error in the error logs. Not sure if this helps.

ERROR [CompactionExecutor:6] 2019-09-10 11:26:20,356 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:6,1,main]
java.lang.NullPointerException: null
        at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.7.jar:3.7]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_162]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]

________________________________
From: Jeff Jirsa <jj...@gmail.com>
Sent: Monday, September 9, 2019 3:38 PM
To: cassandra <us...@cassandra.apache.org>
Subject: Re: Is this something to be concerned about? Compaction Error reported in the log every minute

It's unfortunate you dont have a better stack trace to know what's actually going on here. 3.7 is pretty old, I'd be inclined to upgrade to the latest 3.11 branch to hope that you either get a better stack or an outright fix, but that stack doesn't ring any bells for me.


On Mon, Sep 9, 2019 at 10:20 AM Leena Ghatpande <lg...@hotmail.com>> wrote:
We are on Cassandra 3.7 and have a 8 node cluster , 2DC, with 4 nodes in each DC. RF=3

The below Compaction Error message is being logged to the system.log exactly every Minute.

ERROR [CompactionExecutor:5751] 2019-06-09 03:24:50,585 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:5751,1,main]
java.lang.NullPointerException: null

ERROR [CompactionExecutor:5751] 2019-06-09 03:25:50,592 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:5751,1,main]
java.lang.NullPointerException: null

ERROR [CompactionExecutor:5753] 2019-06-09 03:26:50,709 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:5753,1,main]
java.lang.NullPointerException: null

ERROR [CompactionExecutor:5753] 2019-06-09 03:27:50,734 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:5753,1,main]

We have been seeing this error for some time now. But this error is showing only on 5 out of the 8 nodes.

It does not provide any more details of the error even if we enable the Debug Mode. I am not sure what table that this error is related to either.
We run repairs with -pr option every alternate day and they have been running successfully.

nodetool compactionstats shows 0 pending tasks.
pending tasks: 0

We are trying to figure out what this error indicates and if its a concern that we need to address.

Any other suggestions also will be greatly appreciated.