You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nikhil Sharma <ni...@gmail.com> on 2016/09/26 06:36:34 UTC
Exceptions whenever compaction happens
Hi,
We are not exactly sure what is causing this problem. But after compaction
happens (after 1 week ttl). We start getting this exception:
WARN [SharedPool-Worker-1] 2016-09-26 04:07:19,849
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
WARN [SharedPool-Worker-2] 2016-09-26 04:07:20,639
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.lang.NullPointerException: null
at
org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9]
... 5 common frames omitted
We have no idea how to solve this. I am losing sleep over this, please help!
Cassandra: 3.0.9 (also happened in 3.0.8)
Java: Oracle jdk "1.8.0_102"
jemalloc enabled
Regards,
Nikhil Sharma
Re: Exceptions whenever compaction happens
Posted by Nikhil Sharma <ni...@gmail.com>.
Hi Ben,
Thanks for your help.
We have created an issue here:
https://issues.apache.org/jira/browse/CASSANDRA-12706
Let's see if we can get some comments there.
Regards,
Nikhil Sharma
On Mon, Sep 26, 2016 at 1:12 PM, Ben Slater <be...@instaclustr.com>
wrote:
> Hi Nikhil,
>
> If you haven’t already done so I would recommend logging Cassandra project
> JIRA for this issue - it looks like a defect to me and occurring during
> compaction will probably be hard to work around. If you can work out what
> table is being compacted when the issue occurs and include the schema of
> the table that might help.
>
> Beyond that the only thing I can think of is running scrub if you haven’t
> already done so.
>
> Cheers
> Ben
>
> On Mon, 26 Sep 2016 at 16:36 Nikhil Sharma <ni...@gmail.com> wrote:
>
>> Hi,
>>
>> We are not exactly sure what is causing this problem. But after
>> compaction happens (after 1 week ttl). We start getting this exception:
>>
>> WARN [SharedPool-Worker-1] 2016-09-26 04:07:19,849
>> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
>> thread Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.NullPointerException: null
>> at org.apache.cassandra.db.Slices$ArrayBackedSlices$
>> ComponentOfSlice.isEQ(Slices.java:748) ~[apache-cassandra-3.0.9.jar:
>> 3.0.9]
>> at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.
>> toCQLString(ClusteringIndexSliceFilter.java:150)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.SinglePartitionReadCommand.
>> appendCQLWhereClause(SinglePartitionReadCommand.java:911)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.transform.BasePartitions.
>> runOnClose(BasePartitions.java:70) ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(
>> ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_102]
>> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>> [apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-3.0.9.jar:3.0.9]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> WARN [SharedPool-Worker-2] 2016-09-26 04:07:20,639
>> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
>> thread Thread[SharedPool-Worker-2,5,main]: {}
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at org.apache.cassandra.service.StorageProxy$
>> DroppableRunnable.run(StorageProxy.java:2470)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_102]
>> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>> [apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-3.0.9.jar:3.0.9]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> Caused by: java.lang.NullPointerException: null
>> at org.apache.cassandra.db.Slices$ArrayBackedSlices$
>> ComponentOfSlice.isEQ(Slices.java:748) ~[apache-cassandra-3.0.9.jar:
>> 3.0.9]
>> at org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.
>> toCQLString(ClusteringIndexSliceFilter.java:150)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.SinglePartitionReadCommand.
>> appendCQLWhereClause(SinglePartitionReadCommand.java:911)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.transform.BasePartitions.
>> runOnClose(BasePartitions.java:70) ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.service.StorageProxy$
>> LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at org.apache.cassandra.service.StorageProxy$
>> DroppableRunnable.run(StorageProxy.java:2466)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> ... 5 common frames omitted
>>
>>
>> We have no idea how to solve this. I am losing sleep over this, please
>> help!
>>
>> Cassandra: 3.0.9 (also happened in 3.0.8)
>> Java: Oracle jdk "1.8.0_102"
>> jemalloc enabled
>>
>>
>> Regards,
>>
>> Nikhil Sharma
>>
> --
> ————————
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>
Re: Exceptions whenever compaction happens
Posted by Ben Slater <be...@instaclustr.com>.
Hi Nikhil,
If you haven’t already done so I would recommend logging Cassandra project
JIRA for this issue - it looks like a defect to me and occurring during
compaction will probably be hard to work around. If you can work out what
table is being compacted when the issue occurs and include the schema of
the table that might help.
Beyond that the only thing I can think of is running scrub if you haven’t
already done so.
Cheers
Ben
On Mon, 26 Sep 2016 at 16:36 Nikhil Sharma <ni...@gmail.com> wrote:
> Hi,
>
> We are not exactly sure what is causing this problem. But after compaction
> happens (after 1 week ttl). We start getting this exception:
>
> WARN [SharedPool-Worker-1] 2016-09-26 04:07:19,849
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.9.jar:3.0.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> WARN [SharedPool-Worker-2] 2016-09-26 04:07:20,639
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-2,5,main]: {}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.9.jar:3.0.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> ... 5 common frames omitted
>
>
> We have no idea how to solve this. I am losing sleep over this, please
> help!
>
> Cassandra: 3.0.9 (also happened in 3.0.8)
> Java: Oracle jdk "1.8.0_102"
> jemalloc enabled
>
>
> Regards,
>
> Nikhil Sharma
>
--
————————
Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798