You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Thomas De Keulenaer (Jira)" <ji...@apache.org> on 2020/08/19 13:39:00 UTC
[jira] [Updated] (CASSANDRA-16060) Cassandra crashes with
OutOfMemory Exception
[ https://issues.apache.org/jira/browse/CASSANDRA-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas De Keulenaer updated CASSANDRA-16060:
--------------------------------------------
Description:
Out Cassandra instance has run perfectly for almost 5 years, but aout a month ago, the performance has dropped significantly. Reads are incredibly slow or time out, making the cluster almost useless. The load on the nodes has skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with OutOfMemoryError.
I also have [heap dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
Cassandra version: 3.11.1
Java version: OpenJDK 1.8.0_201
We have upgraded our nodes to version 3.11.7, but this has not improved performance
{code:java}
ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_201]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown Source) ~[na:na]
at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) ~[apache-cassandra-3.11.1.jar:3.11.1]
{code}
was:
Out Cassandra instance has run perfectly for almost 5 years, but aout a month ago, the performance has dropped significantly. Reads are incredibly slow or time out, making the cluster almost useless. The load on the nodes has skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with OutOfMemoryError.
I also have [heap dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
Cassandra version: 3.11.7
Java version: OpenJDK 1.8.0_201
{code:java}
ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_201]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown Source) ~[na:na]
at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) ~[apache-cassandra-3.11.1.jar:3.11.1]
{code}
> Cassandra crashes with OutOfMemory Exception
> --------------------------------------------
>
> Key: CASSANDRA-16060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16060
> Project: Cassandra
> Issue Type: Bug
> Reporter: Thomas De Keulenaer
> Priority: Normal
>
> Out Cassandra instance has run perfectly for almost 5 years, but aout a month ago, the performance has dropped significantly. Reads are incredibly slow or time out, making the cluster almost useless. The load on the nodes has skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with OutOfMemoryError.
> I also have [heap dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
> Cassandra version: 3.11.1
> Java version: OpenJDK 1.8.0_201
> We have upgraded our nodes to version 3.11.7, but this has not improved performance
> {code:java}
> ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 - JVM state determined to be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_201]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
> at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown Source) ~[na:na]
> at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) ~[apache-cassandra-3.11.1.jar:3.11.1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org