You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2016/07/06 09:10:11 UTC
[jira] [Commented] (CASSANDRA-12005) Out of memory error in
MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364025#comment-15364025 ]
Sylvain Lebresne commented on CASSANDRA-12005:
----------------------------------------------
This happens when deserializing a mutation on a replica, and more precisely it seems to OOMs when allocating space for a value of that mutation. So in theory, it's possible that you are sometimes writing too many pretty large values in a short time span and that's too much for the nodes.
That said, I'm not pretending the way C* reacts is terribly helpful (nor that I'm sure that this is what happens). At the very least, maybe we should intercept the error and log the size of the value we tried to allocate, to see if that's crazy or not.
> Out of memory error in MessagingService
> ---------------------------------------
>
> Key: CASSANDRA-12005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP x86_64
> Cassandra ReleaseVersion: 2.2.5
> java version "1.8.0_31"
> Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
> Reporter: Chris Powell
>
> I am periodically loosing nodes due to the below OOM error. The nodes restart perfectly fine. It appears intermittent and randomly affects nodes. There are no other warnings or errors in the log files.
> I am using the {{GCG1}} with the following options:
> {quote}
> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
> JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
> JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
> JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
> JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
> JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
> {quote}
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237 CassandraDaemon.java:185 - Exception in thread Thread[MessagingService-Incoming-/10.184.11.109,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) ~[apache-cassandra-2.2.5.jar:2.2.5]
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) ~[apache-cassandra-2.2.5.jar:2.2.5]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)