You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chakravarthi Manepalli (JIRA)" <ji...@apache.org> on 2019/06/14 12:55:00 UTC
[jira] [Commented] (CASSANDRA-12005) Out of memory error in
MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864029#comment-16864029 ]
Chakravarthi Manepalli commented on CASSANDRA-12005:
----------------------------------------------------
I have faced a similar issue, heap overflows after 15-30 min time from the node restart.
Setup environment - single node running in localhost with 64GB RAM and 8GB HEAP memory
nodetool info:
ID : 35c11cc6-f881-4685-b211-654c4a21bf59
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 4.57 MiB
Generation No : 1560507197
Uptime (seconds) : 250
Heap Memory (MB) : 301.67 / 8112.00
Off Heap Memory (MB) : 0.01
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 128, size 16.5 KiB, capacity 100 MiB, 375 hits, 510 requests, 0.735 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 19, size 1.19 MiB, capacity 480 MiB, 495 misses, 1126 requests, 0.560 recent hit rate, 48.843 microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
The node shows running in service status but unable to take cqlsh or access nodetool.
ERROR [Native-Transport-Requests-1] 2019-06-14 14:41:35,747 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.Clustering.make(Clustering.java:81) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.3.jar:3.11.3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
WARN [epollEventLoopGroup-2-5] 2019-06-14 14:41:35,748 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.Clustering.make(Clustering.java:81) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) ~[apache-cassandra-3.11.3.jar:3.11.3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
> Out of memory error in MessagingService
> ---------------------------------------
>
> Key: CASSANDRA-12005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Streaming and Messaging
> Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP x86_64
> Cassandra ReleaseVersion: 2.2.5
> java version "1.8.0_31"
> Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
> Reporter: Chris Powell
> Priority: Normal
>
> I am periodically loosing nodes due to the below OOM error. The nodes restart perfectly fine. It appears intermittent and randomly affects nodes. There are no other warnings or errors in the log files.
> I am using the {{GCG1}} with the following options:
> {quote}
> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
> JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
> JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
> JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
> JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
> JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
> {quote}
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237 CassandraDaemon.java:185 - Exception in thread Thread[MessagingService-Incoming-/10.184.11.109,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) ~[apache-cassandra-2.2.5.jar:2.2.5]
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) ~[apache-cassandra-2.2.5.jar:2.2.5]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org