You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Gary Ogden (JIRA)" <ji...@apache.org> on 2015/05/19 15:52:00 UTC
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3
with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550468#comment-14550468 ]
Gary Ogden commented on CASSANDRA-8917:
---------------------------------------
I'm actually able to reproduce this same error under a different scenario in a test cluster using devcenter.
One of our nodes got into a funny state but is still actually running. Here's the log output:
ERROR 13:29:32 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:69) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:56) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1832) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.sstable.SSTableScanner.getScanner(SSTableScanner.java:57) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1602) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:67) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1988) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2105) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:132) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2
.1.3.jar:2.1.3]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$TraceSessionFutureTask.run(AbstractTracingAwareExecutorService.java:136) [apache-
cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/
cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/
cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
When this node is up but in this state and I try to execute a query in devcenter, I get this same error in the dev center popup, but not every time I run a query.
> Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-8917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8917
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9
> Reporter: Gary Ogden
> Fix For: 2.1.x
>
> Attachments: Screen Shot 2015-05-19 at 10.50.23 AM.png, b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log
>
>
> We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9.
> we have the CL set to quorum for all reads and writes.
> When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes:
> com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
> How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)