You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Subramanya M (JIRA)" <ji...@apache.org> on 2016/08/25 20:04:21 UTC
[jira] [Commented] (CASSANDRA-4981) Error when starting a node with
vnodes while counter-add operations underway
[ https://issues.apache.org/jira/browse/CASSANDRA-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437546#comment-15437546 ]
Subramanya M commented on CASSANDRA-4981:
-----------------------------------------
Its still happening. I have setup 2 nodes on Ubuntu on VM. Had powered off VM when nodes are UP.
Now when I restart the VM and bring up only one node. insert into simple table works fine. But when I bring up the other node and try the insert, it fails with below error in cassandra node 1:
ERROR 18:45:56 Unexpected exception during request; channel = [id: 0xbdb30e19, L:/127.0.0.1:9042 - R:/127.0.0.1:39944]
java.lang.AssertionError: null
at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:946) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:960) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:107) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:3612) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1832) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$RangeIterator.computeNext(StorageProxy.java:1910) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$RangeIterator.computeNext(StorageProxy.java:1880) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1149) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageProxy$RangeMerger.computeNext(StorageProxy.java:1932) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$RangeMerger.computeNext(StorageProxy.java:1917) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2044) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2010) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:735) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:410) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:363) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:237) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:78) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.7.jar:3.7]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.36.Final.jar:4.0.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) [netty-all-4.0.36.Final.jar:4.0.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) [netty-all-4.0.36.Final.jar:4.0.36.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283) [netty-all-4.0.36.Final.jar:4.0.36.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
INFO 19:22:48 Redistributing index summaries
INFO 19:28:17 ConcurrentMarkSweep GC in 221ms. CMS Old Gen: 46231032 -> 23371736; Par Eden Space: 44030248 -> 0; Par Survivor Space: 3730896 -> 0
cqlsh:arc> select * from user;
ServerError: <ErrorMessage code=0000 [Server error] message="java.lang.AssertionError">
Cassandra version 3.7.
When only first node is started:
$ ./nodetool -h 127.0.0.1 -p 7199 status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 236.88 KiB 256 ? e2c88dd6-18f5-447b-a05c-ac4a2112969b rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
After starting 2nd node$ ./nodetool -h 127.0.0.2 -p 7199 status
nodetool: Failed to connect to '127.0.0.2:7199' - ConnectException: 'Connection refused'.
> Error when starting a node with vnodes while counter-add operations underway
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-4981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4981
> Project: Cassandra
> Issue Type: Bug
> Environment: 2-node cluster on ec2, ubuntu, cassandra-1.2.0 commit a32eb9f7d2f2868e8154d178e96e045859e1d855
> Reporter: Tyler Patterson
> Assignee: Ryan McGuire
> Priority: Minor
> Labels: qa-resolved
> Attachments: system.log
>
>
> Start both nodes, start stress on one node like this: "cassandra-stress --replication-factor=2 --operation=COUNTER_ADD"
> While that is running: On the other node, kill cassandra, wait for "nodetool status" to show the node as down, and restart cassandra. I sometimes have to kill and restart cassandra several times to get the problem to happen.
> I get this error several times in the log:
> {code}
> ERROR 15:39:33,198 Exception in thread Thread[MutationStage:16,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:748)
> at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:762)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:95)
> at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2426)
> at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:396)
> at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:755)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)