You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Russell Alexander Spitzer (JIRA)" <ji...@apache.org> on 2013/12/16 19:39:06 UTC
[jira] [Created] (CASSANDRA-6493) Exceptions when a second
Datacenter is Added
Russell Alexander Spitzer created CASSANDRA-6493:
----------------------------------------------------
Summary: Exceptions when a second Datacenter is Added
Key: CASSANDRA-6493
URL: https://issues.apache.org/jira/browse/CASSANDRA-6493
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu, EC2 M1.large
Reporter: Russell Alexander Spitzer
On adding a second datacenter several exceptions were raised.
Test outline:
Start 25 Node DC1
Keyspace Setup Replication 3
Begin insert against DC1 Using Stress
While the inserts are occuring
Start up 25 Node DC2
Alter Keyspace to include Replication in 2nd DC
Run rebuild on DC2
Wait for stress to finish
Run repair on Cluster
... Some other operations
At the point when the second datacenter is added several warnings go off because nodetool status is not functioning, and a few moments later the start operation reports a failure because a node has not successfully turned on.
The first start attempt yielded the following exception on a node in the second DC.
{code}
CassandraDaemon.java (line 464) Exception encountered during startup
java.lang.AssertionError: -7560216458456714666 not found in -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ..., 9218575851928340117, 9219681798686280387
at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
{code}
The test automatically tries to restart nodes if they fail during startup, The second attempt for this node succeeded but a nodetool still failed and a different node in the second DC logged the following and failed to start up.
{code}
ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception encountered during startup
java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
at org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)