You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org> on 2012/01/27 20:28:13 UTC
[jira] [Commented] (CASSANDRA-3417) InvocationTargetException
ConcurrentModificationException at startup
[ https://issues.apache.org/jira/browse/CASSANDRA-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195035#comment-13195035 ]
Peter Schuller commented on CASSANDRA-3417:
-------------------------------------------
How is this a correct fix?
http://google-collections.googlecode.com/svn/trunk/javadoc/com/google/common/collect/Maps.html#synchronizedBiMap(com.google.common.collect.BiMap
In order for concurrent iteration to be safe it's not enough to make individual atomic operations thread-safe. I didn't dig deep but the docs claims HashBiMap is just backed by two hash maps, and also specifically there is mention of needing to synchronize iterations. Which makes sense; imagine trying to implement locking across an iteration and the issues that involves.
Did anyone see the problem actually go away with this patch under circumstances where it was previously reliably re-producible?
> InvocationTargetException ConcurrentModificationException at startup
> --------------------------------------------------------------------
>
> Key: CASSANDRA-3417
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3417
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Fix For: 1.0.2
>
> Attachments: 3417.txt
>
>
> I was starting up the new DataStax AMI where the seed starts first and 34 nodes would latch on together. So far things have been working decently for launching, but right now I just got this during startup.
> {CODE}
> ubuntu@ip-10-40-190-143:~$ sudo cat /var/log/cassandra/output.log
> INFO 09:24:38,453 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26
> INFO 09:24:38,456 Heap size: 1936719872/1937768448
> INFO 09:24:38,457 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.3.jar:/usr/share/cassandra/apache-cassandra-1.0.0.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
> INFO 09:24:39,891 JNA mlockall successful
> INFO 09:24:39,901 Loading settings from file:/etc/cassandra/cassandra.yaml
> INFO 09:24:40,057 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> INFO 09:24:40,069 Global memtable threshold is enabled at 616MB
> INFO 09:24:40,159 EC2Snitch using region: us-east, zone: 1d.
> INFO 09:24:40,475 Creating new commitlog segment /raid0/cassandra/commitlog/CommitLog-1319793880475.log
> INFO 09:24:40,486 Couldn't detect any schema definitions in local storage.
> INFO 09:24:40,486 Found table data in data directories. Consider using the CLI to define your schema.
> INFO 09:24:40,497 No commitlog files found; skipping replay
> INFO 09:24:40,501 Cassandra version: 1.0.0
> INFO 09:24:40,502 Thrift API version: 19.18.0
> INFO 09:24:40,502 Loading persisted ring state
> INFO 09:24:40,506 Starting up server gossip
> INFO 09:24:40,529 Enqueuing flush of Memtable-LocationInfo@1388314661(190/237 serialized/live bytes, 4 ops)
> INFO 09:24:40,530 Writing Memtable-LocationInfo@1388314661(190/237 serialized/live bytes, 4 ops)
> INFO 09:24:40,600 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-1-Data.db (298 bytes)
> INFO 09:24:40,613 Ec2Snitch adding ApplicationState ec2region=us-east ec2zone=1d
> INFO 09:24:40,621 Starting Messaging Service on /10.40.190.143:7000
> INFO 09:24:40,628 Joining: waiting for ring and schema information
> INFO 09:24:43,389 InetAddress /10.194.29.156 is now dead.
> INFO 09:24:43,391 InetAddress /10.85.11.38 is now dead.
> INFO 09:24:43,392 InetAddress /10.34.42.28 is now dead.
> INFO 09:24:43,393 InetAddress /10.77.63.49 is now dead.
> INFO 09:24:43,394 InetAddress /10.194.22.191 is now dead.
> INFO 09:24:43,395 InetAddress /10.34.74.58 is now dead.
> INFO 09:24:43,395 Node /10.34.33.16 is now part of the cluster
> INFO 09:24:43,396 InetAddress /10.34.33.16 is now UP
> INFO 09:24:43,397 Enqueuing flush of Memtable-LocationInfo@1629818866(20/25 serialized/live bytes, 1 ops)
> INFO 09:24:43,398 Writing Memtable-LocationInfo@1629818866(20/25 serialized/live bytes, 1 ops)
> INFO 09:24:43,417 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-2-Data.db (74 bytes)
> INFO 09:24:43,418 InetAddress /10.202.67.43 is now dead.
> INFO 09:24:43,419 InetAddress /10.116.215.81 is now dead.
> INFO 09:24:43,420 InetAddress /10.99.39.242 is now dead.
> INFO 09:24:43,421 InetAddress /10.80.110.28 is now dead.
> INFO 09:24:43,422 InetAddress /10.118.233.198 is now dead.
> INFO 09:24:43,423 InetAddress /10.40.177.173 is now dead.
> INFO 09:24:43,424 InetAddress /10.205.23.34 is now dead.
> INFO 09:24:43,425 InetAddress /10.101.41.8 is now dead.
> INFO 09:24:43,669 InetAddress /10.118.230.219 is now dead.
> INFO 09:24:43,670 InetAddress /10.80.41.192 is now dead.
> INFO 09:24:43,671 InetAddress /10.40.22.224 is now dead.
> INFO 09:24:43,672 InetAddress /10.39.107.114 is now dead.
> INFO 09:24:46,164 InetAddress /10.118.185.68 is now dead.
> INFO 09:24:46,166 InetAddress /10.84.205.93 is now dead.
> INFO 09:24:46,167 InetAddress /10.116.134.183 is now dead.
> INFO 09:24:46,670 InetAddress /10.118.179.67 is now dead.
> INFO 09:24:46,671 InetAddress /10.116.241.250 is now dead.
> INFO 09:24:48,441 InetAddress /10.118.94.62 is now dead.
> INFO 09:24:48,442 InetAddress /10.99.86.251 is now dead.
> INFO 09:24:50,176 InetAddress /10.113.42.21 is now dead.
> INFO 09:24:50,177 InetAddress /10.34.159.72 is now dead.
> INFO 09:24:50,178 InetAddress /10.32.79.134 is now dead.
> INFO 09:24:50,179 InetAddress /10.80.210.38 is now dead.
> INFO 09:24:50,180 InetAddress /10.34.70.73 is now dead.
> INFO 09:24:50,181 InetAddress /10.196.79.240 is now dead.
> INFO 09:25:01,713 InetAddress /10.82.210.172 is now dead.
> INFO 09:25:06,202 InetAddress /10.80.110.28 is now UP
> INFO 09:25:06,908 InetAddress /10.99.39.242 is now UP
> INFO 09:25:07,696 InetAddress /10.118.233.198 is now UP
> INFO 09:25:07,697 InetAddress /10.205.23.34 is now UP
> INFO 09:25:08,704 InetAddress /10.194.22.191 is now UP
> INFO 09:25:08,705 InetAddress /10.40.177.173 is now UP
> INFO 09:25:08,706 InetAddress /10.101.41.8 is now UP
> INFO 09:25:09,489 InetAddress /10.202.67.43 is now UP
> INFO 09:25:09,698 InetAddress /10.77.63.49 is now UP
> INFO 09:25:10,628 Joining: getting bootstrap token
> INFO 09:25:10,631 Enqueuing flush of Memtable-LocationInfo@1733057335(36/45 serialized/live bytes, 1 ops)
> INFO 09:25:10,631 Writing Memtable-LocationInfo@1733057335(36/45 serialized/live bytes, 1 ops)
> INFO 09:25:10,647 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-3-Data.db (87 bytes)
> INFO 09:25:10,649 Joining: sleeping 30000 ms for pending range setup
> INFO 09:25:10,689 InetAddress /10.85.11.38 is now UP
> INFO 09:25:10,708 InetAddress /10.34.74.58 is now UP
> INFO 09:25:10,912 InetAddress /10.194.29.156 is now UP
> INFO 09:25:11,261 Applying migration bb843dd0-0146-11e1-0000-b877c09da5ff Add keyspace: OpsCenter, rep strategy:SimpleStrategy{org.apache.cassandra.config.CFMetaData@55e29b99[cfId=1000,ksName=OpsCenter,cfName=pdps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=300.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@105585dc,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@5ec736e4[cfId=1004,ksName=OpsCenter,cfName=rollups86400,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=2,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@68e4e358,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@b09dc35[cfId=1003,ksName=OpsCenter,cfName=rollups7200,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=2,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@3458213c,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@5ee04fd[cfId=1002,ksName=OpsCenter,cfName=rollups300,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=16,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@4d898115,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@7e79b177[cfId=1005,ksName=OpsCenter,cfName=events,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=OpsCenter raw event storage,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=8,maxCompactionThreshold=12,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@67723c7f,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@540523be[cfId=1006,ksName=OpsCenter,cfName=events_timeline,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=OpsCenter event timelines,rowCacheSize=0.0,keyCacheSize=5.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=8,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=0,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@1d6dba0a,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@ed0f59e[cfId=1007,ksName=OpsCenter,cfName=settings,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=OpsCenter settings,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=8,maxCompactionThreshold=12,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@38ad5fab,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}], org.apache.cassandra.config.CFMetaData@7e63f09e[cfId=1001,ksName=OpsCenter,cfName=rollups60,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=50.0,readRepairChance=0.25,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=43200,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@534a55e5,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={}]}, durable_writes: true
> INFO 09:25:11,273 Enqueuing flush of Memtable-Migrations@1767199109(12925/16156 serialized/live bytes, 1 ops)
> INFO 09:25:11,273 Writing Memtable-Migrations@1767199109(12925/16156 serialized/live bytes, 1 ops)
> INFO 09:25:11,274 Enqueuing flush of Memtable-Schema@1616586953(5820/7275 serialized/live bytes, 3 ops)
> INFO 09:25:11,358 Completed flushing /raid0/cassandra/data/system/Migrations-h-1-Data.db (12989 bytes)
> INFO 09:25:11,358 Writing Memtable-Schema@1616586953(5820/7275 serialized/live bytes, 3 ops)
> INFO 09:25:11,390 Completed flushing /raid0/cassandra/data/system/Schema-h-1-Data.db (5970 bytes)
> INFO 09:25:11,727 InetAddress /10.116.215.81 is now UP
> INFO 09:25:11,744 InetAddress /10.34.42.28 is now UP
> INFO 09:25:11,750 InetAddress /10.40.22.224 is now UP
> INFO 09:25:12,023 InetAddress /10.80.41.192 is now UP
> INFO 09:25:12,712 InetAddress /10.39.107.114 is now UP
> INFO 09:25:12,717 InetAddress /10.118.185.68 is now UP
> INFO 09:25:12,721 InetAddress /10.116.134.183 is now UP
> INFO 09:25:13,322 InetAddress /10.118.230.219 is now UP
> INFO 09:25:13,632 InetAddress /10.84.205.93 is now UP
> INFO 09:25:14,713 InetAddress /10.118.179.67 is now UP
> INFO 09:25:14,717 InetAddress /10.116.241.250 is now UP
> INFO 09:25:17,468 InetAddress /10.34.159.72 is now UP
> INFO 09:25:17,476 InetAddress /10.118.94.62 is now UP
> INFO 09:25:17,480 InetAddress /10.80.210.38 is now UP
> INFO 09:25:17,716 InetAddress /10.32.79.134 is now UP
> INFO 09:25:17,721 InetAddress /10.99.86.251 is now UP
> INFO 09:25:18,717 InetAddress /10.196.79.240 is now UP
> INFO 09:25:18,727 InetAddress /10.34.70.73 is now UP
> INFO 09:25:19,596 InetAddress /10.113.42.21 is now UP
> INFO 09:25:25,750 InetAddress /10.82.210.172 is now UP
> INFO 09:25:37,743 Enqueuing flush of Memtable-LocationInfo@288976631(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:37,744 Writing Memtable-LocationInfo@288976631(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:37,764 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-4-Data.db (89 bytes)
> INFO 09:25:37,773 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-1-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-3-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-4-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-2-Data.db')]
> INFO 09:25:37,776 Enqueuing flush of Memtable-LocationInfo@1950702248(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:37,777 Writing Memtable-LocationInfo@1950702248(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:37,821 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-5-Data.db (89 bytes)
> INFO 09:25:37,869 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-6-Data.db,]. 548 to 443 (~80% of original) bytes for 3 keys at 0.006500MB/s. Time: 65ms.
> INFO 09:25:38,740 Enqueuing flush of Memtable-LocationInfo@92917455(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,740 Writing Memtable-LocationInfo@92917455(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,757 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-8-Data.db (89 bytes)
> INFO 09:25:38,766 Enqueuing flush of Memtable-LocationInfo@1096488363(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,767 Writing Memtable-LocationInfo@1096488363(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,814 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-9-Data.db (89 bytes)
> INFO 09:25:38,816 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-6-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-9-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-8-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-5-Data.db')]
> INFO 09:25:38,823 Enqueuing flush of Memtable-LocationInfo@1734564525(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,823 Writing Memtable-LocationInfo@1734564525(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:38,893 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-10-Data.db (89 bytes)
> INFO 09:25:38,916 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-11-Data.db,]. 710 to 548 (~77% of original) bytes for 3 keys at 0.005226MB/s. Time: 100ms.
> INFO 09:25:39,538 Enqueuing flush of Memtable-LocationInfo@811507066(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,539 Writing Memtable-LocationInfo@811507066(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,555 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-13-Data.db (89 bytes)
> INFO 09:25:39,578 Enqueuing flush of Memtable-LocationInfo@1125690366(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,578 Writing Memtable-LocationInfo@1125690366(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,594 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-14-Data.db (89 bytes)
> INFO 09:25:39,596 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-11-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-10-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-14-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-13-Data.db')]
> INFO 09:25:39,613 Enqueuing flush of Memtable-LocationInfo@1870148830(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,614 Writing Memtable-LocationInfo@1870148830(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,652 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-16-Data.db (89 bytes)
> INFO 09:25:39,692 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-15-Data.db,]. 815 to 653 (~80% of original) bytes for 3 keys at 0.006487MB/s. Time: 96ms.
> INFO 09:25:39,731 Enqueuing flush of Memtable-LocationInfo@1279866611(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,731 Writing Memtable-LocationInfo@1279866611(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:39,747 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-18-Data.db (89 bytes)
> INFO 09:25:40,649 Starting to bootstrap...
> INFO 09:25:40,701 Finished streaming session 304272969286 from /10.205.23.34
> INFO 09:25:40,703 Enqueuing flush of Memtable-LocationInfo@1868577756(53/66 serialized/live bytes, 2 ops)
> INFO 09:25:40,703 Writing Memtable-LocationInfo@1868577756(53/66 serialized/live bytes, 2 ops)
> INFO 09:25:40,721 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-19-Data.db (163 bytes)
> INFO 09:25:40,722 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-19-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-15-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-18-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-16-Data.db')]
> INFO 09:25:40,726 Node /10.40.190.143 state jump to normal
> INFO 09:25:40,734 Enqueuing flush of Memtable-LocationInfo@641287650(35/43 serialized/live bytes, 1 ops)
> INFO 09:25:40,735 Writing Memtable-LocationInfo@641287650(35/43 serialized/live bytes, 1 ops)
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at com.google.common.collect.AbstractBiMap$EntrySet$1.next(AbstractBiMap.java:301)
> at com.google.common.collect.AbstractBiMap$EntrySet$1.next(AbstractBiMap.java:293)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:1127)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:1084)
> at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:920)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:805)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:880)
> at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1027)
> at org.apache.cassandra.service.StorageService.setToken(StorageService.java:226)
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:573)
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:460)
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:381)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:215)
> at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:238)
> ... 5 more
> Cannot load daemon
> Service exit with a return value of 3
> INFO 09:35:35,156 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26
> INFO 09:35:35,159 Heap size: 1936719872/1937768448
> INFO 09:35:35,160 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.3.jar:/usr/share/cassandra/apache-cassandra-1.0.0.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
> INFO 09:35:36,626 JNA mlockall successful
> INFO 09:35:36,636 Loading settings from file:/etc/cassandra/cassandra.yaml
> INFO 09:35:36,757 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> INFO 09:35:36,769 Global memtable threshold is enabled at 616MB
> INFO 09:35:36,811 EC2Snitch using region: us-east, zone: 1d.
> INFO 09:35:37,030 Opening /raid0/cassandra/data/system/Schema-h-1 (5970 bytes)
> INFO 09:35:37,067 Opening /raid0/cassandra/data/system/Migrations-h-1 (12989 bytes)
> INFO 09:35:37,075 Opening /raid0/cassandra/data/system/LocationInfo-h-19 (163 bytes)
> INFO 09:35:37,075 Opening /raid0/cassandra/data/system/LocationInfo-h-18 (89 bytes)
> INFO 09:35:37,083 Opening /raid0/cassandra/data/system/LocationInfo-h-15 (653 bytes)
> INFO 09:35:37,085 Opening /raid0/cassandra/data/system/LocationInfo-h-16 (89 bytes)
> INFO 09:35:37,131 Loading schema version bb843dd0-0146-11e1-0000-b877c09da5ff
> INFO 09:35:37,372 Creating new commitlog segment /raid0/cassandra/commitlog/CommitLog-1319794537372.log
> INFO 09:35:37,384 Replaying /raid0/cassandra/commitlog/CommitLog-1319793880475.log
> INFO 09:35:37,416 Finished reading /raid0/cassandra/commitlog/CommitLog-1319793880475.log
> INFO 09:35:37,422 Enqueuing flush of Memtable-events@1830423861(164/205 serialized/live bytes, 5 ops)
> INFO 09:35:37,423 Writing Memtable-events@1830423861(164/205 serialized/live bytes, 5 ops)
> INFO 09:35:37,424 Enqueuing flush of Memtable-Versions@817138449(83/103 serialized/live bytes, 3 ops)
> INFO 09:35:37,472 Completed flushing /raid0/cassandra/data/OpsCenter/events-h-1-Data.db (230 bytes)
> INFO 09:35:37,479 Writing Memtable-Versions@817138449(83/103 serialized/live bytes, 3 ops)
> INFO 09:35:37,497 Completed flushing /raid0/cassandra/data/system/Versions-h-1-Data.db (247 bytes)
> INFO 09:35:37,497 Log replay complete, 4 replayed mutations
> INFO 09:35:37,509 Cassandra version: 1.0.0
> INFO 09:35:37,510 Thrift API version: 19.18.0
> INFO 09:35:37,510 Loading persisted ring state
> INFO 09:35:37,528 Starting up server gossip
> INFO 09:35:37,530 Enqueuing flush of Memtable-LocationInfo@1655441108(29/36 serialized/live bytes, 1 ops)
> INFO 09:35:37,530 Writing Memtable-LocationInfo@1655441108(29/36 serialized/live bytes, 1 ops)
> INFO 09:35:37,554 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-20-Data.db (80 bytes)
> INFO 09:35:37,555 Ec2Snitch adding ApplicationState ec2region=us-east ec2zone=1d
> INFO 09:35:37,562 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-16-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-18-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-19-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-20-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-15-Data.db')]
> INFO 09:35:37,566 Starting Messaging Service on /10.40.190.143:7000
> INFO 09:35:37,592 Using saved token 19444706681196483626478548996101040654
> INFO 09:35:37,593 Enqueuing flush of Memtable-LocationInfo@995684858(53/66 serialized/live bytes, 2 ops)
> INFO 09:35:37,593 Writing Memtable-LocationInfo@995684858(53/66 serialized/live bytes, 2 ops)
> INFO 09:35:37,616 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-22-Data.db (163 bytes)
> INFO 09:35:37,620 Node /10.40.190.143 state jump to normal
> INFO 09:35:37,639 Bootstrap/Replace/Move completed! Now serving reads.
> INFO 09:35:37,640 Will not load MX4J, mx4j-tools.jar is not in the classpath
> INFO 09:35:37,684 Binding thrift service to /0.0.0.0:9160
> INFO 09:35:37,687 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-21-Data.db,]. 1,074 to 799 (~74% of original) bytes for 4 keys at 0.007620MB/s. Time: 100ms.
> INFO 09:35:37,688 Using TFastFramedTransport with a max frame size of 15728640 bytes.
> INFO 09:35:37,692 Using synchronous/threadpool thrift server on /0.0.0.0 : 9160
> INFO 09:35:37,695 Listening for thrift clients...
> INFO 09:35:37,706 Node /10.118.230.219 is now part of the cluster
> INFO 09:35:37,707 InetAddress /10.118.230.219 is now UP
> INFO 09:35:37,708 Enqueuing flush of Memtable-LocationInfo@2035487037(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,709 Writing Memtable-LocationInfo@2035487037(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,725 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-24-Data.db (89 bytes)
> INFO 09:35:37,726 Node /10.34.42.28 is now part of the cluster
> INFO 09:35:37,727 InetAddress /10.34.42.28 is now UP
> INFO 09:35:37,729 Enqueuing flush of Memtable-LocationInfo@321887181(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,729 Writing Memtable-LocationInfo@321887181(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,747 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-25-Data.db (89 bytes)
> INFO 09:35:37,748 Node /10.77.63.49 has restarted, now UP
> INFO 09:35:37,748 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-24-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-22-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-25-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-21-Data.db')]
> INFO 09:35:37,748 InetAddress /10.77.63.49 is now UP
> INFO 09:35:37,749 Node /10.77.63.49 state jump to normal
> INFO 09:35:37,750 Node /10.34.70.73 is now part of the cluster
> INFO 09:35:37,750 InetAddress /10.34.70.73 is now UP
> INFO 09:35:37,752 Enqueuing flush of Memtable-LocationInfo@1354749546(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,752 Writing Memtable-LocationInfo@1354749546(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,789 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-26-Data.db,]. 1,140 to 877 (~76% of original) bytes for 4 keys at 0.020399MB/s. Time: 41ms.
> INFO 09:35:37,801 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-27-Data.db (89 bytes)
> INFO 09:35:37,801 Node /10.99.86.251 is now part of the cluster
> INFO 09:35:37,802 InetAddress /10.99.86.251 is now UP
> INFO 09:35:37,803 Enqueuing flush of Memtable-LocationInfo@793374785(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,804 Writing Memtable-LocationInfo@793374785(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,825 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-29-Data.db (89 bytes)
> INFO 09:35:37,826 Node /10.202.67.43 has restarted, now UP
> INFO 09:35:37,827 InetAddress /10.202.67.43 is now UP
> INFO 09:35:37,827 Node /10.202.67.43 state jump to normal
> INFO 09:35:37,828 Node /10.116.134.183 is now part of the cluster
> INFO 09:35:37,828 InetAddress /10.116.134.183 is now UP
> INFO 09:35:37,829 Enqueuing flush of Memtable-LocationInfo@1728699027(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,830 Writing Memtable-LocationInfo@1728699027(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,850 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-30-Data.db (89 bytes)
> INFO 09:35:37,852 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-30-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-27-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-26-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-29-Data.db')]
> INFO 09:35:37,853 Node /10.118.94.62 is now part of the cluster
> INFO 09:35:37,853 InetAddress /10.118.94.62 is now UP
> INFO 09:35:37,855 Enqueuing flush of Memtable-LocationInfo@2001229122(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,855 Writing Memtable-LocationInfo@2001229122(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,885 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-31-Data.db (89 bytes)
> INFO 09:35:37,886 Node /10.116.215.81 is now part of the cluster
> INFO 09:35:37,887 InetAddress /10.116.215.81 is now UP
> INFO 09:35:37,888 Enqueuing flush of Memtable-LocationInfo@1748800276(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,888 Writing Memtable-LocationInfo@1748800276(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,909 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-33-Data.db (89 bytes)
> INFO 09:35:37,910 Node /10.80.110.28 has restarted, now UP
> INFO 09:35:37,911 InetAddress /10.80.110.28 is now UP
> INFO 09:35:37,911 Node /10.80.110.28 state jump to normal
> INFO 09:35:37,912 Node /10.80.210.38 is now part of the cluster
> INFO 09:35:37,912 InetAddress /10.80.210.38 is now UP
> INFO 09:35:37,914 Enqueuing flush of Memtable-LocationInfo@1761382005(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,914 Writing Memtable-LocationInfo@1761382005(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,925 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-32-Data.db,]. 1,144 to 982 (~85% of original) bytes for 4 keys at 0.014190MB/s. Time: 66ms.
> INFO 09:35:37,927 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-35-Data.db (89 bytes)
> INFO 09:35:37,928 Node /10.40.177.173 has restarted, now UP
> INFO 09:35:37,929 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-31-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-32-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-33-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-35-Data.db')]
> INFO 09:35:37,929 InetAddress /10.40.177.173 is now UP
> INFO 09:35:37,929 Node /10.40.177.173 state jump to normal
> INFO 09:35:37,930 Node /10.101.41.8 has restarted, now UP
> INFO 09:35:37,931 InetAddress /10.101.41.8 is now UP
> INFO 09:35:37,931 Node /10.101.41.8 state jump to normal
> INFO 09:35:37,931 Node /10.205.23.34 has restarted, now UP
> INFO 09:35:37,932 InetAddress /10.205.23.34 is now UP
> INFO 09:35:37,932 Node /10.205.23.34 state jump to normal
> INFO 09:35:37,933 Node /10.118.185.68 is now part of the cluster
> INFO 09:35:37,933 InetAddress /10.118.185.68 is now UP
> INFO 09:35:37,934 Enqueuing flush of Memtable-LocationInfo@260440278(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,935 Writing Memtable-LocationInfo@260440278(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,970 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-36-Data.db (89 bytes)
> INFO 09:35:37,971 Node /10.116.241.250 is now part of the cluster
> INFO 09:35:37,972 InetAddress /10.116.241.250 is now UP
> INFO 09:35:37,973 Enqueuing flush of Memtable-LocationInfo@768673839(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:37,974 Writing Memtable-LocationInfo@768673839(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,003 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-38-Data.db (89 bytes)
> INFO 09:35:38,004 Node /10.113.42.21 is now part of the cluster
> INFO 09:35:38,005 InetAddress /10.113.42.21 is now UP
> INFO 09:35:38,007 Enqueuing flush of Memtable-LocationInfo@1610335061(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,008 Writing Memtable-LocationInfo@1610335061(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,014 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-37-Data.db,]. 1,249 to 1,087 (~87% of original) bytes for 4 keys at 0.012196MB/s. Time: 85ms.
> INFO 09:35:38,024 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-40-Data.db (89 bytes)
> INFO 09:35:38,024 Node /10.194.29.156 is now part of the cluster
> INFO 09:35:38,025 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-37-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-40-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-36-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-38-Data.db')]
> INFO 09:35:38,025 InetAddress /10.194.29.156 is now UP
> INFO 09:35:38,026 Enqueuing flush of Memtable-LocationInfo@1625488363(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,027 Writing Memtable-LocationInfo@1625488363(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,042 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-41-Data.db (89 bytes)
> INFO 09:35:38,043 Node /10.85.11.38 has restarted, now UP
> INFO 09:35:38,044 InetAddress /10.85.11.38 is now UP
> INFO 09:35:38,044 Node /10.85.11.38 state jump to normal
> INFO 09:35:38,045 Node /10.34.159.72 is now part of the cluster
> INFO 09:35:38,046 InetAddress /10.34.159.72 is now UP
> INFO 09:35:38,047 Enqueuing flush of Memtable-LocationInfo@747881713(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,048 Writing Memtable-LocationInfo@747881713(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,065 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-42-Data.db (89 bytes)
> INFO 09:35:38,067 Node /10.194.22.191 is now part of the cluster
> INFO 09:35:38,067 InetAddress /10.194.22.191 is now UP
> INFO 09:35:38,069 Enqueuing flush of Memtable-LocationInfo@709926392(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,069 Writing Memtable-LocationInfo@709926392(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,092 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-44-Data.db (89 bytes)
> INFO 09:35:38,093 Node /10.34.74.58 is now part of the cluster
> INFO 09:35:38,097 InetAddress /10.34.74.58 is now UP
> INFO 09:35:38,098 Enqueuing flush of Memtable-LocationInfo@1356841826(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,099 Writing Memtable-LocationInfo@1356841826(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,105 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-43-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-41-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-44-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-42-Data.db')]
> INFO 09:35:38,106 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-43-Data.db,]. 1,354 to 1,192 (~88% of original) bytes for 4 keys at 0.014034MB/s. Time: 81ms.
> INFO 09:35:38,144 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-46-Data.db (89 bytes)
> INFO 09:35:38,145 Node /10.40.22.224 is now part of the cluster
> INFO 09:35:38,146 InetAddress /10.40.22.224 is now UP
> INFO 09:35:38,147 Enqueuing flush of Memtable-LocationInfo@422797318(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,148 Writing Memtable-LocationInfo@422797318(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,155 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-47-Data.db,]. 1,459 to 1,297 (~88% of original) bytes for 4 keys at 0.024738MB/s. Time: 50ms.
> INFO 09:35:38,164 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-49-Data.db (89 bytes)
> INFO 09:35:38,165 Node /10.32.79.134 is now part of the cluster
> INFO 09:35:38,166 InetAddress /10.32.79.134 is now UP
> INFO 09:35:38,167 Enqueuing flush of Memtable-LocationInfo@1455093129(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,168 Writing Memtable-LocationInfo@1455093129(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,199 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-50-Data.db (89 bytes)
> INFO 09:35:38,200 Node /10.118.179.67 is now part of the cluster
> INFO 09:35:38,200 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-50-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-47-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-49-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-46-Data.db')]
> INFO 09:35:38,200 InetAddress /10.118.179.67 is now UP
> INFO 09:35:38,202 Enqueuing flush of Memtable-LocationInfo@1105436908(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,202 Writing Memtable-LocationInfo@1105436908(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,248 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-51-Data.db (89 bytes)
> INFO 09:35:38,249 Node /10.84.205.93 is now part of the cluster
> INFO 09:35:38,249 InetAddress /10.84.205.93 is now UP
> INFO 09:35:38,251 Enqueuing flush of Memtable-LocationInfo@1306980591(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,251 Writing Memtable-LocationInfo@1306980591(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,262 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-52-Data.db,]. 1,564 to 1,402 (~89% of original) bytes for 4 keys at 0.021919MB/s. Time: 61ms.
> INFO 09:35:38,294 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-54-Data.db (89 bytes)
> INFO 09:35:38,294 Node /10.34.33.16 has restarted, now UP
> INFO 09:35:38,295 InetAddress /10.34.33.16 is now UP
> INFO 09:35:38,296 Node /10.34.33.16 state jump to normal
> INFO 09:35:38,296 Node /10.39.107.114 is now part of the cluster
> INFO 09:35:38,297 InetAddress /10.39.107.114 is now UP
> INFO 09:35:38,298 Enqueuing flush of Memtable-LocationInfo@1038389338(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,299 Writing Memtable-LocationInfo@1038389338(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,311 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-55-Data.db (89 bytes)
> INFO 09:35:38,312 Node /10.196.79.240 is now part of the cluster
> INFO 09:35:38,312 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-52-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-55-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-54-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-51-Data.db')]
> INFO 09:35:38,313 InetAddress /10.196.79.240 is now UP
> INFO 09:35:38,314 Enqueuing flush of Memtable-LocationInfo@1850278722(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,315 Writing Memtable-LocationInfo@1850278722(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,354 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-56-Data.db (89 bytes)
> INFO 09:35:38,355 Node /10.99.39.242 has restarted, now UP
> INFO 09:35:38,356 InetAddress /10.99.39.242 is now UP
> INFO 09:35:38,356 Node /10.99.39.242 state jump to normal
> INFO 09:35:38,357 Node /10.118.233.198 has restarted, now UP
> INFO 09:35:38,358 InetAddress /10.118.233.198 is now UP
> INFO 09:35:38,358 Node /10.118.233.198 state jump to normal
> INFO 09:35:38,359 Node /10.82.210.172 is now part of the cluster
> INFO 09:35:38,359 InetAddress /10.82.210.172 is now UP
> INFO 09:35:38,364 Enqueuing flush of Memtable-LocationInfo@786665924(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,364 Writing Memtable-LocationInfo@786665924(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,439 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-58-Data.db (89 bytes)
> INFO 09:35:38,440 Node /10.80.41.192 is now part of the cluster
> INFO 09:35:38,440 InetAddress /10.80.41.192 is now UP
> INFO 09:35:38,442 Enqueuing flush of Memtable-LocationInfo@1647844754(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,442 Writing Memtable-LocationInfo@1647844754(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,451 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-57-Data.db,]. 1,669 to 1,515 (~90% of original) bytes for 4 keys at 0.010470MB/s. Time: 138ms.
> INFO 09:35:38,459 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-60-Data.db (89 bytes)
> INFO 09:35:38,459 Node /10.76.243.129 is now part of the cluster
> INFO 09:35:38,460 Compacting [SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-56-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-58-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-57-Data.db'), SSTableReader(path='/raid0/cassandra/data/system/LocationInfo-h-60-Data.db')]
> INFO 09:35:38,460 InetAddress /10.76.243.129 is now UP
> INFO 09:35:38,462 Enqueuing flush of Memtable-LocationInfo@585652261(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,462 Writing Memtable-LocationInfo@585652261(35/43 serialized/live bytes, 1 ops)
> INFO 09:35:38,478 Completed flushing /raid0/cassandra/data/system/LocationInfo-h-61-Data.db (89 bytes)
> INFO 09:35:38,486 Node /10.34.42.28 state jump to normal
> INFO 09:35:38,487 Node /10.34.70.73 state jump to normal
> INFO 09:35:38,488 Node /10.99.86.251 state jump to normal
> INFO 09:35:38,489 Node /10.118.94.62 state jump to normal
> INFO 09:35:38,489 Node /10.80.110.28 state jump to normal
> INFO 09:35:38,490 Node /10.80.210.38 state jump to normal
> INFO 09:35:38,491 Node /10.40.177.173 state jump to normal
> INFO 09:35:38,493 Node /10.101.41.8 state jump to normal
> INFO 09:35:38,493 Node /10.113.42.21 state jump to normal
> INFO 09:35:38,494 Node /10.85.11.38 state jump to normal
> INFO 09:35:38,495 Node /10.34.159.72 state jump to normal
> INFO 09:35:38,496 Node /10.34.74.58 state jump to normal
> INFO 09:35:38,497 Node /10.84.205.93 state jump to normal
> INFO 09:35:38,497 Node /10.118.179.67 state jump to normal
> INFO 09:35:38,498 Node /10.34.33.16 state jump to normal
> INFO 09:35:38,499 Node /10.196.79.240 state jump to normal
> INFO 09:35:38,500 Node /10.118.233.198 state jump to normal
> INFO 09:35:38,501 Node /10.80.41.192 state jump to normal
> INFO 09:35:38,502 Node /10.76.243.129 state jump to normal
> INFO 09:35:38,508 Node /10.118.185.68 state jump to normal
> INFO 09:35:38,524 Node /10.118.230.219 state jump to normal
> INFO 09:35:38,536 Compacted to [/raid0/cassandra/data/system/LocationInfo-h-62-Data.db,]. 1,782 to 1,620 (~90% of original) bytes for 4 keys at 0.020328MB/s. Time: 76ms.
> INFO 09:35:38,537 Node /10.80.110.28 state jump to normal
> INFO 09:35:38,537 Node /10.40.177.173 state jump to normal
> INFO 09:35:38,538 Node /10.101.41.8 state jump to normal
> INFO 09:35:38,539 Node /10.116.241.250 state jump to normal
> INFO 09:35:38,540 Node /10.194.29.156 state jump to normal
> INFO 09:35:38,540 Node /10.34.74.58 state jump to normal
> INFO 09:35:38,541 Node /10.40.22.224 state jump to normal
> INFO 09:35:38,542 Node /10.32.79.134 state jump to normal
> INFO 09:35:38,543 Node /10.39.107.114 state jump to normal
> INFO 09:35:38,543 Node /10.99.39.242 state jump to normal
> INFO 09:35:38,550 Node /10.77.63.49 state jump to normal
> INFO 09:35:38,550 Node /10.34.42.28 state jump to normal
> INFO 09:35:38,551 Node /10.116.134.183 state jump to normal
> INFO 09:35:38,553 Node /10.76.243.129 state jump to normal
> INFO 09:35:38,557 Node /10.202.67.43 state jump to normal
> INFO 09:35:38,558 Node /10.118.94.62 state jump to normal
> INFO 09:35:38,562 Node /10.116.215.81 state jump to normal
> INFO 09:35:38,563 Node /10.80.210.38 state jump to normal
> INFO 09:35:38,564 Node /10.205.23.34 state jump to normal
> INFO 09:35:38,565 Node /10.39.107.114 state jump to normal
> {CODE}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira