You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saurabh Sethi <Sa...@symantec.com> on 2015/01/29 03:19:49 UTC
Unable to create a keyspace
I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4 driver to create a keyspace followed by creating a column family within that keyspace from my unit test.
But I do not see the keyspace getting created and the code for creating column family fails because it cannot find the keyspace. I see the following in the system.log file:
INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472 MigrationManager.java:229 - Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
INFO [MigrationStage:1] 2015-01-28 17:59:08,476 ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326 - Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360 - Completed flushing /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803, position=10514)
This issue doesn’t happen always. My test runs fine sometimes but once it gets into this state, it remains there for a while and I can constantly reproduce this.
Also, when this issue happens for the first time, I also see the following error message in system.log file:
ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 - Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.8.0_05]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Anyone has any idea what might be going on here?
Thanks,
Saurabh
Re: Unable to create a keyspace
Posted by Saurabh Sethi <Sa...@symantec.com>.
Thanks Carlos for pointing that out. The clock on one of the nodes was not in sync and fixing that solved the issue.
From: Jan <cn...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Jan <cn...@yahoo.com>>
Date: Saturday, January 31, 2015 at 9:59 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Unable to create a keyspace
Saurabh;
a) How exactly are the three nodes hosted.
b) Can you take down node 2 and create the keyspace from node 1
c) Can you take down node 1 and create the keyspace from node2
d) Do the nodes see each other with 'nodetool status'
cheers
Jan/
C* Architect
On Saturday, January 31, 2015 5:40 AM, Carlos Rolo <ro...@pythian.com>> wrote:
Something that can cause weird behavior is the machine clocks not being properly synced. I didn't read the thread in full detail, so disregard this if it is not the case.
--
Re: Unable to create a keyspace
Posted by Jan <cn...@yahoo.com>.
Saurabh;
a) How exactly are the three nodes hosted. b) Can you take down node 2 and create the keyspace from node 1c) Can you take down node 1 and create the keyspace from node2d) Do the nodes see each other with 'nodetool status'
cheersJan/
C* Architect
On Saturday, January 31, 2015 5:40 AM, Carlos Rolo <ro...@pythian.com> wrote:
Something that can cause weird behavior is the machine clocks not being properly synced. I didn't read the thread in full detail, so disregard this if it is not the case.
--
Re: Unable to create a keyspace
Posted by Carlos Rolo <ro...@pythian.com>.
Something that can cause weird behavior is the machine clocks not being
properly synced. I didn't read the thread in full detail, so disregard
this if it is not the case.
--
--
Re: Unable to create a keyspace
Posted by Saurabh Sethi <Sa...@symantec.com>.
I repeat the test after verifying that the keyspace has been dropped.
Also, I do not want any replication as of now that’s why replication factor is 1. But I don’t think that should be the reason for the command not propagating to other nodes.
One more thing that I observed right now is that if I create the keyspace from node 1 and node 2, it doesn’t get created but if I try to create it from node 3, it does get created. Node1 and Node 2 are Vms hosted on the same physical hardware, not sure if that has anything to do.
Thanks,
Saurabh
From: Asit KAUSHIK <as...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Friday, January 30, 2015 at 5:33 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Unable to create a keyspace
Saurabh a vague suggestion when you are dropping can you wait for sometime to let the change propagate to other node.Also I see a replication factor 1 but you have 3 nodes....
On Jan 31, 2015 6:28 AM, "Saurabh Sethi" <Sa...@symantec.com>> wrote:
Thanks Adam. I upgraded to 2.1.2 but still seeing this issue. Let me go in a bit more detail as to what I am seeing – When I create a keyspace for the first time in a 3 node cluster, it works fine but if I drop the keyspace and try to recreate it, I see that the node received the request to create it but it didn’t delegate the request to other nodes because of which the keyspace didn’t get created on other two nodes.
But the request for creating a table does get propagated to other nodes and since they can’t find that keyspace they throw an exception.
I have 2 nodes as seeds, so the seeds property in cassandra.yaml for all 3 nodes looks like – seeds: "<node1ip>,<node3ip>"
Following is that I see in system.log file when I enable DEBUG mode:
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,363 Message.java:437 - Received: STARTUP {CQL_VERSION=3.0.0}, v=3
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,364 Message.java:452 - Responding: READY, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,365 Message.java:437 - Received: QUERY select cluster_name from system.local, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,366 StorageProxy.java:1534 - Estimated result rows per range: 20.057144; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Message.java:452 - Responding: ROWS [cluster_name(system, local), org.apache.cassandra.db.marshal.UTF8Type]
| CASSANDRAEDPDEVCLUSTER
---, v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,386 Message.java:437 - Received: QUERY CREATE KEYSPACE IF NOT EXISTS TestMaxColumnsKeySpace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };, v=3
INFO [SharedPool-Worker-1] 2015-01-30 16:39:37,387 MigrationManager.java:229 - Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@5485ec68}
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,388 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 262698
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,389 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-3-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 394047
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
INFO [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:840 - Enqueuing flush of schema_keyspaces: 496 (0%) on-heap, 0 (0%) off-heap
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:166 - scheduling flush in 3600000 ms
INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,394 Memtable.java:325 - Writing Memtable-schema_keyspaces@922450609(138 serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Statistics.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Statistics.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Filter.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Filter.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-TOC.txt to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-TOC.txt
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Digest.sha1 to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Digest.sha1
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-CompressionInfo.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-CompressionInfo.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Index.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Index.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,412 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Data.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 Memtable.java:364 - Completed flushing /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db (175 bytes) for commitlog position ReplayPosition(segmentId=1422663061374, position=3040)
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 DataTracker.java:440 - adding bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10 to list of files tracked for system.schema_keyspaces
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 CompactionManager.java:179 - Scheduling a background task check for system.schema_keyspaces with SizeTieredCompactionStrategy
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:265 - discard completed log segments for ReplayPosition(segmentId=1422663061374, position=3040), column family b0f22357-4458-3cdb-9631-c43e59ce3676
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,414 CompactionManager.java:218 - Checking system.schema_keyspaces
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:283 - Not safe to delete active commit log segment CommitLogSegment(bin/../data/commitlog/CommitLog-4-1422663061374.log); dirty is peers (37f71aca-7dc2-383b-a706-72528af04d4f), compaction_history (b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca), local (7ad54392-bcdd-35a6-8417-4e047860b377),
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_columnfamilies
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 SizeTieredCompactionStrategy.java:86 - Compaction buckets are [[SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db'), SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-9-Data.db')]]
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_columns
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 SizeTieredCompactionStrategy.java:86 - Compaction buckets are []
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_triggers
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 CompactionManager.java:229 - No tasks available
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,416 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_usertypes
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,416 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,425 MigrationManager.java:418 - Gossiping my schema version b5291c1d-6635-3627-928f-f5a0f0c27ec1
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Message.java:452 - Responding: RESULT schema change CREATED KEYSPACE testmaxcolumnskeyspace, v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,502 Message.java:437 - Received: QUERY SELECT peer, rpc_address, schema_version FROM system.peers, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,502 Message.java:437 - Received: QUERY SELECT schema_version FROM system.local WHERE key='local', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,503 StorageProxy.java:1534 - Estimated result rows per range: 23.2875; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,504 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 394047
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 StorageProxy.java:1271 - Read: 2 ms.
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,505 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,506 Message.java:452 - Responding: ROWS [peer(system, peers), org.apache.cassandra.db.marshal.InetAddressType][rpc_address(system, peers), org.apache.cassandra.db.marshal.InetAddressType][schema_version(system, peers), org.apache.cassandra.db.marshal.UUIDType]
| 10.65.119.193 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
| 10.65.58.24 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Message.java:452 - Responding: ROWS [schema_version(system, local), org.apache.cassandra.db.marshal.UUIDType]
| b5291c1d-6635-3627-928f-f5a0f0c27ec1
---, v=3
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,509 Message.java:437 - Received: QUERY SELECT * FROM system.schema_keyspaces WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,509 Message.java:437 - Received: QUERY SELECT * FROM system.schema_usertypes WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,510 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,510 Message.java:437 - Received: QUERY SELECT * FROM system.schema_columnfamilies WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,510 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 394047
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,510 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,510 Message.java:437 - Received: QUERY SELECT * FROM system.schema_columns WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,510 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,511 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type][durable_writes(system, schema_keyspaces), org.apache.cassandra.db.marshal.BooleanType][strategy_class(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type][strategy_options(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,511 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-3-Data.db
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,511 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_usertypes), org.apache.cassandra.db.marshal.UTF8Type][type_name(system, schema_usertypes), org.apache.cassandra.db.marshal.UTF8Type][field_names(system, schema_usertypes), org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)][field_types(system, schema_usertypes), org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)]
---, v=3
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,513 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 656745
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][bloom_filter_fp_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][caching(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][cf_id(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UUIDType][column_aliases(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][comment(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_class(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_options(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][comparator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compression_parameters(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][default_time_to_live(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][default_validator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][dropped_columns(system, schema_columnfamilies), org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)][gc_grace_seconds(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][is_dense(system, schema_columnfamilies), org.apache.cassandra.db.marshal.BooleanType][key_aliases(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][key_validator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][local_read_repair_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][max_compaction_threshold(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][max_index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][memtable_flush_period_in_ms(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][min_compaction_threshold(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][min_index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][read_repair_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][speculative_retry(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][subcomparator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][type(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][value_alias(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 StorageProxy.java:1271 - Read: 2 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][column_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][component_index(system, schema_columns), org.apache.cassandra.db.marshal.Int32Type][index_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][index_options(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][index_type(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][type(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][validator(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,516 Message.java:437 - Received: QUERY SELECT * FROM system.local WHERE key='local', v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 Message.java:437 - Received: QUERY SELECT * FROM system.peers, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 StorageProxy.java:1534 - Estimated result rows per range: 23.2875; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 525396
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 656745
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 StorageProxy.java:1271 - Read: 1 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Message.java:452 - Responding: ROWS [key(system, local), org.apache.cassandra.db.marshal.UTF8Type][bootstrapped(system, local), org.apache.cassandra.db.marshal.UTF8Type][cluster_name(system, local), org.apache.cassandra.db.marshal.UTF8Type][cql_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][data_center(system, local), org.apache.cassandra.db.marshal.UTF8Type][gossip_generation(system, local), org.apache.cassandra.db.marshal.Int32Type][host_id(system, local), org.apache.cassandra.db.marshal.UUIDType][native_protocol_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system, local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local), org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][schema_version(system, local), org.apache.cassandra.db.marshal.UUIDType][thrift_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][tokens(system, local), org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system, local), org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]
| local | COMPLETED | CASSANDRAEDPDEVCLUSTER | 3.2.0 | DC1 | 1422663061 | 19dbb3ec-ada4-4af0-9d39-95eb89a0a8e7 | 3 | org.apache.cassandra.dht.Murmur3Partitioner | RAC1 | 2.1.2 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | 19.39.0 | …
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Message.java:452 - Responding: ROWS [peer(system, peers), org.apache.cassandra.db.marshal.InetAddressType][data_center(system, peers), org.apache.cassandra.db.marshal.UTF8Type][host_id(system, peers), org.apache.cassandra.db.marshal.UUIDType][preferred_ip(system, peers), org.apache.cassandra.db.marshal.InetAddressType][rack(system, peers), org.apache.cassandra.db.marshal.UTF8Type][release_version(system, peers), org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, peers), org.apache.cassandra.db.marshal.InetAddressType][schema_version(system, peers), org.apache.cassandra.db.marshal.UUIDType][tokens(system, peers), org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)]
| 10.65.119.193 | DC1 | 12bdc695-1647-462c-8167-e386603643c9 | null | RAC1 | 2.1.2 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
| 10.65.58.24 | DC1 | a1cf7f2c-fa1b-4d7d-95c9-381bcc2b36f9 | null | RAC1 | 2.1.2 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
---, v=3
Thanks,
Saurabh
From: Adam Holmberg <ad...@datastax.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Friday, January 30, 2015 at 9:25 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Unable to create a keyspace
I would first ask if you could upgrade to the latest version of Cassandra 2.1.x (presently 2.1.2).
If the issue still occurs consistently, it would be interesting to turn up logging on the client side and see if something is causing the client to disconnect during the metadata refresh following the schema change. If this yields further information, please raise the issue on the driver's user mailing list.
Adam Holmberg
On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi <Sa...@symantec.com>> wrote:
I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4 driver to create a keyspace followed by creating a column family within that keyspace from my unit test.
But I do not see the keyspace getting created and the code for creating column family fails because it cannot find the keyspace. I see the following in the system.log file:
INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472 MigrationManager.java:229 - Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
INFO [MigrationStage:1] 2015-01-28 17:59:08,476 ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326 - Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360 - Completed flushing /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803, position=10514)
This issue doesn’t happen always. My test runs fine sometimes but once it gets into this state, it remains there for a while and I can constantly reproduce this.
Also, when this issue happens for the first time, I also see the following error message in system.log file:
ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 - Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.8.0_05]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Anyone has any idea what might be going on here?
Thanks,
Saurabh
Re: Unable to create a keyspace
Posted by Asit KAUSHIK <as...@gmail.com>.
Saurabh a vague suggestion when you are dropping can you wait for sometime
to let the change propagate to other node.Also I see a replication factor 1
but you have 3 nodes....
On Jan 31, 2015 6:28 AM, "Saurabh Sethi" <Sa...@symantec.com> wrote:
> Thanks Adam. I upgraded to 2.1.2 but still seeing this issue. Let me go in
> a bit more detail as to what I am seeing – When I create a keyspace for the
> first time in a 3 node cluster, it works fine but if I drop the keyspace
> and try to recreate it, I see that the node received the request to create
> it but it didn’t delegate the request to other nodes because of which the
> keyspace didn’t get created on other two nodes.
>
> But the request for creating a table does get propagated to other nodes
> and since they can’t find that keyspace they throw an exception.
>
> I have 2 nodes as seeds, so the seeds property in cassandra.yaml for all 3
> nodes looks like – seeds: "<node1ip>,<node3ip>"
>
> Following is that I see in system.log file when I enable DEBUG mode:
>
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,363 Message.java:437 -
> Received: STARTUP {CQL_VERSION=3.0.0}, v=3
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,364 Message.java:452 -
> Responding: READY, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,365 Message.java:437 -
> Received: QUERY select cluster_name from system.local, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,366 StorageProxy.java:1534
> - Estimated result rows per range: 20.057144; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Message.java:452 -
> Responding: ROWS [cluster_name(system, local),
> org.apache.cassandra.db.marshal.UTF8Type]
> | CASSANDRAEDPDEVCLUSTER
> ---, v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,386 Message.java:437 -
> Received: QUERY CREATE KEYSPACE IF NOT EXISTS TestMaxColumnsKeySpace WITH
> REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };, v=3
> INFO [SharedPool-Worker-1] 2015-01-30 16:39:37,387
> MigrationManager.java:229 - Create new Keyspace:
> KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
> strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
> userTypes=org.apache.cassandra.config.UTMetaData@5485ec68}
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,388 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 262698
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,389 FileCacheService.java:102
> - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-3-Data.db
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 394047
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:102
> - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1-Data.db
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> INFO [MigrationStage:1] 2015-01-30 16:39:37,392
> ColumnFamilyStore.java:840 - Enqueuing flush of schema_keyspaces: 496 (0%)
> on-heap, 0 (0%) off-heap
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,392
> ColumnFamilyStore.java:166 - scheduling flush in 3600000 ms
> INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,394 Memtable.java:325 -
> Writing Memtable-schema_keyspaces@922450609(138 serialized bytes, 3 ops,
> 0%/0% of on/off-heap limit)
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Statistics.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Statistics.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Filter.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Filter.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-TOC.txt
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-TOC.txt
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Digest.sha1
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Digest.sha1
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-CompressionInfo.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-CompressionInfo.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Index.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Index.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,412 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Data.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
> INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 Memtable.java:364 -
> Completed flushing
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
> (175 bytes) for commitlog position ReplayPosition(segmentId=1422663061374,
> position=3040)
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413
> DataTracker.java:440 - adding
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10
> to list of files tracked for system.schema_keyspaces
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413
> CompactionManager.java:179 - Scheduling a background task check for
> system.schema_keyspaces with SizeTieredCompactionStrategy
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:265 -
> discard completed log segments for ReplayPosition(segmentId=1422663061374,
> position=3040), column family b0f22357-4458-3cdb-9631-c43e59ce3676
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,414
> CompactionManager.java:218 - Checking system.schema_keyspaces
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:283 -
> Not safe to delete active commit log segment
> CommitLogSegment(bin/../data/commitlog/CommitLog-4-1422663061374.log);
> dirty is peers (37f71aca-7dc2-383b-a706-72528af04d4f), compaction_history
> (b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca), local
> (7ad54392-bcdd-35a6-8417-4e047860b377),
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_columnfamilies
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> SizeTieredCompactionStrategy.java:86 - Compaction buckets are
> [[SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db'),
> SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-9-Data.db')]]
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_columns
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> SizeTieredCompactionStrategy.java:86 - Compaction buckets are []
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_triggers
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> CompactionManager.java:229 - No tasks available
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,416
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_usertypes
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,416 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,425 MigrationManager.java:418
> - Gossiping my schema version b5291c1d-6635-3627-928f-f5a0f0c27ec1
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Message.java:452 -
> Responding: RESULT schema change CREATED KEYSPACE testmaxcolumnskeyspace,
> v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,502 Message.java:437 -
> Received: QUERY SELECT peer, rpc_address, schema_version FROM system.peers,
> v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,502 Message.java:437 -
> Received: QUERY SELECT schema_version FROM system.local WHERE key='local',
> v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,503 StorageProxy.java:1534
> - Estimated result rows per range: 23.2875; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,504
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 394047
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 StorageProxy.java:1271
> - Read: 2 ms.
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,505 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,506 Message.java:452 -
> Responding: ROWS [peer(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][rpc_address(system,
> peers),
> org.apache.cassandra.db.marshal.InetAddressType][schema_version(system,
> peers), org.apache.cassandra.db.marshal.UUIDType]
> | 10.65.119.193 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> | 10.65.58.24 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Message.java:452 -
> Responding: ROWS [schema_version(system, local),
> org.apache.cassandra.db.marshal.UUIDType]
> | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> ---, v=3
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,509 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_keyspaces WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,509 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_usertypes WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,510
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,510 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_columnfamilies WHERE
> keyspace_name = 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,510
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 394047
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,510 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,510 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_columns WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,510 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,511
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_keyspaces),
> org.apache.cassandra.db.marshal.UTF8Type][durable_writes(system,
> schema_keyspaces),
> org.apache.cassandra.db.marshal.BooleanType][strategy_class(system,
> schema_keyspaces),
> org.apache.cassandra.db.marshal.UTF8Type][strategy_options(system,
> schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,511
> FileCacheService.java:102 - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-3-Data.db
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,511 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_usertypes),
> org.apache.cassandra.db.marshal.UTF8Type][type_name(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.UTF8Type][field_names(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)][field_types(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)]
> ---, v=3
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,513
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 656745
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][bloom_filter_fp_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][caching(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][cf_id(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UUIDType][column_aliases(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][comment(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_class(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_options(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][comparator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compression_parameters(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][default_time_to_live(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][default_validator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][dropped_columns(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)][gc_grace_seconds(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][is_dense(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.BooleanType][key_aliases(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][key_validator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][local_read_repair_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][max_compaction_threshold(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][max_index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][memtable_flush_period_in_ms(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][min_compaction_threshold(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][min_index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][read_repair_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][speculative_retry(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][subcomparator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][type(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][value_alias(system,
> schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 StorageProxy.java:1271
> - Read: 2 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][column_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][component_index(system,
> schema_columns),
> org.apache.cassandra.db.marshal.Int32Type][index_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][index_options(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][index_type(system,
> schema_columns), org.apache.cassandra.db.marshal.UTF8Type][type(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][validator(system,
> schema_columns), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,516 Message.java:437 -
> Received: QUERY SELECT * FROM system.local WHERE key='local', v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 Message.java:437 -
> Received: QUERY SELECT * FROM system.peers, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 StorageProxy.java:1534
> - Estimated result rows per range: 23.2875; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 656745
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 StorageProxy.java:1271
> - Read: 1 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Message.java:452 -
> Responding: ROWS [key(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][bootstrapped(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][cluster_name(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][cql_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][data_center(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][gossip_generation(system, local),
> org.apache.cassandra.db.marshal.Int32Type][host_id(system, local),
> org.apache.cassandra.db.marshal.UUIDType][native_protocol_version(system,
> local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system,
> local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][schema_version(system, local),
> org.apache.cassandra.db.marshal.UUIDType][thrift_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][tokens(system, local),
> org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system,
> local),
> org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]
> | local | COMPLETED | CASSANDRAEDPDEVCLUSTER | 3.2.0 | DC1 | 1422663061 |
> 19dbb3ec-ada4-4af0-9d39-95eb89a0a8e7 | 3 |
> org.apache.cassandra.dht.Murmur3Partitioner | RAC1 | 2.1.2 |
> b5291c1d-6635-3627-928f-f5a0f0c27ec1 | 19.39.0 | …
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Message.java:452 -
> Responding: ROWS [peer(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][data_center(system,
> peers), org.apache.cassandra.db.marshal.UTF8Type][host_id(system, peers),
> org.apache.cassandra.db.marshal.UUIDType][preferred_ip(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][rack(system, peers),
> org.apache.cassandra.db.marshal.UTF8Type][release_version(system, peers),
> org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][schema_version(system,
> peers), org.apache.cassandra.db.marshal.UUIDType][tokens(system, peers),
> org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)]
> | 10.65.119.193 | DC1 | 12bdc695-1647-462c-8167-e386603643c9 | null |
> RAC1 | 2.1.2 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
> | 10.65.58.24 | DC1 | a1cf7f2c-fa1b-4d7d-95c9-381bcc2b36f9 | null | RAC1
> | 2.1.2 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
> ---, v=3
>
> Thanks,
> Saurabh
> From: Adam Holmberg <ad...@datastax.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Friday, January 30, 2015 at 9:25 AM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: Unable to create a keyspace
>
> I would first ask if you could upgrade to the latest version of Cassandra
> 2.1.x (presently 2.1.2).
>
> If the issue still occurs consistently, it would be interesting to turn up
> logging on the client side and see if something is causing the client to
> disconnect during the metadata refresh following the schema change. If this
> yields further information, please raise the issue on the driver's user
> mailing list.
>
> Adam Holmberg
>
> On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi <Saurabh_Sethi@symantec.com
> > wrote:
>
>> I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4
>> driver to create a keyspace followed by creating a column family within
>> that keyspace from my unit test.
>>
>> But I do not see the keyspace getting created and the code for creating
>> column family fails because it cannot find the keyspace. I see the
>> following in the system.log file:
>>
>> INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472
>> MigrationManager.java:229 - Create new Keyspace:
>> KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
>> strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
>> userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
>> INFO [MigrationStage:1] 2015-01-28 17:59:08,476
>> ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%)
>> on-heap, 0 (0%) off-heap
>> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326
>> - Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3
>> ops, 0%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360
>> - Completed flushing
>> /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db
>> (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803,
>> position=10514)
>>
>> This issue doesn’t happen always. My test runs fine sometimes but once it
>> gets into this state, it remains there for a while and I can constantly
>> reproduce this.
>>
>> Also, when this issue happens for the first time, I also see the
>> following error message in system.log file:
>>
>> ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 - Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_05]
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_05]
>> at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.8.0_05]
>> at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
>>
>>
>> Anyone has any idea what might be going on here?
>>
>> Thanks,
>> Saurabh
>>
>
>
Re: Unable to create a keyspace
Posted by Saurabh Sethi <Sa...@symantec.com>.
Thanks Adam. I upgraded to 2.1.2 but still seeing this issue. Let me go in a bit more detail as to what I am seeing – When I create a keyspace for the first time in a 3 node cluster, it works fine but if I drop the keyspace and try to recreate it, I see that the node received the request to create it but it didn’t delegate the request to other nodes because of which the keyspace didn’t get created on other two nodes.
But the request for creating a table does get propagated to other nodes and since they can’t find that keyspace they throw an exception.
I have 2 nodes as seeds, so the seeds property in cassandra.yaml for all 3 nodes looks like – seeds: "<node1ip>,<node3ip>"
Following is that I see in system.log file when I enable DEBUG mode:
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,363 Message.java:437 - Received: STARTUP {CQL_VERSION=3.0.0}, v=3
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,364 Message.java:452 - Responding: READY, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,365 Message.java:437 - Received: QUERY select cluster_name from system.local, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,366 StorageProxy.java:1534 - Estimated result rows per range: 20.057144; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Message.java:452 - Responding: ROWS [cluster_name(system, local), org.apache.cassandra.db.marshal.UTF8Type]
| CASSANDRAEDPDEVCLUSTER
---, v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,386 Message.java:437 - Received: QUERY CREATE KEYSPACE IF NOT EXISTS TestMaxColumnsKeySpace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };, v=3
INFO [SharedPool-Worker-1] 2015-01-30 16:39:37,387 MigrationManager.java:229 - Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@5485ec68}
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,388 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 262698
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,389 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-3-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 394047
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
INFO [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:840 - Enqueuing flush of schema_keyspaces: 496 (0%) on-heap, 0 (0%) off-heap
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:166 - scheduling flush in 3600000 ms
INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,394 Memtable.java:325 - Writing Memtable-schema_keyspaces@922450609(138 serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Statistics.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Statistics.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Filter.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Filter.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-TOC.txt to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-TOC.txt
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Digest.sha1 to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Digest.sha1
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-CompressionInfo.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-CompressionInfo.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Index.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Index.db
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,412 FileUtils.java:161 - Renaming bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Data.db to bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 Memtable.java:364 - Completed flushing /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db (175 bytes) for commitlog position ReplayPosition(segmentId=1422663061374, position=3040)
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 DataTracker.java:440 - adding bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10 to list of files tracked for system.schema_keyspaces
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 CompactionManager.java:179 - Scheduling a background task check for system.schema_keyspaces with SizeTieredCompactionStrategy
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:265 - discard completed log segments for ReplayPosition(segmentId=1422663061374, position=3040), column family b0f22357-4458-3cdb-9631-c43e59ce3676
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,414 CompactionManager.java:218 - Checking system.schema_keyspaces
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:283 - Not safe to delete active commit log segment CommitLogSegment(bin/../data/commitlog/CommitLog-4-1422663061374.log); dirty is peers (37f71aca-7dc2-383b-a706-72528af04d4f), compaction_history (b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca), local (7ad54392-bcdd-35a6-8417-4e047860b377),
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_columnfamilies
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 SizeTieredCompactionStrategy.java:86 - Compaction buckets are [[SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db'), SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-9-Data.db')]]
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_columns
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 SizeTieredCompactionStrategy.java:86 - Compaction buckets are []
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_triggers
DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415 CompactionManager.java:229 - No tasks available
DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,416 ColumnFamilyStore.java:879 - forceFlush requested but everything is clean in schema_usertypes
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,416 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,425 MigrationManager.java:418 - Gossiping my schema version b5291c1d-6635-3627-928f-f5a0f0c27ec1
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Message.java:452 - Responding: RESULT schema change CREATED KEYSPACE testmaxcolumnskeyspace, v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,502 Message.java:437 - Received: QUERY SELECT peer, rpc_address, schema_version FROM system.peers, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,502 Message.java:437 - Received: QUERY SELECT schema_version FROM system.local WHERE key='local', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,503 StorageProxy.java:1534 - Estimated result rows per range: 23.2875; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,504 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 394047
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 StorageProxy.java:1271 - Read: 2 ms.
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,505 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,506 Message.java:452 - Responding: ROWS [peer(system, peers), org.apache.cassandra.db.marshal.InetAddressType][rpc_address(system, peers), org.apache.cassandra.db.marshal.InetAddressType][schema_version(system, peers), org.apache.cassandra.db.marshal.UUIDType]
| 10.65.119.193 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
| 10.65.58.24 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Message.java:452 - Responding: ROWS [schema_version(system, local), org.apache.cassandra.db.marshal.UUIDType]
| b5291c1d-6635-3627-928f-f5a0f0c27ec1
---, v=3
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,509 Message.java:437 - Received: QUERY SELECT * FROM system.schema_keyspaces WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,509 Message.java:437 - Received: QUERY SELECT * FROM system.schema_usertypes WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,510 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,510 Message.java:437 - Received: QUERY SELECT * FROM system.schema_columnfamilies WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,510 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 394047
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,510 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,510 Message.java:437 - Received: QUERY SELECT * FROM system.schema_columns WHERE keyspace_name = 'testmaxcolumnskeyspace', v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,510 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,511 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 525396
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type][durable_writes(system, schema_keyspaces), org.apache.cassandra.db.marshal.BooleanType][strategy_class(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type][strategy_options(system, schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,511 FileCacheService.java:102 - Evicting cold readers for /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-3-Data.db
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,511 StorageProxy.java:1271 - Read: 0 ms.
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_usertypes), org.apache.cassandra.db.marshal.UTF8Type][type_name(system, schema_usertypes), org.apache.cassandra.db.marshal.UTF8Type][field_names(system, schema_usertypes), org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)][field_types(system, schema_usertypes), org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)]
---, v=3
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,513 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 656745
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][bloom_filter_fp_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][caching(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][cf_id(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UUIDType][column_aliases(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][comment(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_class(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_options(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][comparator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][compression_parameters(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][default_time_to_live(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][default_validator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][dropped_columns(system, schema_columnfamilies), org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)][gc_grace_seconds(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][is_dense(system, schema_columnfamilies), org.apache.cassandra.db.marshal.BooleanType][key_aliases(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][key_validator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][local_read_repair_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][max_compaction_threshold(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][max_index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][memtable_flush_period_in_ms(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][min_compaction_threshold(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][min_index_interval(system, schema_columnfamilies), org.apache.cassandra.db.marshal.Int32Type][read_repair_chance(system, schema_columnfamilies), org.apache.cassandra.db.marshal.DoubleType][speculative_retry(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][subcomparator(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][type(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type][value_alias(system, schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 StorageProxy.java:1271 - Read: 2 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Message.java:452 - Responding: ROWS [keyspace_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][column_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][component_index(system, schema_columns), org.apache.cassandra.db.marshal.Int32Type][index_name(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][index_options(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][index_type(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][type(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type][validator(system, schema_columns), org.apache.cassandra.db.marshal.UTF8Type]
---, v=3
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,516 Message.java:437 - Received: QUERY SELECT * FROM system.local WHERE key='local', v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 Message.java:437 - Received: QUERY SELECT * FROM system.peers, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 StorageProxy.java:1534 - Estimated result rows per range: 23.2875; requested rows: 2147483647, ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517 FileCacheService.java:150 - Estimated memory usage is 1050792 compared to actual usage 525396
DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517 FileCacheService.java:150 - Estimated memory usage is 1182141 compared to actual usage 656745
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 StorageProxy.java:1271 - Read: 1 ms.
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Message.java:452 - Responding: ROWS [key(system, local), org.apache.cassandra.db.marshal.UTF8Type][bootstrapped(system, local), org.apache.cassandra.db.marshal.UTF8Type][cluster_name(system, local), org.apache.cassandra.db.marshal.UTF8Type][cql_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][data_center(system, local), org.apache.cassandra.db.marshal.UTF8Type][gossip_generation(system, local), org.apache.cassandra.db.marshal.Int32Type][host_id(system, local), org.apache.cassandra.db.marshal.UUIDType][native_protocol_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system, local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local), org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][schema_version(system, local), org.apache.cassandra.db.marshal.UUIDType][thrift_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][tokens(system, local), org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system, local), org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]
| local | COMPLETED | CASSANDRAEDPDEVCLUSTER | 3.2.0 | DC1 | 1422663061 | 19dbb3ec-ada4-4af0-9d39-95eb89a0a8e7 | 3 | org.apache.cassandra.dht.Murmur3Partitioner | RAC1 | 2.1.2 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | 19.39.0 | …
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Tracing.java:157 - request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Message.java:452 - Responding: ROWS [peer(system, peers), org.apache.cassandra.db.marshal.InetAddressType][data_center(system, peers), org.apache.cassandra.db.marshal.UTF8Type][host_id(system, peers), org.apache.cassandra.db.marshal.UUIDType][preferred_ip(system, peers), org.apache.cassandra.db.marshal.InetAddressType][rack(system, peers), org.apache.cassandra.db.marshal.UTF8Type][release_version(system, peers), org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, peers), org.apache.cassandra.db.marshal.InetAddressType][schema_version(system, peers), org.apache.cassandra.db.marshal.UUIDType][tokens(system, peers), org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)]
| 10.65.119.193 | DC1 | 12bdc695-1647-462c-8167-e386603643c9 | null | RAC1 | 2.1.2 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
| 10.65.58.24 | DC1 | a1cf7f2c-fa1b-4d7d-95c9-381bcc2b36f9 | null | RAC1 | 2.1.2 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
---, v=3
Thanks,
Saurabh
From: Adam Holmberg <ad...@datastax.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Friday, January 30, 2015 at 9:25 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Unable to create a keyspace
I would first ask if you could upgrade to the latest version of Cassandra 2.1.x (presently 2.1.2).
If the issue still occurs consistently, it would be interesting to turn up logging on the client side and see if something is causing the client to disconnect during the metadata refresh following the schema change. If this yields further information, please raise the issue on the driver's user mailing list.
Adam Holmberg
On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi <Sa...@symantec.com>> wrote:
I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4 driver to create a keyspace followed by creating a column family within that keyspace from my unit test.
But I do not see the keyspace getting created and the code for creating column family fails because it cannot find the keyspace. I see the following in the system.log file:
INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472 MigrationManager.java:229 - Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
INFO [MigrationStage:1] 2015-01-28 17:59:08,476 ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326 - Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360 - Completed flushing /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803, position=10514)
This issue doesn’t happen always. My test runs fine sometimes but once it gets into this state, it remains there for a while and I can constantly reproduce this.
Also, when this issue happens for the first time, I also see the following error message in system.log file:
ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 - Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_05]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.8.0_05]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Anyone has any idea what might be going on here?
Thanks,
Saurabh
Re: Unable to create a keyspace
Posted by Adam Holmberg <ad...@datastax.com>.
I would first ask if you could upgrade to the latest version of Cassandra
2.1.x (presently 2.1.2).
If the issue still occurs consistently, it would be interesting to turn up
logging on the client side and see if something is causing the client to
disconnect during the metadata refresh following the schema change. If this
yields further information, please raise the issue on the driver's user
mailing list.
Adam Holmberg
On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi <Sa...@symantec.com>
wrote:
> I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4
> driver to create a keyspace followed by creating a column family within
> that keyspace from my unit test.
>
> But I do not see the keyspace getting created and the code for creating
> column family fails because it cannot find the keyspace. I see the
> following in the system.log file:
>
> INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472
> MigrationManager.java:229 - Create new Keyspace:
> KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
> strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
> userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
> INFO [MigrationStage:1] 2015-01-28 17:59:08,476
> ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%)
> on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326 -
> Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3 ops,
> 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360 -
> Completed flushing
> /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db
> (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803,
> position=10514)
>
> This issue doesn’t happen always. My test runs fine sometimes but once it
> gets into this state, it remains there for a while and I can constantly
> reproduce this.
>
> Also, when this issue happens for the first time, I also see the following
> error message in system.log file:
>
> ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 - Unexpected exception during request
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_05]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_05]
> at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.8.0_05]
> at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
>
>
> Anyone has any idea what might be going on here?
>
> Thanks,
> Saurabh
>