You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Lajos Bodrogi (JIRA)" <ji...@apache.org> on 2013/12/20 13:08:11 UTC
[jira] [Commented] (CASSANDRA-6224) CQL3 Column family / tables
"disappear", get unconfigured columnfamily errors
[ https://issues.apache.org/jira/browse/CASSANDRA-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853891#comment-13853891 ]
Lajos Bodrogi commented on CASSANDRA-6224:
------------------------------------------
Maybe i have a related problem. (Cassandra 2.0.3, Ubuntu 12.04, Cluster of 3 nodes)
I got the following Perl error:
ARGH! NEW PRIMARY FAILED: OPCODE_ERROR: unconfigured columnfamily source 8704 Protocol::CassandraCQL::Frame=SCALAR(0x10f79a0)
And one node stopped responding (the java process was still running) while the others in the cluster kept working.
Here are the server logs:
INFO 12:23:26,299 flushing high-traffic column family CFS(Keyspace='vst8', ColumnFamily='series') (estimated 38461316 bytes)
INFO 12:23:26,299 Enqueuing flush of Memtable-series@1926422961(11944355/38461316 serialized/live bytes, 230603 ops)
INFO 12:23:26,300 Writing Memtable-series@1926422961(11944355/38461316 serialized/live bytes, 230603 ops)
ERROR 12:23:26,511 Exception in thread Thread[FlushWriter:46,5,main]
FSWriteError in /var/lib/cassandra/data/vst8/series/vst8-series-tmp-jb-760-Filter.db
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:462)
at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:210)
at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:288)
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:387)
at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:320)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.write(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:282)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
at org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:455)
... 9 more
ERROR 12:23:26,513 Stopping gossiper
WARN 12:23:26,513 Stopping gossip by operator request
INFO 12:23:26,513 Announcing shutdown
ERROR 12:23:28,513 Stopping RPC server
INFO 12:23:28,513 Stop listening to thrift clients
ERROR 12:23:28,517 Stopping native transport
INFO 12:23:28,532 Stop listening for CQL clients
As you can see, the disk went full.
> CQL3 Column family / tables "disappear", get unconfigured columnfamily errors
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-6224
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6224
> Project: Cassandra
> Issue Type: Bug
> Components: API, Core
> Environment: Cassandra 2.0.1 Amazon AWS Ubuntu Single-node
> Reporter: Constance Eustace
>
> We're seeing CQL3 tables seemingly arbitrarily disappear. Need to repair for prod meant we reconstructed the affected schema before nodetool repairs or similar attempts were done.
> It seems to take a few days to appear. Volumes are not tremendously high yet...
> Caused by: java.sql.SQLSyntaxErrorException: InvalidRequestException(why:unconfigured columnfamily entity_hierarchydef)
> at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:103) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:388) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:372) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:50) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281) ~[commons-dbcp-1.3.jar:1.3]
> at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313) ~[commons-dbcp-1.3.jar:1.3]
> at com.bestbuy.contentsystems.cupcake.storage.cassandra.cqlentity.CQL.tool.CassPSC.createPreparedStatement(CassPSC.java:61) ~[ingest-storage-QA-SNAPSHOT.jar:QA-SNAPSHOT]
> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:583) ~[spring-jdbc-3.2.4.RELEASE.jar:3.2.4.RELEASE]
> ... 148 common frames omitted
> Caused by: org.apache.cassandra.thrift.InvalidRequestException: null
> at org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567) ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.7.0.jar:0.7.0]
> at org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625) ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611) ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:517) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:532) ~[cassandra-jdbc-1.2.5.jar:na]
> at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:96) ~[cassandra-jdbc-1.2.5.jar:na]
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)