You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sam Tunnicliffe (JIRA)" <ji...@apache.org> on 2016/08/31 10:26:20 UTC
[jira] [Commented] (CASSANDRA-12575) A Cassandra node gets
corrupted and contains only "key" and "column1" columns
[ https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451840#comment-15451840 ]
Sam Tunnicliffe commented on CASSANDRA-12575:
---------------------------------------------
The stacktraces reported during startup are unrelated to the schema problem. They are the result of a race triggered when multiple nodes are started concurrently, which is fixed by CASSANDRA-10761, and included in 2.2.5.
> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-12575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
> Project: Cassandra
> Issue Type: Bug
> Environment: We use a 3 data center environment, each data center contains two Cassandra nodes.
> Reporter: Michal Lefler
> Attachments: 122310b.jpg, Site2_Tables.JPG, system.log, system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were functioning correctly.
> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24] [TopologyManager.<init>()] com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
> at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
> at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
> at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
> at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
> {code}
> After some investigation, we found out that we got this kind of an exception whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
> When we tried to query the tables directly using CQL, we got the following result:
> All the tables only had one column named "key" for the partition key, and a "columnx" column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and flowType is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId | flowType | currentVersion | flowName | flowOwner | flowState | lockUpdatedTime | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi | 5.1-787 | installation | | idle | 1472618536062180 | 5.1-787 | 1
> EMS-roger-01-02 | topology | null | topology_inserted | 10.240.32.130 | idle | 1472618335876404 | null | 1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key | column1
> -----------------+-----------
> EMS-roger-01-02 | statusApi
> EMS-roger-01-02 | topology
> {code}
> We tried to restart the node, and also to performed a full repair, but nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception encountered during startup
> java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table roles
> at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529) [apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table roles
> at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798) ~[apache-cassandra-2.2.4.jar:2.2.4]
> at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454) ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 8 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)