You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jens Rantil <je...@tink.se> on 2014/10/22 16:05:08 UTC
Cluster/node with inconsistent schema
Hi,
I have a table that I dropped, recreated with two clustering primary keys (only had a single partition key before), and loaded previous data into the table.
I started noticing that a single node of mine was not able to do `ORDER BY` executions on the table (while the other nodes were). What was interesting was that `DESCRIBE TABLE mytable` showed correct PRIMARY KEY, and schema version was the same on all machines when I looked at system.peers as well as system.local.
On the failing node I was seeing exceptions such as https://gist.github.com/JensRantil/c6b2df5a5a2e12cdd3df.
I restarted the failing node in the belief the maybe I would force the gossip to get into a consistent state. Now I am, instead, getting RPC timeout when trying to SELECT against the table while logs are giving me https://gist.github.com/JensRantil/3b238e47dd6cd33732c1.
Any input appreciated. Would you suggest I drain the node, clear all sstables (rm -fr /var/lib/cassandra/mykeyspace/mytable/*), boot up Cassandra and run a full repair?
Cheers,
Jens
———
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook Linkedin Twitter
Re: Cluster/node with inconsistent schema
Posted by Jens Rantil <je...@tink.se>.
Hi again,
Follow-up: The incorrect schema propagated to other servers. Luckily this was a smaller table. I dropped the table and noticed that no sstables were removed. I then created the table again, and truncated it instead. This removed all the sstables and things look good now.
Cheers,
Jens
———
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook Linkedin Twitter
On Wed, Oct 22, 2014 at 4:05 PM, Jens Rantil <je...@tink.se> wrote:
> Hi,
> I have a table that I dropped, recreated with two clustering primary keys (only had a single partition key before), and loaded previous data into the table.
> I started noticing that a single node of mine was not able to do `ORDER BY` executions on the table (while the other nodes were). What was interesting was that `DESCRIBE TABLE mytable` showed correct PRIMARY KEY, and schema version was the same on all machines when I looked at system.peers as well as system.local.
> On the failing node I was seeing exceptions such as https://gist.github.com/JensRantil/c6b2df5a5a2e12cdd3df.
> I restarted the failing node in the belief the maybe I would force the gossip to get into a consistent state. Now I am, instead, getting RPC timeout when trying to SELECT against the table while logs are giving me https://gist.github.com/JensRantil/3b238e47dd6cd33732c1.
> Any input appreciated. Would you suggest I drain the node, clear all sstables (rm -fr /var/lib/cassandra/mykeyspace/mytable/*), boot up Cassandra and run a full repair?
> Cheers,
> Jens
> ———
> Jens Rantil
> Backend engineer
> Tink AB
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
> Facebook Linkedin Twitter