[jira] [Comment Edited] (CASSANDRA-7186) alter table add column not always propogating

["Alexander Bulaev (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231945#comment-14231945 ] 

Alexander Bulaev edited comment on CASSANDRA-7186 at 12/2/14 7:15 PM:
----------------------------------------------------------------------

We have 9 nodes total in 3 DCs for our production cluster.
Even after applying the workaround posted here (run ALTER on each node), we have multiple schema versions in the cluster:
{noformat}
Cluster Information:
	Name: music-cass cluster
	Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
	Schema versions:
		c9039c04-afee-3e5e-bd78-b3d7201cb154: [2a02:6b8:0:c08:0:0:0:22, 2a02:6b8:0:c08:0:0:0:23, 2a02:6b8:0:c08:0:0:0:21, 2a02:6b8:0:2514:0:0:0:39, 2a02:6b8:0:2514:0:0:0:40, 2a02:6b8:0:2514:0:0:0:41]

		3dff776f-fad6-3150-b7c3-0415366cc85e: [2a02:6b8:0:1a1b:0:0:0:30, 2a02:6b8:0:1a1b:0:0:0:31, 2a02:6b8:0:1a1b:0:0:0:29]
{noformat}

This also reproduced on both our testing clusters (3 nodes total in 3 DCs).



was (Author: alexbool):
We have 9 nodes total in 3 DCs for our production cluster.
Even after applying the workaround posted here (run ALTER on each node), we have multiple schema version in the cluster:
{noformat}
Cluster Information:
	Name: music-cass cluster
	Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
	Schema versions:
		c9039c04-afee-3e5e-bd78-b3d7201cb154: [2a02:6b8:0:c08:0:0:0:22, 2a02:6b8:0:c08:0:0:0:23, 2a02:6b8:0:c08:0:0:0:21, 2a02:6b8:0:2514:0:0:0:39, 2a02:6b8:0:2514:0:0:0:40, 2a02:6b8:0:2514:0:0:0:41]

		3dff776f-fad6-3150-b7c3-0415366cc85e: [2a02:6b8:0:1a1b:0:0:0:30, 2a02:6b8:0:1a1b:0:0:0:31, 2a02:6b8:0:1a1b:0:0:0:29]
{noformat}

This also reproduced on both our testing clusters (3 nodes total in 3 DCs).


> alter table add column not always propogating
> ---------------------------------------------
>
>                 Key: CASSANDRA-7186
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7186
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Martin Meyer
>            Assignee: Philip Thompson
>             Fix For: 2.0.12
>
>
> I've been many times in Cassandra 2.0.6 that adding columns to existing tables seems to not fully propagate to our entire cluster. We add an extra column to various tables maybe 0-2 times a week, and so far many of these ALTERs have resulted in at least one node showing the old table description a pretty long time (~30 mins) after the original ALTER command was issued.
> We originally identified this issue when a connected clients would complain that a column it issued a SELECT for wasn't a known column, at which point we have to ask each node to describe the most recently altered table. One of them will not know about the newly added field. Issuing the original ALTER statement on that node makes everything work correctly.
> We have seen this issue on multiple tables (we don't always alter the same one). It has affected various nodes in the cluster (not always the same one is not getting the mutation propagated). No new nodes have been added to the cluster recently. All nodes are homogenous (hardware and software), running 2.0.6. We don't see any particular errors or exceptions on the node that didn't get the schema update, only the later error from a Java client about asking for an unknown column in a SELECT. We have to check each node manually to find the offender. The tables he have seen this on are under fairly heavy read and write load, but we haven't altered any tables that are not, so that might not be important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)