You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tyler Hobbs (JIRA)" <ji...@apache.org> on 2011/04/22 00:57:05 UTC
[jira] [Created] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Schema disagreements when using connections to multiple hosts
-------------------------------------------------------------
Key: CASSANDRA-2536
URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8 beta 1
Environment: Two node 0.8-beta1 cluster with one seed and JNA.
Reporter: Tyler Hobbs
Attachments: schema_disagree.py
If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023257#comment-13023257 ]
Tyler Hobbs commented on CASSANDRA-2536:
----------------------------------------
Actually, I can also reproduce this with a two node 0.7.4 cluster. I'm pretty sure that this does not happen with 0.7.3, but I'll go ahead and verify that.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Hobbs updated CASSANDRA-2536:
-----------------------------------
Attachment: schema_disagree.py
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2536:
--------------------------------------
Priority: Minor (was: Major)
Affects Version/s: (was: 0.8 beta 1)
0.7.0
Fix Version/s: 0.8.1
0.7.6
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Priority: Minor
> Fix For: 0.7.6, 0.8.1
>
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Hobbs reassigned CASSANDRA-2536:
--------------------------------------
Assignee: Tyler Hobbs (was: Gary Dusbabek)
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023263#comment-13023263 ]
Tyler Hobbs commented on CASSANDRA-2536:
----------------------------------------
Nevermind my thoughts that it doesn't happen in 0.7.3 -- it seems to happen there too. It appears this is not a recent problem.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026292#comment-13026292 ]
Gary Dusbabek commented on CASSANDRA-2536:
------------------------------------------
bq. any thoughts...
I was going to add some jmx to get the last N schema versions (seems like it would be handy anyway and will be necessary if we ever get the rollback pony). Send schema to node A, verify that schema is propagated to B, send schema to B and watch the problem happen. The code to start looking at are the Definitions*VerbHandlers.
Schema version is tracked in two places: gossip and in DatabaseDescriptor.defsVersion. Make sure those are reasonably in sync (was the sourced of one bug in the past).
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026070#comment-13026070 ]
Jonathan Ellis commented on CASSANDRA-2536:
-------------------------------------------
Gary, any thoughts on where to start looking?
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Mike Bulman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022978#comment-13022978 ]
Mike Bulman commented on CASSANDRA-2536:
----------------------------------------
I feel like a better, more critical sounding explanation, is: create a keyspace on node1, create a cf in that keyspace on node2 = hang + schema disagreement.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026674#comment-13026674 ]
Tyler Hobbs commented on CASSANDRA-2536:
----------------------------------------
The issue is the clocks being out of sync between nodes. Sometimes the v1 UUID generated by the second node has an earlier timestamp than the current schema UUID has.
There are a couple of things that could be fixed here:
1. A node shouldn't accept a schema change if the timestamp for the new schema would be earlier than its current schema.
2. Schema modification calls should accept an optional client-side timestamp that will be used for the v1 UUID.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026896#comment-13026896 ]
Sylvain Lebresne commented on CASSANDRA-2536:
---------------------------------------------
I'll hijack this conversation by saying that I think we should start advertising that people should try to keep their server clocks in sync unless they have a good reason not too (which would legitimize the fact that "timestamp comparison is good enough"). Counter removes for instance use server side timestamps and would be screwed up by diverging clocks (and by that I mean more screwed up than they already are by design). And really, is there any reason not to install a ntpd server in the first place anyway?
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Dhaivat Pandit (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025942#comment-13025942 ]
Dhaivat Pandit commented on CASSANDRA-2536:
-------------------------------------------
encountered this on fresh 0.7.4 - 5 nodes - 100G+ per node.
decommissioning the bad node and rejoin fixed the problem.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026698#comment-13026698 ]
Jonathan Ellis commented on CASSANDRA-2536:
-------------------------------------------
bq. A node shouldn't accept a schema change if the timestamp for the new schema would be earlier than its current schema.
You need this with or without the client-side timestamp, though; there's no sense in letting people blow their leg off.
And once you have that you don't need to add a client-side timestamp with all the PITA-ness that involves.
(And unlike with data modification, I can't think of a use for doing "clever" things w/ a client side timesamp. So pushing it to the client doesn't really solve anything, just means you need to sync clocks across more machines.)
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026710#comment-13026710 ]
Tyler Hobbs commented on CASSANDRA-2536:
----------------------------------------
{quote}
Wouldn't that update be DOA then? I thought we checked to make sure the new migration compared after the current migration (as well as making sure the new migration's previous version matches with the current version).
{quote}
We do check that the previous version matches, but the migration is applied locally without comparing the current and new uuids.
{quote}
If the clocks are that far off sync, I think the cluster has bigger problems (like writes not being applied).
{quote}
This can theoretically happen with clocks being off by only tens of milliseconds.
{quote}
And unlike with data modification, I can't think of a use for doing "clever" things w/ a client side timesamp. So pushing it to the client doesn't really solve anything, just means you need to sync clocks across more machines.
{quote}
Not for clever purposes -- it seems to me that clients making schema modifications are more likely to be centralized, so schema changes coming from a single client will (almost) always have increasing timestamps.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026801#comment-13026801 ]
Jonathan Ellis commented on CASSANDRA-2536:
-------------------------------------------
bq. it seems to me that clients making schema modifications are more likely to be centralized
I would have also argued that they are likely to use the same connection (to the same server), and look where that got us. :)
bq. I personally think the timestamp comparison is good enough for now
I am okay with this. What do you think, Gary?
(Nit: the exception message says "older" but the comparison is "older or equal.")
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026684#comment-13026684 ]
Gary Dusbabek commented on CASSANDRA-2536:
------------------------------------------
bq. Sometimes the v1 UUID generated by the second node has an earlier timestamp than the current schema UUID has.
Wouldn't that update be DOA then? I thought we checked to make sure the new migration compared after the current migration (as well as making sure the new migration's previous version matches with the current version).
bq. A node shouldn't accept a schema change if the timestamp for the new schema would be earlier than its current schema.
If the clocks are *that* far off sync, I think the cluster has bigger problems (like writes not being applied). Plus, it would be easy for a node whose clock is way head to 'poison' schema updates from the rest of the cluster who are, in effect, behind the times.
bq. Schema modification calls should accept an optional client-side timestamp that will be used for the v1 UUID.
Seems like a better approach.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Tyler Hobbs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Hobbs updated CASSANDRA-2536:
-----------------------------------
Attachment: 2536-compare-timestamp.txt
Attached patch compares version timestamps before applying migration locally.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Brian Lovett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024804#comment-13024804 ]
Brian Lovett commented on CASSANDRA-2536:
-----------------------------------------
Just bumped into this on a fresh 0.7.4 install on our test cluster. Does this only happen in a 2 node ring?
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2536:
-----------------------------------------
Assignee: Gary Dusbabek
And this is fine in 0.7?
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Gary Dusbabek
> Attachments: schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027046#comment-13027046 ]
Hudson commented on CASSANDRA-2536:
-----------------------------------
Integrated in Cassandra-0.7 #462 (See [https://builds.apache.org/hudson/job/Cassandra-0.7/462/])
refuse to apply migrations with older timestamps than the current schema
patch by Tyler Hobbs; reviewed by jbellis and gdusbabek for CASSANDRA-2536
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Priority: Minor
> Fix For: 0.7.6, 0.8.1
>
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2536:
--------------------------------------
Fix Version/s: (was: 0.8.1)
0.8.0
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Priority: Minor
> Fix For: 0.7.6, 0.8.0
>
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2536) Schema disagreements when using
connections to multiple hosts
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026953#comment-13026953 ]
Gary Dusbabek commented on CASSANDRA-2536:
------------------------------------------
I think timestamp comparisons will be fine.
> Schema disagreements when using connections to multiple hosts
> -------------------------------------------------------------
>
> Key: CASSANDRA-2536
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2536
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: Two node 0.8-beta1 cluster with one seed and JNA.
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Attachments: 2536-compare-timestamp.txt, schema_disagree.py
>
>
> If you have two thrift connections open to different nodes and you create a KS using the first, then a CF in that KS using the second, you wind up with a schema disagreement even if you wait/sleep after creating the KS.
> The attached script reproduces the issue using pycassa (1.0.6 should work fine, although it has the 0.7 thrift-gen code). It's also reproducible by hand with two cassandra-cli sessions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira