You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2012/12/04 18:53:59 UTC
[jira] [Created] (CASSANDRA-5025) Schema push/pull race
Jonathan Ellis created CASSANDRA-5025:
-----------------------------------------
Summary: Schema push/pull race
Key: CASSANDRA-5025
URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 1.1.8
When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5025) Schema push/pull race
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510087#comment-13510087 ]
Chris Herron commented on CASSANDRA-5025:
-----------------------------------------
For patch 5025.txt:
A single schema migration will result in N (num nodes) gossips of the new schema version (as before). Through MigrationManager.onChange()->rectifySchema(), those will each result in a delayed comparison of value 'theirVersion', but that value is now one minute old.
Further, if some new schema migration happens to be underway, the same effect of redundant repeat RowMutations will occur.
Schema migrations tend to happen in bursts - so this patch seems like it might reduce the problem but not eliminate it.
Would it not be better to have DefsTable.mergeSchema call Schema.instance.updateVersion instead of Schema.instance.updateVersionAndAnnounce and then deal with temporarily unavailable nodes by doing a MigrationManager.passiveAnnounce(version) if/when we see them come back online?
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5025) Schema push/pull race
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510023#comment-13510023 ]
Jonathan Ellis commented on CASSANDRA-5025:
-------------------------------------------
patch attached to add a delay to rectifySchema to give concurrent changes a chance to propagate first
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5025) Schema push/pull race
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510585#comment-13510585 ]
Chris Herron commented on CASSANDRA-5025:
-----------------------------------------
Clarifying for anyone else who encounters this issue:
* This problem was introduced in CASSANDRA-3931
* For use cases that involve creation/update/deletion of multiple keyspaces or column families, the symptom will be increasingly slow schema migrations as the KS/CF population grows. Depending on client RPC timeout config, schema change requests may fail.
* In a test environment running stock C* 1.1.7, for a test that creates new CFs in sequence, we see the following CF creation times:
** Empty cluster: sub-second
** 200+ CFs: 15s ave.
** 400+ CFs: 30s+ with eventual failure due to 30s client side (Hector) RPC timeout.
* In the same test environment running 1.1.7 patched with 5025.txt:
** For the first 60s duration of the test, CF creation times are sub-second
** At 60s, the delayed rectifySchema migration calls kick in and creation times drop to 50s+ (including waits for schema agreement) with eventual failure due to 30s client side RPC timeout.
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5025) Schema push/pull race
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510914#comment-13510914 ]
Pavel Yaskevich commented on CASSANDRA-5025:
--------------------------------------------
[~cherro] I don't think that we every want to make such a frequent schema changes, it's not considered a good practice for a number of reasons, unless you are trying to do something like temp-tables which has it's own implications... Agree on [~jbellis] that it doesn't seem like a good idea to re-invent HH for schema until we have a good reason to do so.
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5025) Schema push/pull race
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-5025:
--------------------------------------
Attachment: 5025.txt
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5025) Schema push/pull race
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510088#comment-13510088 ]
Jonathan Ellis commented on CASSANDRA-5025:
-------------------------------------------
I really don't want to reinvent HH poorly for schema migrations. Maybe Pavel has a better suggestion.
> Schema push/pull race
> ---------------------
>
> Key: CASSANDRA-5025
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5025
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.8
>
> Attachments: 5025.txt
>
>
> When a schema change is made, the coordinator pushes the delta to the other nodes in the cluster. This is more efficient than sending the entire schema. But the coordinator also announces the new schema version, so the other nodes' reception of the new version races with processing the delta, and usually seeing the new schema wins. So the other nodes also issue a pull to the coordinator for the entire schema.
> Thus, schema changes tend to become O(n) in the number of KS and CF present.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira