You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "James Baker (Jira)" <ji...@apache.org> on 2021/02/08 19:17:00 UTC
[jira] [Commented] (CASSANDRA-15355) Schema push/pull race on
continuous schema changes
[ https://issues.apache.org/jira/browse/CASSANDRA-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281320#comment-17281320 ]
James Baker commented on CASSANDRA-15355:
-----------------------------------------
The code has now been refactored and I believe that the bug is located here: [https://github.com/apache/cassandra/blob/159e021aa36a44cdc7674dd234d4a6e189478280/src/java/org/apache/cassandra/schema/MigrationCoordinator.java#L205] though I might be mistaken. I think it's still a problem though. Would be happy to rebase my fix if it's desired.
> Schema push/pull race on continuous schema changes
> --------------------------------------------------
>
> Key: CASSANDRA-15355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15355
> Project: Cassandra
> Issue Type: Bug
> Reporter: James Baker
> Priority: Normal
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In https://issues.apache.org/jira/browse/CASSANDRA-5025, pull based schema updates were scheduled 1 minute after the schema change was first visible, so as to prefer the push codepath as much as possible.
> Unfortunately, this does not handle the case where there are many schema changes happening - imagine a scenario where we create a table every 5 seconds for 2 minutes - the first update tasks execute 60 seconds in and the schemas may well be out of sync between nodes at that point.
> In this case, there is some chance that when the task runs, the schemas are out of sync because a subsequent schema update has occurred, and so the same push/pull race has happened.
> A fix is to modify the codepath such that the scheduled task is only run if the other node's schema version is the same as when the task was scheduled. A different (later scheduled) task should run otherwise.
> For us, what we see is that when we have a reasonably large number of changes, a few schema changes can have the unfortunate outcome of causing our nodes to run out of memory and crash - if we have a 30 node cluster, create a table every second for 2 minutes, and for some reason we pause for 10 seconds after 60 seconds with no progress, we can easily end up currently running 300 schema pulls for a single node. These can cause further piling up which causes cascading failures. This change stops that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org