You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sam Overton (JIRA)" <ji...@apache.org> on 2011/07/26 17:49:10 UTC
[jira] [Updated] (CASSANDRA-2948) Nodetool move fails to stream out
data from moved node to new endpoint.
[ https://issues.apache.org/jira/browse/CASSANDRA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Overton updated CASSANDRA-2948:
-----------------------------------
Attachment: 2948.patch
Attached patch which will cause Gossiper to broadcast notifications when the local application state is updated.
It also fixes the calculation of the endpoints to stream from, which should use the old state of the ring, not the new one.
> Nodetool move fails to stream out data from moved node to new endpoint.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Reporter: Sam Overton
> Priority: Critical
> Labels: move, streaming
> Attachments: 2948.patch
>
>
> When moving a node in the ring with nodetool move, that node streams its data to itself instead of to the new endpoint responsible for its old range.
> Steps to reproduce:
> * Create a cluster (A,B,C,D) with tokens (0,4,8,C) using ByteOrderedPartitioner
> * Create a keyspace and CF with RF=1
> * Insert keys (2,6,A,E). This should put one key on each node.
> * Move node A to token 7. This should cause:
> + node C streams key 6 to node A
> + node A streams key E to node B
> However instead, node A streams key E to itself.
> Selected log messages from node A:
> INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,075 StorageService.java (line 1878) Moving miles/10.2.129.41 from Token(bytes[00]) to Token(bytes[07]).
> DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,080 StorageService.java (line 1941) Table ks: work map {/10.2.129.16=[(Token(bytes[04]),Token(bytes[07])]]}.
> INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:17,080 StorageService.java (line 1946) Sleeping 30000 ms before start streaming/fetching ranges.
> ...
> INFO [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,728 StorageService.java (line 522) Moving: fetching new ranges and streaming old ranges
> DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,728 StorageService.java (line 1960) [Move->STREAMING] Work Map: {ks={(Token(bytes[0c]),Token(bytes[00])]=[miles/10.2.129.41]}}
> DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,729 StorageService.java (line 1965) [Move->FETCHING] Work Map: {ks={/10.2.129.16=[(Token(bytes[04]),Token(bytes[07])]]}}
> DEBUG [RMI TCP Connection(6)-10.2.129.41] 2011-07-26 16:29:46,730 StorageService.java (line 2411) Requesting from /10.2.129.16 ranges (Token(bytes[04]),Token(bytes[07])]
> ...
> INFO [StreamStage:1] 2011-07-26 16:29:46,737 StreamOut.java (line 90) Beginning transfer to miles/10.2.129.41
> DEBUG [StreamStage:1] 2011-07-26 16:29:46,737 StreamOut.java (line 91) Ranges are (Token(bytes[0c]),Token(bytes[00])]
> This appears to be caused because in StorageService.move we call
> Gossiper.instance.addLocalApplicationState(ApplicationState.STATUS, valueFactory.moving(newToken));
> and then get the new token metadata in order to calculate where the new endpoint is that we should stream to
> TokenMetadata tokenMetaClone = tokenMetadata_.cloneAfterAllSettled();
> however, in addLocalApplicationState there is no notification broadcast for the change in local state, so tokenMetadata_ never updates the list of moving nodes, and the tokenMetaClone is still the state of the ring from before the move.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira