You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/11/11 19:12:39 UTC
[jira] Created: (CASSANDRA-541) allow moving a live node with data
on it
allow moving a live node with data on it
----------------------------------------
Key: CASSANDRA-541
URL: https://issues.apache.org/jira/browse/CASSANDRA-541
Project: Cassandra
Issue Type: New Feature
Components: Core, Tools
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 0.5
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777034#action_12777034 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
rebased and folded Jaakko's changes into 541-v2
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541-v2.patch, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: 541-v2.patch
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541-v2.patch, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777261#action_12777261 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
waitForWritable calls for waitForIO(SelectionKey.OP_WRITE)
and waitForIO is
void waitForIO(int ops) throws IOException {
if (selector.select(channel, ops, timeout) == 0) {
throw new SocketTimeoutException(timeoutExceptionString(ops));
}
}
I don't think this will work directly easily with our wrapper around select, but we can hack something in...
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777224#action_12777224 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
r/m'd old patches. new ones are
01: the old patches, combined
02: a bunch of fixes (split out for easy review)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776996#action_12776996 ]
Jaakko Laine commented on CASSANDRA-541:
----------------------------------------
Move would seem to allways cause this:
java.lang.IllegalArgumentException: value already present: /192.168.0.108
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:116)
at com.google.common.collect.StandardBiMap.putInBothMaps(StandardBiMap.java:106)
at com.google.common.collect.StandardBiMap.put(StandardBiMap.java:91)
at com.google.common.collect.HashBiMap.put(HashBiMap.java:82)
at org.apache.cassandra.locator.TokenMetadata.update(TokenMetadata.java:91)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:205)
at org.apache.cassandra.service.StorageService.updateBootstrapRanges(StorageService.java:473)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:396)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:678)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:643)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:595)
at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(Gossiper.java:1014)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Most likely the same reason as with #543.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Attachment: 0005-fix-pending-ranges-handling-in-node-move.txt
I think there are two cases in movement:
(I) Node moves to a new position in the ring
(II) Node extends its range by modifying token (special case of (I))
For (I) I think we need to do the following transition: -> LEAVING(oldToken) -> LEFT(oldToken) -> BOOTSTRAPPING (newToken) -> NORMAL(newToken). I don't see any way to cut corners here as we need to stream our old ranges away, signal the end of that operation, and then bootstrap to the new range. This is exactly the same as decommission -> bootsrap. Attached patch modifies StorageService accordingly. We could perhaps jump from LEAVING directly to BOOTSRAPPING, but removing pending ranges reliably becomes slightly fragile as old token is not part of bootstrapping message. Better to signal that clearly by a separate state I think.
(II) is a special case of (I) as the node already has part of its future range. For this case we could use STATE_MOVING to signal that we're not changing our whole range, only adding to it. Since this transition only involves streaming data between the node and its successor, I think NORMAL - MOVING(newToken) -> NORMAL(newToken) would be sufficient. The only nagging worry here is what happens to the data that moves from the successor to the node enlarging its range. If done directly between these two nodes, we might easily violate replica placement rules for this range.
Anyway, since (I) works for all move scenarios (only wastes a lot of energy if (II) would be sufficient), I think we could move that to a separate task and first commit this patchset.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 0001-CASSANDRA-541-add-Move-command.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777904#action_12777904 ]
Hudson commented on CASSANDRA-541:
----------------------------------
Integrated in Cassandra #258 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/258/])
workaround for "Resource temporarily unavailable"
patch by jbellis; reviewed by Jaakko Laine for
fix for when bootstrap source has no data in the range requested
patch by jbellis; reviewed by Jaakko Laine for
add Move command
patch by jbellis; reviewed by Jaakko Laine for
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776846#action_12776846 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
yes, good catch (can also r/m tokenverbhandler constant)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 541.patch)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776837#action_12776837 ]
Chris Goffinet commented on CASSANDRA-541:
------------------------------------------
+1
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: 541.patch
patch to add Move and remove tokenupdater.
requires patch from CASSANDRA-536.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 541-2-small-fixes.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777229#action_12777229 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
btw jaakko the traceback you posted was caused by trying to add a new (token, ip) mapping to a BiMap that already had a (oldtoken, ip) entry. I just made update do a remove before a put to fix that.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 0002-fixes.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 0002-fixes.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Comment: was deleted
(was: I think there are two cases in movement:
(I) Node moves to a new position in the ring
(II) Node extends its range by modifying token (special case of (I))
For (I) I think we need to do the following transition: -> LEAVING(oldToken) -> LEFT(oldToken) -> BOOTSTRAPPING (newToken) -> NORMAL(newToken). I don't see any way to cut corners here as we need to stream our old ranges away, signal the end of that operation, and then bootstrap to the new range. This is exactly the same as decommission -> bootsrap. Attached patch modifies StorageService accordingly. We could perhaps jump from LEAVING directly to BOOTSRAPPING, but removing pending ranges reliably becomes slightly fragile as old token is not part of bootstrapping message. Better to signal that clearly by a separate state I think.
(II) is a special case of (I) as the node already has part of its future range. For this case we could use STATE_MOVING to signal that we're not changing our whole range, only adding to it. Since this transition only involves streaming data between the node and its successor, I think NORMAL - MOVING(newToken) -> NORMAL(newToken) would be sufficient. The only nagging worry here is what happens to the data that moves from the successor to the node enlarging its range. If done directly between these two nodes, we might easily violate replica placement rules for this range.
Anyway, since (I) works for all move scenarios (only wastes a lot of energy if (II) would be sufficient), I think we could move that to a separate task and first commit this patchset.
)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Attachment: (was: 0005-fix-pending-ranges-handling-in-node-move.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777238#action_12777238 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
Occasionally I see this exception on a source node. Not sure of the cause.
java.io.IOException: Resource temporarily unavailable
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:418)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:519)
at org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:221)
at org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Attachment: (was: 541-2-small-fixes.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 0001-CASSANDRA-541-add-Move-command.txt)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: (was: 541-v2.patch)
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777254#action_12777254 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
apparently this is a jdk bug. some discussion on http://issues.apache.org/jira/browse/HADOOP-3164 including a workaround
[...] try {
nTransfered = (int) inChannel.transferTo(startPos, len, outChannel);
} catch (IOException e) {
/* at least jdk1.6.0 on Linux seems to throw IOException
* when the socket is full. Hopefully near future verisions will
* handle EAGAIN better. For now look for a specific string in for
* the message for the exception.
*/
if (e.getMessage().startsWith("Resource temporarily unavailable")) {
out.waitForWritable();
continue;
} else {
throw e;
}
}
and a link to fix in jdk7: http://bugs.sun.com/view_bug.do?bug_id=5103988
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777494#action_12777494 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
1) we don't need to know old token since we just remove obsolete ranges now, instead of doing something error-prone based on old token. so LEAVING -> BOOTSTRAPPING is fine.
2) right, I'm deliberately avoiding optimizing this here
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777513#action_12777513 ]
Jaakko Laine commented on CASSANDRA-541:
----------------------------------------
Yeah, we can move from LEAVING to BOOTSTRAPPING, but then we'll need to remove the endpoint from tokenmetadata and remove obsolete pending ranges whenever a node bootstraps. This is exactly what LEFT does, so seems like a nice way to go through that state, but we can of course do that in bootstrapping. The only problem then is that if we always clear tokenmetadata and pending ranges in bootstrap, we cannot distinguish between a legitimate move-bootsrap and a node that enters the cluster using an old IP address and a new token. It is a matter of taste what should be the proper action in this case, but if we clear tokenmetadata, we won't be able to even report an error on this.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777273#action_12777273 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
turnOnInterestOps(key_, SelectionKey.OP_WRITE);
condition_.await();
is probably better than the sleep() i have. untested.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Attachment: 541-2-small-fixes.txt
I'll run more bootstrapping/move tests today, but aren't these two changes (see patch) at least needed?
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777524#action_12777524 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
I see what you are saying -- it will still be correct to go from LEAVE to BS but not as efficient if we do not clear pending ranges first.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt, 0005-fix-pending-ranges-handling-in-node-move.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: 0004-workaround-for-Resource-temporarily-unavailable.txt
0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt
0002-miscellaneous-fixes.txt
0001-CASSANDRA-541-add-Move-command.txt
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-541) allow moving a live node with
data on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777271#action_12777271 ]
Jonathan Ellis commented on CASSANDRA-541:
------------------------------------------
attached workaround.
with these patches move is working for me pretty reliably.
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-miscellaneous-fixes.txt, 0003-fix-for-when-bootstrap-source-has-no-data-in-the-range.txt, 0004-workaround-for-Resource-temporarily-unavailable.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-541:
-----------------------------------
Attachment: 541-2-small-fixes.txt
remove also tokenVerbHandler_
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 541-2-small-fixes.txt, 541-2-small-fixes.txt, 541.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: 0002-fixes.txt
0001-CASSANDRA-541-add-Move-command.txt
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-541) allow moving a live node with data
on it
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-541:
-------------------------------------
Attachment: 0002-fixes.txt
0001-CASSANDRA-541-add-Move-command.txt
> allow moving a live node with data on it
> ----------------------------------------
>
> Key: CASSANDRA-541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-541
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-541-add-Move-command.txt, 0002-fixes.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.