You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kaide Mu (JIRA)" <ji...@apache.org> on 2016/07/27 09:55:20 UTC
[jira] [Comment Edited] (CASSANDRA-12008) Make decommission
operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392776#comment-15392776 ]
Kaide Mu edited comment on CASSANDRA-12008 at 7/27/16 9:54 AM:
---------------------------------------------------------------
Here's 3 operations I'm proposing for STREAMED_RANGES:
- updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Set<Range<Token>> range)
- Set<Range<Token>> getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner)
- void resetAvailableRanges()
Briefly the flow is:
# StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection<Range<Token>> ranges)
# StreamSession.addTransferRanges(String keyspace, Collection<Range<Token>> ranges, Collection<String> columnFamilies, boolean flushTables, long repairedAt)
# StreamSession.addTransferFiles(Collection<SSTableStreamingSections> sstableDetails)
# And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}}
As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges.
WDYT? [~pauloricardomg] [~yukim]
was (Author: kdmu):
Here's 3 operations I'm proposing for STREAMED_RANGES:
- updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Set<Range<Token>> range)
- MultiMap<Range<Token>> getAvailableRanges(String keyspace, IPartitioner partitioner)
- void resetAvailableRanges()
Briefly the flow is:
# StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection<Range<Token>> ranges)
# StreamSession.addTransferRanges(String keyspace, Collection<Range<Token>> ranges, Collection<String> columnFamilies, boolean flushTables, long repairedAt)
# StreamSession.addTransferFiles(Collection<SSTableStreamingSections> sstableDetails)
# And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}}
As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges.
WDYT? [~pauloricardomg] [~yukim]
> Make decommission operations resumable
> --------------------------------------
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Reporter: Tom van der Woerdt
> Assignee: Kaide Mu
> Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and sometimes we need to add or remove nodes. These operations are very dependent on the entire cluster being up, so while we're joining a new node (which sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to become normal or restart
> See 'nodetool help' or 'nodetool help <command>'.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/<ipaddr>] 2016-06-14 18:05:47,275 StreamSession.java:520 - [Stream #<streamid>] Streaming error occurred
> java.net.SocketTimeoutException: null
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_77]
> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_77]
> at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) ~[apache-cassandra-3.0.6.jar:3.0.6]
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)