You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paulo Motta (JIRA)" <ji...@apache.org> on 2016/02/19 23:32:18 UTC
[jira] [Comment Edited] (CASSANDRA-8343) Secondary index creation
causes moves/bootstraps to fail
[ https://issues.apache.org/jira/browse/CASSANDRA-8343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154892#comment-15154892 ]
Paulo Motta edited comment on CASSANDRA-8343 at 2/19/16 10:31 PM:
------------------------------------------------------------------
Thanks for the input [~slebresne]!
I created a simpler version of the patch that basically increases the incoming socket timeout on the sending side to {{3 * streaming_socket_timeout}} when it reaches the {{WAIT_COMPLETE}} state. I also close the outgoing message handler on the sender side after the "complete" message is sent, and similarly close the incoming message handler on the receiving side after receiving the "complete" message since they are no longer necessary.
This should give the reader more time (3 hours with current default {{streaming_socket_timeout}} of 1 hour) to process the received data, rebuild indexes, etc. If even this larger timeout is reached, the session is failed a message is logged asking the user to increase the value of {{streaming_socket_timeout}} (as suggested by Sylvain) and the stream session is failed.
I think this should be a good enough approach for the time being and we can revisit adding a new {{KeepAlive}} message if necessary when bumping the streaming message version later. We could also revisit that in the context of CASSANDRA-8621 (streaming retry on socket timeout).
Below is new patch and tests:
||2.2||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-8343]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:8343]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8343-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8343-dtest/lastCompletedBuild/testReport/]|
WDYT [~yukim]?
was (Author: pauloricardomg):
Thanks for the input [~slebresne]!
I created a simpler version of the patch that does not involve adding a new message to the streaming protocol but may tolerate an index rebuild larger than {{streaming_socket_timeout}}.
The basic idea is to tolerate up to 3 socket timeouts on the sender side if it's on the {{WAIT_COMPLETE}} state, giving a total of {{3*streaming_socket_timeout}} for the receiver to process the data which would be 3 hours with the current default, which IMO should be sufficient for most scenarios. In case there are more than 3 consecutive socket timeouts an informative message is logged asking the user to increase the value of {{streaming_socket_timeout}} (as suggested by Sylvain) and the stream session is failed. Below is an example of this new approach:
{noformat}
debug.log:DEBUG [STREAM-OUT-/127.0.0.2] 2016-02-19 18:00:09,947 ConnectionHandler.java:351 - [Stream #c2b5c6c0-d74b-11e5-a490-dfb88388eec7] Sending Complete
debug.log:DEBUG [STREAM-IN-/127.0.0.2] 2016-02-19 18:00:10,948 StreamSession.java:221 - [Stream #c2b5c6c0-d74b-11e5-a490-dfb88388eec7] Received socket timeout but will ignore it because stream session is on WAIT_COMPLETE state, so other peer might be still processing received data beforereplying to this node. Ignored 1 out of 3 times.
debug.log:DEBUG [STREAM-IN-/127.0.0.2] 2016-02-19 18:00:11,948 StreamSession.java:221 - [Stream #c2b5c6c0-d74b-11e5-a490-dfb88388eec7] Received socket timeout but will ignore it because stream session is on WAIT_COMPLETE state, so other peer might be still processing received data beforereplying to this node. Ignored 2 out of 3 times.
debug.log:DEBUG [STREAM-IN-/127.0.0.2] 2016-02-19 18:00:12,096 ConnectionHandler.java:273 - [Stream #c2b5c6c0-d74b-11e5-a490-dfb88388eec7] Received Complete
debug.log:DEBUG [STREAM-IN-/127.0.0.2] 2016-02-19 18:00:12,096 ConnectionHandler.java:112 - [Stream #c2b5c6c0-d74b-11e5-a490-dfb88388eec7] Closing stream connection handler on /127.0.0.2
{noformat}
IMO this should be a good enough approach for the time being and we can revisit adding a new {{KeepAlive}} message if necessary when bumping the streaming message version later. We could also revisit that in the context of CASSANDRA-8621 (streaming retry on socket timeout).
I also closed the outgoing message handler on the sender side after the "complete" message is sent, and similarly closed the incoming message handler on the receiving side after receiving the "complete" message since they are no longer necessary.
Below is new patch and tests:
||2.2||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-8343]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:8343]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8343-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8343-dtest/lastCompletedBuild/testReport/]|
WDYT of this approach [~yukim]?
> Secondary index creation causes moves/bootstraps to fail
> --------------------------------------------------------
>
> Key: CASSANDRA-8343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8343
> Project: Cassandra
> Issue Type: Bug
> Reporter: Michael Frisch
> Assignee: Paulo Motta
>
> Node moves/bootstraps are failing if the stream timeout is set to a value in which secondary index creation cannot complete. This happens because at the end of the very last stream the StreamInSession.closeIfFinished() function calls maybeBuildSecondaryIndexes on every column family. If the stream time + all CF's index creation takes longer than your stream timeout then the socket closes from the sender's side, the receiver of the stream tries to write to said socket because it's not null, an IOException is thrown but not caught in closeIfFinished(), the exception is caught somewhere and not logged, AbstractStreamSession.close() is never called, and the CountDownLatch is never decremented. This causes the move/bootstrap to continue forever until the node is restarted.
> This problem of stream time + secondary index creation time exists on decommissioning/unbootstrap as well but since it's on the sending side the timeout triggers the onFailure() callback which does decrement the CountDownLatch leading to completion.
> A cursory glance at the 2.0 code leads me to believe this problem would exist there as well.
> Temporary workaround: set a really high/infinite stream timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)