[jira] [Commented] (CASSANDRA-12509) Shutdown process triggered twice during if the node is drained

["Stefania (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534961#comment-15534961 ] 

Stefania commented on CASSANDRA-12509:
--------------------------------------

Unfortunately the 3.0 patch has broken [test_archive_commitlog_with_active_commitlog|http://cassci.datastax.com/job/ifesdjeen-12509-3.0-dtest/lastCompletedBuild/testReport/snapshot_test/TestArchiveCommitlog/test_archive_commitlog_with_active_commitlog_2/] in 3.0. It's a known failure, CASSANDRA-11811, but whilst it currently fails rarely, this patch makes it fail every single time, even locally.

The problem is that an archived commitlog segment from a destroyed cluster is replayed, which adds the bootstrap tokens of the old cluster, but subsequently new bootstrap tokens are generated and the node fails to start because it thinks it has twice as many tokens.

I've proposed this [pull request|https://github.com/riptano/cassandra-dtest/pull/1354] to fix the problem, which prevents the new cluster from generating new tokens, let's see if it gets accepted.

> Shutdown process triggered twice during if the node is drained
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-12509
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12509
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alex Petrov
>            Assignee: Alex Petrov
>
> If the node is drained, the {{StorageService#drain}} [method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L4212] is called, which triggers shutdown of mutation stage, messaging service, compaction, batchlog etc. In the end of this process, the node is moved to {{DRAINED}} status with the process still running. 
> When JVM is shutdown, the JVM shutdown hooks are ran, which are subscribed during the server initialisation: {{Runtime.getRuntime().addShutdownHook(drainOnShutdown);}} [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L575-L636]. 
> I noticed this behaviour while reviewing [CASSANDRA-12461], as if we'd like add custom pre and post-shutdown hooks, most likely it makes sense to run them once (or user might expect such behaviour). 
> Is this behaviour correct? Should we run whole shutdown process twice or just once in "drain" and no-op during JVM shutdown?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)