You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2020/07/25 20:58:00 UTC

[jira] [Comment Edited] (CASSANDRA-15191) stop_paranoid disk failure policy is ignored on CorruptSSTableException after node is up

    [ https://issues.apache.org/jira/browse/CASSANDRA-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165117#comment-17165117 ] 

David Capwell edited comment on CASSANDRA-15191 at 7/25/20, 8:57 PM:
---------------------------------------------------------------------

I took a stab at back porting to 3.0: https://github.com/dcapwell/cassandra/tree/CASSANDRA-15191-3.0

I also made a few small changes from your original patch

1) calls to FSError handler now go through jvm stability
2) Instance adds default fs handler.  this was working on trunk since we fixed it there, but didn't backport; so adding it so we don't need in the test


was (Author: dcapwell):
I took a stab at back porting to 3.0: https://github.com/dcapwell/cassandra/commit/fb3162efa1308bc00fd8bd479e91c563160dea0e

I also made a few small changes from your original patch

1) calls to FSError handler now go through jvm stability
2) Instance adds default fs handler.  this was working on trunk since we fixed it there, but didn't backport; so adding it so we don't need in the test
3) CL didn't need a lot of changes, it looks like the issue you faced was a change between 3.0 and 3.11, so only had to update the CL method in jvm stability

> stop_paranoid disk failure policy is ignored on CorruptSSTableException after node is up
> ----------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-15191
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15191
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Config
>            Reporter: Vincent White
>            Assignee: Stefan Miklosovic
>            Priority: Normal
>             Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>         Attachments: log.txt
>
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> There is a bug when disk_failure_policy is set to stop_paranoid and CorruptSSTableException is thrown after server is up. The problem is that this setting is ignored. Normally, it should stop gossip and transport but it just continues to serve requests and an exception is just logged.
>  
> This patch unifies the exception handling in JVMStabilityInspector and code is reworked in such way that this inspector acts as a central place where such exceptions are inspected. 
>  
> The core reason for ignoring that exception is that thrown exception in AbstractLocalAwareExecturorService is not CorruptSSTableException but it is RuntimeException and that exception is as its cause. Hence it is better if we handle this in JVMStabilityInspector which can recursively examine it, hence act accordingly.
> Behaviour before:
> stop_paranoid of disk_failure_policy is ignored when CorruptSSTableException is thrown, e.g. on a regular select statement
> Behaviour after:
> Gossip and transport (cql) is turned off, JVM is still up for further investigation e.g. by jmx.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org