You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Branimir Lambov (JIRA)" <ji...@apache.org> on 2016/04/01 12:57:25 UTC

[jira] [Comment Edited] (CASSANDRA-11448) Running OOS should trigger the disk failure policy

    [ https://issues.apache.org/jira/browse/CASSANDRA-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219524#comment-15219524 ] 

Branimir Lambov edited comment on CASSANDRA-11448 at 4/1/16 10:57 AM:
----------------------------------------------------------------------

Patch here:
|[2.1|https://github.com/blambov/cassandra/tree/11448-2.1]|[utest|http://cassci.datastax.com/job/blambov-11448-2.1-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-2.1-dtest/]|
|[2.2|https://github.com/blambov/cassandra/tree/11448-2.2]|[utest|http://cassci.datastax.com/job/blambov-11448-2.2-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-2.2-dtest/]|
|[3.0|https://github.com/blambov/cassandra/tree/11448-3.0]|[utest|http://cassci.datastax.com/job/blambov-11448-3.0-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-3.0-dtest/]|
|[trunk|https://github.com/blambov/cassandra/tree/11448]|[utest|http://cassci.datastax.com/job/blambov-11448-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-dtest/]|
-2.1 patch applies cleanly up to 3.0 (disregarding {{CHANGES.txt}} conflicts), trunk is different.-
2.1 and 2.2 patches are the same, 3.0 and trunk differ.

Changes the error thrown to {{FSWriteError}}, applies {{JVMStabilityInspector}}, passes it on to the post-flush instead of failing to trigger it, and makes sure commit log is not discarded for failed flushes. Trunk version also removes unused {{DiskAwareRunnable}} and fixes {{null}} return from {{getWriteableLocation}} not being treated properly.


was (Author: blambov):
Patch here:
|[2.1|https://github.com/blambov/cassandra/tree/11448-2.1]|[utest|http://cassci.datastax.com/job/blambov-11448-2.1-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-2.1-dtest/]|
|[2.2|https://github.com/blambov/cassandra/tree/11448-2.2]|[utest|http://cassci.datastax.com/job/blambov-11448-2.2-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-2.2-dtest/]|
|[3.0|https://github.com/blambov/cassandra/tree/11448-3.0]|[utest|http://cassci.datastax.com/job/blambov-11448-3.0-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-3.0-dtest/]|
|[trunk|https://github.com/blambov/cassandra/tree/11448]|[utest|http://cassci.datastax.com/job/blambov-11448-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11448-dtest/]|
2.1 patch applies cleanly up to 3.0 (disregarding {{CHANGES.txt}} conflicts), trunk is different.

Changes the error thrown to {{FSWriteError}}, applies {{JVMStabilityInspector}}, passes it on to the post-flush instead of failing to trigger it, and makes sure commit log is not discarded for failed flushes. Trunk version also removes unused {{DiskAwareRunnable}} and fixes {{null}} return from {{getWriteableLocation}} not being treated properly.

> Running OOS should trigger the disk failure policy
> --------------------------------------------------
>
>                 Key: CASSANDRA-11448
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11448
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Brandon Williams
>            Assignee: Branimir Lambov
>             Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> Currently when you run OOS, this happens:
> {noformat}
> ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047  CassandraDaemon.java:229 - Exception in thread Thread[MemtableFlushWriter:8561,5,main]   java.lang.RuntimeException: Insufficient disk space to write 48 bytes 
>     at org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
>     at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
>     at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:na]
>     at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_66]
>     at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> {noformat}
> Now your flush writer is dead and postflush tasks build up forever.  Instead we should throw FSWE and trigger the failure policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)