You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kirk True (JIRA)" <ji...@apache.org> on 2012/10/22 20:12:12 UTC
[jira] [Created] (CASSANDRA-4847) Bad disk causes death of node
despite disk_failure_policy
Kirk True created CASSANDRA-4847:
------------------------------------
Summary: Bad disk causes death of node despite disk_failure_policy
Key: CASSANDRA-4847
URL: https://issues.apache.org/jira/browse/CASSANDRA-4847
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Kirk True
Assignee: Kirk True
Steps:
# Create a bad disk via device mapper
# Specify good disk and bad disk is data directory
# Set {{disk_failure_policy}} to {{best_effort}} in cassandra.yaml
# Start node
Expected:
Attempts to create system directories to fail (as expected) on bad disk, and have it added to blacklisted directories.
Actual:
Node start up aborts due to uncaught error:
{noformat}
FSWriteError in /mnt/bad_disk/system_traces/sessions
at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:258)
at org.apache.cassandra.db.Directories.<init>(Directories.java:104)
at org.apache.cassandra.db.Directories.create(Directories.java:90)
at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:404)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:393)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:436)
Caused by: java.io.IOException: Failed to mkdirs /mnt/bad_disk/system_traces/sessions
... 7 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4847) Bad disk causes death of node
despite disk_failure_policy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496254#comment-13496254 ]
Jonathan Ellis commented on CASSANDRA-4847:
-------------------------------------------
I do think we should respect the policy wrt {{stop}}. But I would agree that {{best_effort}} and {{ignore}} should both blacklist and move on here.
> Bad disk causes death of node despite disk_failure_policy
> ---------------------------------------------------------
>
> Key: CASSANDRA-4847
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4847
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Kirk True
> Assignee: Kirk True
>
> Steps:
> # Create a bad disk via device mapper
> # Specify good disk and bad disk is data directory
> # Set {{disk_failure_policy}} to {{best_effort}} in cassandra.yaml
> # Start node
> Expected:
> Attempts to create system directories to fail (as expected) on bad disk, and have it added to blacklisted directories.
> Actual:
> Node start up aborts due to uncaught error:
> {noformat}
> FSWriteError in /mnt/bad_disk/system_traces/sessions
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:258)
> at org.apache.cassandra.db.Directories.<init>(Directories.java:104)
> at org.apache.cassandra.db.Directories.create(Directories.java:90)
> at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:404)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:393)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:436)
> Caused by: java.io.IOException: Failed to mkdirs /mnt/bad_disk/system_traces/sessions
> ... 7 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4847) Bad disk causes death of node
despite disk_failure_policy
Posted by "Kirk True (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495960#comment-13495960 ]
Kirk True commented on CASSANDRA-4847:
--------------------------------------
I'd like to know if dying on startup with a bad disk regardless of disk_failure_policy setting is desired behavior. I'd lean toward it being 'no' from a pragmatic standpoint, but would like to know if there's a precedent against _simply_ black-listing the bad disk and moving on. I'd like to fix it from a robustness POV.
> Bad disk causes death of node despite disk_failure_policy
> ---------------------------------------------------------
>
> Key: CASSANDRA-4847
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4847
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Kirk True
> Assignee: Kirk True
>
> Steps:
> # Create a bad disk via device mapper
> # Specify good disk and bad disk is data directory
> # Set {{disk_failure_policy}} to {{best_effort}} in cassandra.yaml
> # Start node
> Expected:
> Attempts to create system directories to fail (as expected) on bad disk, and have it added to blacklisted directories.
> Actual:
> Node start up aborts due to uncaught error:
> {noformat}
> FSWriteError in /mnt/bad_disk/system_traces/sessions
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:258)
> at org.apache.cassandra.db.Directories.<init>(Directories.java:104)
> at org.apache.cassandra.db.Directories.create(Directories.java:90)
> at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:404)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:393)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:436)
> Caused by: java.io.IOException: Failed to mkdirs /mnt/bad_disk/system_traces/sessions
> ... 7 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira