You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "John Vines (Created) (JIRA)" <ji...@apache.org> on 2011/10/27 22:08:32 UTC
[jira] [Created] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Logger should fail into read-only mode with disk full
-----------------------------------------------------
Key: ACCUMULO-84
URL: https://issues.apache.org/jira/browse/ACCUMULO-84
Project: Accumulo
Issue Type: Improvement
Components: logger
Affects Versions: 1.3.5, 1.4.0, 1.5.0
Reporter: John Vines
Assignee: Eric Newton
Fix For: 1.5.0
If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-84:
---------------------------------
Affects Version/s: (was: 1.5.0)
> Logger should fail into read-only mode with disk full
> -----------------------------------------------------
>
> Key: ACCUMULO-84
> URL: https://issues.apache.org/jira/browse/ACCUMULO-84
> Project: Accumulo
> Issue Type: Improvement
> Components: logger
> Affects Versions: 1.3.5, 1.4.0
> Reporter: John Vines
> Assignee: Eric Newton
> Labels: failover
> Fix For: 1.5.0
>
>
> If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Posted by "Adam Fuchs (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151443#comment-13151443 ]
Adam Fuchs commented on ACCUMULO-84:
------------------------------------
1. The table should not be damaged by the logger failing. The way it currently works is that the wal write will fail, the tserver will try to find another logger, and if no logger can be successfully written to then the write will retry forever (I believe). This behavior should be preserved.
2. Administrator intervention to get a logger out of read only mode is fine. This should be rare, and the failure modes of trying to recover automatically could be complex and hard to predict. There should be a periodic notification via the warning log that the logger is in read only mode so that we know to fix it. This should propagate up to the monitor page.
> Logger should fail into read-only mode with disk full
> -----------------------------------------------------
>
> Key: ACCUMULO-84
> URL: https://issues.apache.org/jira/browse/ACCUMULO-84
> Project: Accumulo
> Issue Type: Improvement
> Components: logger
> Affects Versions: 1.3.5, 1.4.0, 1.5.0
> Reporter: John Vines
> Assignee: Eric Newton
> Labels: failover
> Fix For: 1.5.0
>
>
> If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Posted by "Eric Newton (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137496#comment-13137496 ]
Eric Newton commented on ACCUMULO-84:
-------------------------------------
This is especially true now that the Logger uses multiple disks (1.4).
> Logger should fail into read-only mode with disk full
> -----------------------------------------------------
>
> Key: ACCUMULO-84
> URL: https://issues.apache.org/jira/browse/ACCUMULO-84
> Project: Accumulo
> Issue Type: Improvement
> Components: logger
> Affects Versions: 1.3.5, 1.4.0, 1.5.0
> Reporter: John Vines
> Assignee: Eric Newton
> Labels: failover
> Fix For: 1.5.0
>
>
> If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Posted by "Jim Zombek (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151386#comment-13151386 ]
Jim Zombek commented on ACCUMULO-84:
------------------------------------
A couple of requirement questions.
1. If the Cloudbase WAL fails to log the mutation (e.g. out of disk space), the requirement is for the WAL to continue in "READ-ONLY" mode as opposed to stopping as it does now. By continuing without WAL logging, can the Table still be recovered?
2. After the WAL enters READ-ONLY mode, is the requirement to start logging again once disk space becomes available? Or is the requirement that once the WAL enters READ-ONLY mode, it will always be in READ-ONLY mode until it is stopped and restarted?
JimZ
> Logger should fail into read-only mode with disk full
> -----------------------------------------------------
>
> Key: ACCUMULO-84
> URL: https://issues.apache.org/jira/browse/ACCUMULO-84
> Project: Accumulo
> Issue Type: Improvement
> Components: logger
> Affects Versions: 1.3.5, 1.4.0, 1.5.0
> Reporter: John Vines
> Assignee: Eric Newton
> Labels: failover
> Fix For: 1.5.0
>
>
> If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-84) Logger should fail into read-only
mode with disk full
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151409#comment-13151409 ]
Keith Turner commented on ACCUMULO-84:
--------------------------------------
I think when the logger switches to read only, the tablet server should get an error when trying to write to it. At that point the tablet server should switch to another logger. If it can not find any logger, it will keep trying to find a logger and writes will be held.
I think a requirement that the logger leaves READ-ONLY mode and enters READ-WRITE mode once disk space becomes available is reasonable and desirable.
> Logger should fail into read-only mode with disk full
> -----------------------------------------------------
>
> Key: ACCUMULO-84
> URL: https://issues.apache.org/jira/browse/ACCUMULO-84
> Project: Accumulo
> Issue Type: Improvement
> Components: logger
> Affects Versions: 1.3.5, 1.4.0, 1.5.0
> Reporter: John Vines
> Assignee: Eric Newton
> Labels: failover
> Fix For: 1.5.0
>
>
> If the disk is full for a Logger it will kill itself quickly. Instead we should have it fail into a 'read-only' mode, that way tservers can still do log recovery based on the data it has. It needs to be treated in such a way that tservers won't rely on it to log their new data, but can be used to recover older data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira