You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org> on 2012/06/20 08:30:43 UTC
[jira] [Created] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Uma Maheswara Rao G created BOOKKEEPER-301:
----------------------------------------------
Summary: Chance of Data loss on sudden poweroff
Key: BOOKKEEPER-301
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.2.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
This is the JIRA to discuss this problem and address the case.
Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397295#comment-13397295 ]
Uma Maheswara Rao G commented on BOOKKEEPER-301:
------------------------------------------------
Some more discussions related to this in Hadoop:
http://lucene.472066.n3.nabble.com/Blocks-are-getting-corrupted-under-very-high-load-tt3527403.html#a3532671
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397402#comment-13397402 ]
Uma Maheswara Rao G commented on BOOKKEEPER-301:
------------------------------------------------
Yes, Falvio. You are right.
In Hadoop, handled like below.
called FileChannel#force after flush.
{code}
+ ((FileOutputStream)out).getChannel().force(true);
{code}
This is an optional. Who ever wants, they can enable it.
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399124#comment-13399124 ]
Uma Maheswara Rao G commented on BOOKKEEPER-301:
------------------------------------------------
Oh, I did not test this API with my experience, But java doc says.
{quote}
* Forces any updates to this channel's file to be written to the storage
* device that contains it.
*
* <p> If this channel's file resides on a local storage device then when
* this method returns it is guaranteed that all changes made to the file
* since this channel was created, or since this method was last invoked,
* will have been written to that device. This is useful for ensuring that
* critical information is not lost in the event of a system crash.
{quote}
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397407#comment-13397407 ]
Uma Maheswara Rao G commented on BOOKKEEPER-301:
------------------------------------------------
Apologies for the typo in your name.:(
Falvio --> Flavio
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397403#comment-13397403 ]
Flavio Junqueira commented on BOOKKEEPER-301:
---------------------------------------------
Note that calling force is still not sufficient. It will return when data is on the write cache, and not when it written on media, unless the write cache is turned off.
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397380#comment-13397380 ]
Flavio Junqueira commented on BOOKKEEPER-301:
---------------------------------------------
I haven't read the thread carefully, but I'm guessing that this is due to losing data in the disk write cache upon a poweroff. Inside Yahoo!, we have recommended the use of BBWC raid controllers. Another option is to disable the disk write buffer, but performance is going to drop.
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-301) Chance of Data loss on sudden
poweroff
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399165#comment-13399165 ]
Flavio Junqueira commented on BOOKKEEPER-301:
---------------------------------------------
Agreed, but that statement is a bit misleading. The force implementation calls fsync, and if you check the documentation of fsync, it says that it forces the flush of OS buffers to the disk device. It doesn't guarantee that the disk device won't cache the data itself. One needs to turn off the disk cache to avoid that, or use kernel-level barriers. I've seen discussions around about the implementation of barriers in the linux kernel not being very safe.
We have verified in lab experiments that you can lose data upon crashes, but way less compared to not forcing at all. I have never seen such a case in a production setting.
> Chance of Data loss on sudden poweroff
> --------------------------------------
>
> Key: BOOKKEEPER-301
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-301
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
>
> Whatever data Bookie flushed into disk may not be gauranteed that they are really pesisted into device. OS will cache the edits. On suddent power off, we may loose that edits.
> This is the JIRA to discuss this problem and address the case.
> Hadoop related issue HDFS-1539.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira