You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Benjamin Reed (JIRA)" <ji...@apache.org> on 2010/05/08 01:51:50 UTC
[jira] Created: (HDFS-1137) Name node is using the write-ahead log
improperly
Name node is using the write-ahead log improperly
-------------------------------------------------
Key: HDFS-1137
URL: https://issues.apache.org/jira/browse/HDFS-1137
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Reporter: Benjamin Reed
The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when using WAL, changes are written to the log before they are applied to the state. Currently the Namenode does the WAL after applying the change. This means that read may see changes before they are durable. A client may read information and the server fail before the information is written to the WAL, which results in the client reading state that disappears. To fix the Namenode should write changes before (aka ahead of) applying the change.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1137) Name node is using the write-ahead
log improperly
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865509#action_12865509 ]
dhruba borthakur commented on HDFS-1137:
----------------------------------------
This is by design so that all transactions are not serialized on the edits log. As Todd mentions, a client gets a response only after the transaction has been sync-ed to stable storage. if the sync fails, and this is the only location specified in fs.name.dir, then the NameNode actually shuts down!
so, technically you are right that a client can see a newly file in the namespace even before it is written to the transaction log. and if that write to the transaction log fails, then the client would have seen a file that never will exist. This window should be small.
An alternative would be to first write to the transaction log and then update the in-memory data structures. The performance would be impacted unless to go to a per-inode locking model.
> Name node is using the write-ahead log improperly
> -------------------------------------------------
>
> Key: HDFS-1137
> URL: https://issues.apache.org/jira/browse/HDFS-1137
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Benjamin Reed
>
> The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when using WAL, changes are written to the log before they are applied to the state. Currently the Namenode does the WAL after applying the change. This means that read may see changes before they are durable. A client may read information and the server fail before the information is written to the WAL, which results in the client reading state that disappears. To fix the Namenode should write changes before (aka ahead of) applying the change.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1137) Name node is using the write-ahead
log improperly
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865522#action_12865522 ]
Benjamin Reed commented on HDFS-1137:
-------------------------------------
i think durability guarantees of local file systems are a bit different from lose of local file systems because of how failures happen: if a machine fails, all processes on that machine also fail. In a distributed file system a process can continue running even though the machine hosting the dfs crashes and recovers.
you can support nice durability guarantees and even improve performance if you do the WAL correctly: log the request and do batch syncs before even starting to process the request. you can do this without touching the big name node lock. then after the request is synced to disk you process the requests under the big lock in the order that you synced them. this second part is a purely in-memory operation. i think in the end, the performance would be much better, the design cleaner, and the durability guarantee much easier to understand.
> Name node is using the write-ahead log improperly
> -------------------------------------------------
>
> Key: HDFS-1137
> URL: https://issues.apache.org/jira/browse/HDFS-1137
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Benjamin Reed
>
> The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when using WAL, changes are written to the log before they are applied to the state. Currently the Namenode does the WAL after applying the change. This means that read may see changes before they are durable. A client may read information and the server fail before the information is written to the WAL, which results in the client reading state that disappears. To fix the Namenode should write changes before (aka ahead of) applying the change.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1137) Name node is using the write-ahead
log improperly
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865397#action_12865397 ]
Todd Lipcon commented on HDFS-1137:
-----------------------------------
I always assumed this is entirely on purpose. Because of the coarse grained locking in FSNamesystem, "fixing" this would basically serialize all writes 1:1 with syncs to the edit log, which would drastically decrease write throughput.
We already do sync() before returning to the writer, so any write that the writer thinks is successful is guaranteed to be durable. It's just that other readers may see things that were not made durable.
I think this is perfectly acceptable for a filesystem, and it's exactly what you see in systems like ext3 - writes to the metadata journal are not synced unless you explicitly call fsync(), so a reader can read data which will disappear after a crash.
> Name node is using the write-ahead log improperly
> -------------------------------------------------
>
> Key: HDFS-1137
> URL: https://issues.apache.org/jira/browse/HDFS-1137
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Benjamin Reed
>
> The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when using WAL, changes are written to the log before they are applied to the state. Currently the Namenode does the WAL after applying the change. This means that read may see changes before they are durable. A client may read information and the server fail before the information is written to the WAL, which results in the client reading state that disappears. To fix the Namenode should write changes before (aka ahead of) applying the change.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1137) Name node is using the write-ahead
log improperly
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865563#action_12865563 ]
dhruba borthakur commented on HDFS-1137:
----------------------------------------
@Ben, let's walk through with an example. Suppose a request comes in to create a file foo.txt. Now, we need to insert it into the namespace and we need to ensure that new calls to create the same file should fail. When the first request arrives, we have to insert it into some kind of data structure, so that the next suceeding call detects that a file creation operation for the same file is in progress, isn't it?
> Name node is using the write-ahead log improperly
> -------------------------------------------------
>
> Key: HDFS-1137
> URL: https://issues.apache.org/jira/browse/HDFS-1137
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Reporter: Benjamin Reed
>
> The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when using WAL, changes are written to the log before they are applied to the state. Currently the Namenode does the WAL after applying the change. This means that read may see changes before they are durable. A client may read information and the server fail before the information is written to the WAL, which results in the client reading state that disappears. To fix the Namenode should write changes before (aka ahead of) applying the change.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.