You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/07/13 01:16:35 UTC
[jira] [Created] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Todd Lipcon created HDFS-3652:
---------------------------------
Summary: 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
Key: HDFS-3652
URL: https://issues.apache.org/jira/browse/HDFS-3652
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.3, 1.1.0, 1.2.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
{code}
File parentDir = getStorageDirForStream(idx);
if (parentDir.getName().equals(sd.getRoot().getName())) {
{code}
... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413346#comment-13413346 ]
Aaron T. Myers commented on HDFS-3652:
--------------------------------------
+1, the patch looks good to me. Great find/fix, Todd.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3652:
------------------------------
Attachment: hdfs-3652.txt
Attached patch is for branch-1.
I modified the existing storage dir failure test so that all of the name dirs have the same name, and it started to fail. After fixing the bug, it passes.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-3652:
-----------------------------
Target Version/s: 1.0.4, 1.1.0 (was: 1.0.4, 1.1.0, 1.2.0)
Affects Version/s: (was: 1.2.0)
Fix Version/s: (was: 1.2.0)
since 1.2.0 is unreleased, it is sufficient to state it is fixed in 1.1.0.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 1.0.4, 1.1.0
>
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413456#comment-13413456 ]
Eli Collins commented on HDFS-3652:
-----------------------------------
+1 lgtm as well. Nice find.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413326#comment-13413326 ]
Todd Lipcon commented on HDFS-3652:
-----------------------------------
This has data-loss implications as well. I am able to reproduce the following:
- NN is writing to three dirs: /data/1/nn, /data/2/nn, and /data/3/nn
- I modified the NN to inject an IOException when creating "edits.new" in /data/3/nn, which causes "removeEditsForStorageDir" to get called inside {{rollEditLog}}
- Upon triggering a checkpoint:
-- all three logs are closed successfully
-- /data/1/nn and /data/2/nn are successfully opened for "edits.new"
-- /data/3/nn throws an IOE which gets caught. This calls {{removeEditsForStorageDir}}, which removes the wrong stream (augmented logging):
{code}
12/07/12 16:23:54 INFO namenode.FSNamesystem: Roll Edit Log from 127.0.0.1
12/07/12 16:23:54 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 0 0
12/07/12 16:23:54 WARN namenode.FSNamesystem: Removing edits stream /tmp/name1/nn/current/edits.new
12/07/12 16:23:54 WARN common.Storage: Removing storage dir /tmp/name3/nn
java.io.IOException: Injected fault for /tmp/name3/nn/current/edits.new
at org.apache.hadoop.hdfs.server.namenode.FSEditLog$EditLogFileOutputStream.<init>(FSEditLog.java:146)
{code}
- The NN is now _only_ writing to /tmp/name2/nn/current/edits.new, but considers both name1 and name2 to be good from a storage-directory standpoint. However, {{/tmp/name1/nn/current/edits.new}} exists as an empty edit log file (just the header and preallocated region of 0xffs)
- When {{rollFSImage}} is called, it successfully calls {{close}} only on the name2 log - which truncates it to the correct transaction boundary. Then it renames both {{name2/.../edits.new}} and {{name1/.../edits.new}} to {{edits}}, and opens them both for append (assuming they've been truncated to a transaction boundary).
- The NN is now writing to name1 and name2, but name1's log looks like this:
{code}
<valid header> <preallocated bytes of 0xffffffffffff.....> <transactions>
{code}
- Upon the next checkpoint, the 2NN will likely download this log, since it's listed first in the name directory list. Upon doing so, it will see the 0xff at the head of the log and not read any of the edits (which come after all of the 0xffs)
- The 2NN then uploads the "merged" image back to the NN, which blows away the "edits" file. Thus, its in-memory data has gotten out of sync with the disk data, and the next time a checkpoint occurs or the NN restarts, it will fail.
This is not an issue in trunk since the code was largely rewritten by HDFS-1073.
The workaround for existing users is simple: rename the directories to eg /data/1/nn1 and /data/2/nn2. The fix is also simple. I will upload the fix this afternoon.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3652:
------------------------------
Target Version/s: 1.0.4, 1.1.0, 1.2.0 (was: 1.0.3, 1.1.0)
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413320#comment-13413320 ]
Matt Foley commented on HDFS-3652:
----------------------------------
Urk! Quite a catch. When patch available, please commit to branch-1.0 as well as branch-1.1 and branch-1.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-3652:
-----------------------------
Fix Version/s: (was: 1.1.0)
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 1.0.4
>
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-3652:
-----------------------------
Target Version/s: 1.0.4 (was: 1.0.4, 1.1.0)
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 1.0.4
>
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3652) 1.x: FSEditLog failure removes the
wrong edit stream when storage dirs have same name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-3652.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
1.1.0
1.0.4
Hadoop Flags: Reviewed
Committed to branch-1, branch-1.0, and branch-1.1.
> 1.x: FSEditLog failure removes the wrong edit stream when storage dirs have same name
> -------------------------------------------------------------------------------------
>
> Key: HDFS-3652
> URL: https://issues.apache.org/jira/browse/HDFS-3652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.3, 1.1.0, 1.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 1.0.4, 1.1.0, 1.2.0
>
> Attachments: hdfs-3652.txt
>
>
> In {{FSEditLog.removeEditsForStorageDir}}, we iterate over the edits streams trying to find the stream corresponding to a given dir. To check equality, we currently use the following condition:
> {code}
> File parentDir = getStorageDirForStream(idx);
> if (parentDir.getName().equals(sd.getRoot().getName())) {
> {code}
> ... which is horribly incorrect. If two or more storage dirs happen to have the same terminal path component (eg /data/1/nn and /data/2/nn) then it will pick the wrong stream(s) to remove.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira