You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Aaron T. Myers (Created) (JIRA)" <ji...@apache.org> on 2012/01/06 02:45:39 UTC
[jira] [Created] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Pre-allocate HDFS edit log files after writing version number
-------------------------------------------------------------
Key: HDFS-2759
URL: https://issues.apache.org/jira/browse/HDFS-2759
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the standby to read an invalid version. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195629#comment-13195629 ]
Hadoop QA commented on HDFS-2759:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512335/HDFS-2759.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestFSInputChecker
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//console
This message is automatically generated.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195755#comment-13195755 ]
Hudson commented on HDFS-2759:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #973 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/973/])
HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195093#comment-13195093 ]
Aaron T. Myers commented on HDFS-2759:
--------------------------------------
bq. So I think we're OK here.
Awesome. I agree with your analysis.
bq. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?
I bet I can dig that up and will be happy to run it again with this patch.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2759:
------------------------------
Fix Version/s: (was: 2.0.0)
2.0.1
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 2.0.1
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195092#comment-13195092 ]
Todd Lipcon commented on HDFS-2759:
-----------------------------------
bq. The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync.
This isn't quite true - the difference between {{force(true)}} and {{force(false)}} is that {{force(false)}} calls {{fdatasync()}}. The man page says:
{quote}
fdatasync() is similar to fsync(), but does not flush modified metadata
*unless that metadata is needed in order to allow a subsequent data*
*retrieval to be correctly handled*. For example, changes to st_atime or
st_mtime (respectively, time of last access and time of last modifica‐
tion; see stat(2)) do not require flushing because they are not neces‐
sary for a subsequent data read to be handled correctly. *On the other*
*hand, a change to the file size* (st_size, as made by say ftruncate(2)),
*would require a metadata flush*.
{quote}
(emphasis mine)
So I think we're OK here.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Description: In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point. (was: In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the standby to read an invalid version. We should write the header, then pre-allocate the rest of the file after this point.)
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195748#comment-13195748 ]
Hudson commented on HDFS-2759:
------------------------------
Integrated in Hadoop-Hdfs-trunk #940 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/940/])
HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193453#comment-13193453 ]
Aaron T. Myers commented on HDFS-2759:
--------------------------------------
Moved this to a top-level issue, since it's not specific to HA.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Attachment: HDFS-2759.patch
Here's a patch which addresses the issue.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just went ahead and committed this to trunk since the difference between the reviewed patch and the latest one is just a small comment change.
Thanks a lot for the review, Todd.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Component/s: ha
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the standby to read an invalid version. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195620#comment-13195620 ]
Hudson commented on HDFS-2759:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1613 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1613/])
HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Issue Type: Bug (was: Sub-task)
Parent: (was: HDFS-1623)
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195622#comment-13195622 ]
Hudson commented on HDFS-2759:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1685 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1685/])
HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Attachment: HDFS-2759.patch
bq. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?
I just re-ran the benchmark, and saw no performance regressions. In fact, pre-allocation ops and the slowest 0.01% of ops seem to have actually sped up a little bit, since we can now return from the flushAndSync call without waiting for the pre-allocation data to be synced to disk.
Here's a (very slightly) updated patch which is identical to the last, except that it also fixes up an incorrect comment justifying why we don't need to force the syncing of metadata when logging an op.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195074#comment-13195074 ]
Todd Lipcon commented on HDFS-2759:
-----------------------------------
This seems reasonable. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Target Version/s: 0.24.0
Affects Version/s: (was: HA branch (HDFS-1623))
0.24.0
Status: Patch Available (was: Open)
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194020#comment-13194020 ]
Aaron T. Myers commented on HDFS-2759:
--------------------------------------
The javadoc and findbugs warnings are unrelated to this patch.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195623#comment-13195623 ]
Hudson commented on HDFS-2759:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1629 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1629/])
HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2759.patch, HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193686#comment-13193686 ]
Hadoop QA commented on HDFS-2759:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511956/HDFS-2759.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 21 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//console
This message is automatically generated.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193501#comment-13193501 ]
Hadoop QA commented on HDFS-2759:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511911/HDFS-2759.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 21 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//console
This message is automatically generated.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2759) Pre-allocate HDFS edit log files
after writing version number
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193466#comment-13193466 ]
Aaron T. Myers commented on HDFS-2759:
--------------------------------------
Sorry, I should have mentioned what I did for testing. No tests are included since writing a test seems like it would require adding a test hook in the EditLogFileOutputStream#flushAndSync method, which seems undesirable since that method is highly performance-sensitive. I verified that the bug exists by adding a "System.exit(0)" after pre-allocation, but before calling FileChannel#force. This indeed resulted in a file full of FF, which if an NN were to try to read would be interpreted as an invalid header version number. By moving the pre-allocation after the call to FileChannel#force, we guarantee that the valid data will hit the edit log before writing FFs for the purpose of pre-allocation.
In the course of preparing and testing this patch, I think I've discovered another potential issue, though. Note that in the call to FileChannel#force, we pass "false" which indicates that FileChannel#force does not need to wait for metadata of the file to be synced to disk, only the data. I'm not sure of the precise semantics of not syncing metadata. In particular, is the file length included? If not, I think this has the potential to cause some edits to not actually be read back from disk after a crash.
The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync.
Do people agree this is a potential problem? If so, I can either fix it here, or file a new JIRA.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2759) Pre-allocate HDFS edit log files after
writing version number
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2759:
---------------------------------
Attachment: HDFS-2759.patch
A few asserts in tests needed to be updated since a freshly pre-allocated log file is now 4 bytes longer, since pre-allocation happens after writing the 4-byte header.
> Pre-allocate HDFS edit log files after writing version number
> -------------------------------------------------------------
>
> Key: HDFS-2759
> URL: https://issues.apache.org/jira/browse/HDFS-2759
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-2759.patch, HDFS-2759.patch
>
>
> In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira