You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Kihwal Lee (JIRA)" <ji...@apache.org> on 2012/09/12 01:06:07 UTC

[jira] [Created] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Kihwal Lee created HDFS-3922:
--------------------------------

             Summary: 0.22 and 0.23 namenode throws away blocks under construction on restart
                 Key: HDFS-3922
                 URL: https://issues.apache.org/jira/browse/HDFS-3922
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 0.20.203.0, 0.22.1
            Reporter: Kihwal Lee
            Assignee: Kihwal Lee
            Priority: Critical
             Fix For: 0.22.1, 0.23.4


When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 

With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455300#comment-13455300 ] 

Kihwal Lee commented on HDFS-3922:
----------------------------------

I manually ran test-patch for 0.23. There were actually no additional javadoc warnings introduced by the patch.

{noformat}
-1 overall.  

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 7 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version ) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.
{noformat}
                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.1, 0.23.3
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HDFS-3922:
----------------------------------

    Fix Version/s:     (was: 0.23.4)
                       (was: 0.22.1)

Kihwal, I just reset Fix Version/s field. It is set when a fix is committed to a branch with the corresponding release number.

If you intend it for certain releases, add those release numbers to Target Version/s field.
                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.203.0, 0.22.1
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kihwal Lee updated HDFS-3922:
-----------------------------

    Attachment: hdfs-3922.branch-023.patch.txt

Posted a patch for branch-0.23.
                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.203.0, 0.22.1
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465578#comment-13465578 ] 

Hudson commented on HDFS-3922:
------------------------------

Integrated in Hadoop-Hdfs-0.23-Build #388 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/388/])
    HDFS-3922. namenode throws away blocks under construction on restart (Kihwal Lee via daryn) (Revision 1391150)

     Result = UNSTABLE
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391150
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java

                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.1, 0.23.3
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 0.23.4
>
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kihwal Lee updated HDFS-3922:
-----------------------------

    Status: Patch Available  (was: Open)
    
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.203.0, 0.22.1
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455209#comment-13455209 ] 

Hadoop QA commented on HDFS-3922:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12545035/hdfs-3922.branch-023.patch.txt
  against trunk revision .

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3184//console

This message is automatically generated.
                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.203.0, 0.22.1
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kihwal Lee updated HDFS-3922:
-----------------------------

     Target Version/s: 0.23.4
    Affects Version/s:     (was: 0.20.203.0)
                       0.23.3
    
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.1, 0.23.3
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3922) 0.22 and 0.23 namenode throws away blocks under construction on restart

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-3922:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.4
     Hadoop Flags: Reviewed
           Status: Resolved  (was: Patch Available)

I've committed to branch-23.  Thanks Kihwal!

I'll let the branch-22 maintainers decide if they want to integrate this change.
                
> 0.22 and 0.23 namenode throws away blocks under construction on restart
> -----------------------------------------------------------------------
>
>                 Key: HDFS-3922
>                 URL: https://issues.apache.org/jira/browse/HDFS-3922
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.1, 0.23.3
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 0.23.4
>
>         Attachments: hdfs-3922.branch-023.patch.txt
>
>
> When reading edits on startup, namenode may throw away blocks under construction. This is because the file inode is turned into a "under construction" one, but nothing is done to the last block. 
> With append/hsync, this is not acceptable because it may drop sync'ed partial blocks.  In branch 2 and trunk, HDFS-1623 (HA) fixed this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira