You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2009/09/17 02:54:57 UTC

[jira] Created: (HDFS-627) Support replica update in datanode

Support replica update in datanode
----------------------------------

                 Key: HDFS-627
                 URL: https://issues.apache.org/jira/browse/HDFS-627
             Project: Hadoop HDFS
          Issue Type: Sub-task
          Components: data-node
    Affects Versions: Append Branch
            Reporter: Tsz Wo (Nicholas), SZE
             Fix For: Append Branch


This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-627:
-------------------------------

    Attachment: updateReplica1.patch

> you may revert ReplicaUnderRecovery.java and ReplicaInfo.java. It seems that the changes are not needed in your patch. 

Done. Thanks, Nicholas!

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090924.patch

h627_20090924.patch: check "finalized" in ReplicaUnderRecovery.toFinalizedReplica(..).

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762419#action_12762419 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-627:
---------------------------------------------

> What was the reason to remove the following codes from h627_20090924.patch?

Oops, the codes are indeed committed but they are not used by HDFS-658.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch, updateReplica2.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-627:
-------------------------------------

    Attachment: updateReplica2.patch

Merged the patch with trunk. TestBlockUnderConstruction does not fail.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch, updateReplica2.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759286#action_12759286 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

It would be nice if the logic for finalizing a block is at FSDataset#finalizeBlock.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759844#action_12759844 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-627:
---------------------------------------------

Hi Hairong, you may revert ReplicaUnderRecovery.java and ReplicaInfo.java.  It seems that the changes are not needed in your patch.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090917.patch

h627_20090917.patch: patch for review.  Will add unit tests.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758418#action_12758418 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

1. A replica needs to be "detached" before it gets truncated so truncation won't modify snapshot. Sorry this is not described in the design document.
2. When finalizing a RUR, we need to make sure that the finalized data/meta files are located in the "finalized" directory. Check FSDataset#finalizeBlock to see how a replica is finalized.
3. Looks that we need to make sure that rbw#getBytesOnDiks() should be equal to rbw#getNumOfBytes() when stopping a writer..

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759037#action_12759037 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

Calling finalizeBlock in the last may not work because the replicaInfo in the replicasMap is already a finalized one.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759330#action_12759330 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-627:
---------------------------------------------

> It would be nice if the logic for finalizing a block is at FSDataset#finalizeBlock.

Yes, but I don't see any easy way to use it.  Could you point out how to use finalizeBlock(..) in the patch?

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090921.patch

h627_20090921.patch: latest patch.  It is not easy to create new unit tests.  Still working on it.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090921b.patch

h627_20090921b.patch: added a test (finally).

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758787#action_12758787 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

> I cannot see any codes in FSDataset#finalizeBlock checking "finalized" directory. Could you give me more hints?
You need to move the replica from directory "rbw" to directory "finalized". This is done by FSVolume#addBlock(Block, File).

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HDFS-627) Support replica update in datanode

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko resolved HDFS-627.
--------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

I just committed this. Thank you Nicholas and Hairong.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch, updateReplica2.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759856#action_12759856 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

Passed all unit tests except for TestBackupNode, the ones in HDFS-647, and TestBlockUnderConstruction. But the last seemed not related to this patch.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759840#action_12759840 ] 

Hairong Kuang commented on HDFS-627:
------------------------------------

     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.


> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090923b.patch

h627_20090923b.patch: use detachFile(..) instead of detachBlock(..)

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-627:
-------------------------------

    Assignee: Tsz Wo (Nicholas), SZE

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch, updateReplica1.patch, updateReplica2.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090922.patch

h627_20090922.patch: only able to fix (1).

> 2. When finalizing a RUR, we need to make sure that the finalized data/meta files are located in the "finalized" directory. Check FSDataset#finalizeBlock to see how a replica is finalized.

I cannot see any codes in FSDataset#finalizeBlock checking "finalized" directory.  Could you give me more hints?

3. Looks that we need to make sure that rbw#getBytesOnDiks() should be equal to rbw#getNumOfBytes() when stopping a writer.. 

Then, it should be fixed in initReplicaRecovery(..), right?

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-627:
-------------------------------

    Attachment: updateReplica.patch

OK, let me give a shot. With so many code changes, I agree it is not easy to figure it out.

This patch has a few changes to Nicholas' the most recent patch. His most recent patch has the right logic. My new patch simply makes it simpler by reusing existing code.
1. use bumpReplicaGS to bump RUR's GS to be recovery id;
2. refactor finalizeReplica that takes a ReplicaInfo to finalize a replica, and use finalizeReplica to finalize the rur;
2. remove the static method updateReplica and change updateReplicaUnderRecovery to non-static.

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch, h627_20090923b.patch, h627_20090924.patch, updateReplica.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-627) Support replica update in datanode

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-627:
----------------------------------------

    Attachment: h627_20090923.patch

Thanks, Hairong.

h627_20090923.patch: calling finalizeBlock(..).

> Support replica update in datanode
> ----------------------------------
>
>                 Key: HDFS-627
>                 URL: https://issues.apache.org/jira/browse/HDFS-627
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: Append Branch
>            Reporter: Tsz Wo (Nicholas), SZE
>             Fix For: Append Branch
>
>         Attachments: h627_20090917.patch, h627_20090921.patch, h627_20090921b.patch, h627_20090922.patch, h627_20090923.patch
>
>
> This is a followed up issue of HDFS-619.  We are going to implement step 4c described in the block recovery algorithm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.