You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2011/02/01 00:41:29 UTC

[jira] Created: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Provide a stronger data guarantee in the write pipeline
-------------------------------------------------------

                 Key: HDFS-1606
                 URL: https://issues.apache.org/jira/browse/HDFS-1606
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: data-node, hdfs client
            Reporter: Tsz Wo (Nicholas), SZE


In the current design, if there is a datanode/network failures in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may not be accurate under erroneous conditions.

We propose to add a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992909#comment-12992909 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

> 1. Find a datanode D by some means.

I have checked the code.  This is easier than I expect since {{BlockPlacementPolicy}} is able to find an additional datanode, given a list of chosen datanodes.  The remaining work of this part is to add a new method to {{ClientProtocol}} so that {{DFSClient}} could use it.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016472#comment-13016472 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

- In [build #318|https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/318//testReport/org.apache.hadoop.hdfs/TestMultiThreadedHflush/testHflushWhileClosing/],
{noformat}
java.lang.NullPointerException
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.access$2500(DFSOutputStream.java:283)
	at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1470)
	at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:110)
	at org.apache.hadoop.hdfs.TestMultiThreadedHflush$1.run(TestMultiThreadedHflush.java:156)
{noformat}
There are some existing synchronization problem in {{DFSOutputStream}}.  It is possible the call {{hflush()}} after {{close()}} without getting any error.
I will simply check null in this patch.  Will think about the synchronization problem after that.

- For the other failed tests, it is simply not enough datanodes so that addDatanode failed.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991321#comment-12991321 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

Below are some steps for adding a datanode to a pipeline:
# Find a datanode _D_ by some means.
# Add _D_ to the existing pipeline.
# Continue writing.
# Close the pipeline.

\\
Moreover, we have to:
\(*) Transfer the existing data to _D_.

The question is where to put \(*)?

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992392#comment-12992392 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

> In fact, if we can have a system-wide config ...
Will do.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017105#comment-13017105 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475735/h1606_20110407.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 25 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.server.datanode.TestBlockReport

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/325//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/325//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/325//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110217.patch

h1606_20110217.patch: changed DFSClient to get additional datanodes from the NameNode.

Remaining items:
- Transfer a RBW from a datanode to another datanode.
- Add new unit tests.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016297#comment-13016297 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475554/h1606_20110405b.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.server.datanode.TestBlockReport
                  org.apache.hadoop.hdfs.TestFileAppend2
                  org.apache.hadoop.hdfs.TestFileAppend4
                  org.apache.hadoop.hdfs.TestLargeBlock
                  org.apache.hadoop.hdfs.TestMultiThreadedHflush
                  org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/319//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/319//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/319//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110228.patch

h1606_20110228.patch: Seems to have everything except new tests.

Will separate step \(*) to HDFS-1675 for easing the review.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017675#comment-13017675 ] 

Jitendra Nath Pandey commented on HDFS-1606:
--------------------------------------------

1. The method findNewDatanode should return all new datanodes in case there are more than one new datanode.

2. The method addDatanode2ExistingPipeline can be split into following methods
    a) method to check if transfer is needed.
    b) method to get additional datanodes and determine source and destination
    c) method that does actual transfer.

3. DataStreamer#hflush : Should we change it to setHflush(boolean val) to clarify its just setting a flag?

4. Does it make sense to add a unit test for default ReplaceDatanodeOnFailure policy? 

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110407b.patch

I should mention earlier that I cannot reproduce the failure locally.  So I submit to Jenkins multiple times.

h1606_20110407b.patch: don't add datanode when closing the block.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110407c.patch

h1606_20110407c.patch: fixed some bugs in the fault injection tests.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110405.patch

h1606_20110405.patch: fixed bugs, warnings.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018941#comment-13018941 ] 

Hudson commented on HDFS-1606:
------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #588 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/588/])
    HDFS-1606. Provide a stronger data guarantee in the write pipeline by adding a new datanode when an existing datanode failed.


> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016243#comment-13016243 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475550/h1606_20110405.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.server.datanode.TestBlockReport
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/317//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/317//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/317//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110404.patch

h1606_20110404.patch: updated with trunk and added a unit test.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053999#comment-13053999 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

Koji Noguchi also has provided a lot of inputs on this.  Sorry that I failed to mention it in the [acknowledgement|https://issues.apache.org/jira/browse/HDFS-1606?focusedCommentId=13018958&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13018958].

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018958#comment-13018958 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

h3. Acknowledgement
This work cannot be done without the helps and many discussions of Hadoop contributors including (in alphabetical order) Dhruba Borthakur, Hairong Kuang, Owen O'Malley, Jitendra Pandey, Sanjay Radia, Sriram Rao, Suresh Srinivas and Kan Zhang.  I am simply the programmer who has done the implementation of the great ideas provided by them.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

      Resolution: Fixed
    Release Note: Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy.  Added a new feature to replace datanode on failure in DataTransferProtocol.  Added getAdditionalDatanode(..) in ClientProtocol.
    Hadoop Flags: [Incompatible change, Reviewed]
          Status: Resolved  (was: Patch Available)

Thanks Jitendra for the review.

I have committed this.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110408b.patch

Thanks Jitendra for the review.

h1606_20110408b.patch:

> 1. The method findNewDatanode should return all new datanodes ...

Since it is only an internal method but not protocol or public API, we may easily change it later when we add the multiple destination feature.


> 2. The method addDatanode2ExistingPipeline can be split ...

I only split actual transfer out.  The remaining codes only has 20 lines excluding comments.

> 3. DataStreamer#hflush : Should we change it to setHflush(boolean val) to clarify its just setting a flag?

Changed.

> 4. Does it make sense to add a unit test for default ReplaceDatanodeOnFailure policy?

Added {{testDefaultPolicy()}}.


> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991839#comment-12991839 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

A straightforward approach is to
{panel}
\(i) start \(\*) right after #1 and stall #2 until \(*) is done. 
{panel}

If we feel comfortable, we may
{panel}
(ii) start \(*) right after #1 in a separated thread and start #2 concurrently.  Once #3 is done, join the thread and then combine the old data with the new data before #4.
{panel}

Depending on the block size, a partial block may have several hundreds megabytes.  So \(*) is an expensive operation which may potentially take a long time (in the order of seconds).  (ii) has a lower latency but \(i) is a simpler solution.  How about we have \(i) in the first implementation and have (ii) as a future improvement?

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022432#comment-13022432 ] 

Hudson commented on HDFS-1606:
------------------------------

Integrated in Hadoop-Hdfs-trunk #643 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/])
    

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992390#comment-12992390 ] 

dhruba borthakur commented on HDFS-1606:
----------------------------------------

In fact, if we can have  a system-wide config on whether to trigger this behaviour or not, that will be great.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Component/s: name-node

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch, h1606_20110211.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

        Fix Version/s: 0.23.0
    Affects Version/s: 0.23.0
               Status: Patch Available  (was: Open)

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992347#comment-12992347 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

h5. When to add a datanode?
Since adding a datanode to an existing pipeline is an expensive operation (see [the previoius comment|https://issues.apache.org/jira/browse/HDFS-1606?focusedCommentId=12991839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12991839]), it should not be performed for every pipeline failure.  Suppose the number of replications of the file is greater than or equal to 3.  When a pipeline fails, the operation will be invoked if
* the number of datanodes in the pipeline drops from 2 to 1; or
* the block is reopened for append; or
* it is specified by the user.

Note that when the number of replications is specified to less than 3, the operation should not be invoked by default because performance is preferred over data guarantee.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110406b.patch

h1606_20110406b.patch: the fault inject tests may set {{lastAckedSeqno}} to -2, a new state which the previous patch does not handle.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017793#comment-13017793 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475874/h1606_20110408b.patch
  against trunk revision 1090357.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017227#comment-13017227 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475764/h1606_20110407c.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.cli.TestHDFSCLI
                  org.apache.hadoop.hdfs.TestDFSShell
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/329//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/329//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018912#comment-13018912 ] 

Jitendra Nath Pandey commented on HDFS-1606:
--------------------------------------------

+1 for the patch.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016668#comment-13016668 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475654/h1606_20110406b.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 22 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestFileConcurrentReader
                  org.apache.hadoop.hdfs.TestLargeBlock

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/324//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/324//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/324//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110405b.patch

h1606_20110405b.patch: synchronized.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110408.patch

I suspect that the failures of {{TestFiDataTransferProtocol}} were due to resource not cleaning up completely by {{MiniDFSCluster}}.  Moved some the tests around in HDFS-1817.  See if it works.

h1606_20110408.patch: updated with trunk

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Description: 
In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.

We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

  was:
In the current design, if there is a datanode/network failures in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may not be accurate under erroneous conditions.

We propose to add a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

       Assignee: Tsz Wo (Nicholas), SZE

Below are two important use cases:

- h5. Long Living Pipeline (e.g. HBase logging)
When a pipeline is short living, the failure probability may be negligible.  However, when the client writes very slowly, the failure probability becomes significant.

- h5. File Append
When a new file is being written, if all the datanodes in a pipeline fail, then the data written will be lost.  Although the behavior is not ideal, it is acceptable since DFSClient will fail to close the file and we allow data loss in a never-closed file.  Nevertheless, when a closed file is reopened for append, the last block _B_ of the file is reopened and a pipeline is re-created (provided that the pre-append file size is not a multiple of the block size.)  _B_ will not be selected for replication until the pipeline is finished.  Then, the pre-append data stored in _B_ may be lost if all the datanodes in the pipeline fail and the subsequent block recovery fails.  Such behavior is unacceptable since the pre-append data was stored in a closed file.


> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993330#comment-12993330 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

Below are the proposed new configuration properties.
{code:xml}
<property>
  <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
  <value>ture</value>
  <description>
    If there is a datanode/network failure in the write pipeline,
    DFSClient will try to remove the failed datanode from the pipeline
    and then continue writing with the remaining datanodes. As a result,
    the number of datanodes in the pipeline is decreased.  The feature is
    to add new datanodes to the pipeline.

    This is a site-wise property to enable/disable the feature.

    See also dfs.client.block.write.replace-datanode-on-failure.policy
  </description>
</property>

<property>
  <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
  <value>DEFAULT</value>
  <description>
    This property is used only if the value of
    dfs.client.block.write.replace-datanode-on-failure.enable is true.

    ALWAYS: always add a new datanode when an existing datanode is removed.
    
    NEVER: never add a new datanode.

    DEFAULT: add a new datanode only if
             (1) the number of datanodes in the pipeline drops from 2 to 1; or
             (2) the block is reopened for append.
  </description>
</property>
{code}

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110211.patch

h1606_20110211.patch: add new configuration properties.  Still have a long way to go.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch, h1606_20110211.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110406.patch

h1606_20110406.patch: fixed the tests

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110210.patch

h1606_20110210.patch:
- Adding {{ClientProtocol.getAdditionalDatanode(..)}}
- {{NameNode}} side changes

Remaining works:
- implements \(*)
- changes {{DFSCleint}} to use the new datanode.
- adds new tests

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: h1606_20110210.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016264#comment-13016264 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475554/h1606_20110405b.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestFileAppend2
                  org.apache.hadoop.hdfs.TestFileConcurrentReader
                  org.apache.hadoop.hdfs.TestMultiThreadedHflush

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/318//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/318//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/318//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017071#comment-13017071 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-1606:
----------------------------------------------

Still have some FI tests failed.
- Too many open files
{noformat}
2011-04-06 22:27:23,329 WARN  datanode.DataNode (DataXceiverServer.java:run(142)) - DatanodeRegistration(127.0.0.1:41905, storageID=DS-961198735-127.0.1.1-41905-1302128843146, infoPort=35802, ipcPort=59788):DataXceiveServer: java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
	at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135)
	at java.lang.Thread.run(Thread.java:662)
{noformat}
- It also seems that there are some timeout.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016101#comment-13016101 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475446/h1606_20110404.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.permission.TestStickyBit
                  org.apache.hadoop.hdfs.TestFileAppend2
                  org.apache.hadoop.hdfs.TestFileAppend3
                  org.apache.hadoop.hdfs.TestFileAppend4
                  org.apache.hadoop.hdfs.TestLargeBlock
                  org.apache.hadoop.hdfs.TestPipelines
                  org.apache.hadoop.hdfs.TestReadWhileWriting
                  org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/316//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/316//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/316//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017181#comment-13017181 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475747/h1606_20110407b.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 25 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.server.datanode.TestBlockReport

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/327//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/327//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HDFS-1606:
-----------------------------------------

    Attachment: h1606_20110407.patch

h1606_20110407.patch: delayed opening a output stream.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016587#comment-13016587 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475626/h1606_20110406.patch
  against trunk revision 1087900.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 22 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/321//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/321//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017624#comment-13017624 ] 

Hadoop QA commented on HDFS-1606:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475833/h1606_20110408.patch
  against trunk revision 1090357.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestDFSShell
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//console

This message is automatically generated.

> Provide a stronger data guarantee in the write pipeline
> -------------------------------------------------------
>
>                 Key: HDFS-1606
>                 URL: https://issues.apache.org/jira/browse/HDFS-1606
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client, name-node
>    Affects Versions: 0.23.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.23.0
>
>         Attachments: h1606_20110210.patch, h1606_20110211.patch, h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, h1606_20110407c.patch, h1606_20110408.patch
>
>
> In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes.  As a result, the number of datanodes in the pipeline is decreased.  Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira