You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ravi Phulari (JIRA)" <ji...@apache.org> on 2009/12/05 00:11:20 UTC
[jira] Created: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Additional tests(Unit tests & Functional tests) for HDFS-457.
--------------------------------------------------------------
Key: HDFS-811
URL: https://issues.apache.org/jira/browse/HDFS-811
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 0.22.0
Reporter: Ravi Phulari
Priority: Minor
Fix For: 0.22.0
HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
Description of HDFS-457
{quote}
Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
{quote}
I suggest following additional tests for this improvement.
#1 Test successive volume failures ( Minimum 4 volumes )
#2 Test if each volume failure reports reduction in available DFS space and remaining space.
#3 Test if failure of all volumes on a data nodes leads to the data node failure.
#4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876969#action_12876969 ]
Hadoop QA commented on HDFS-811:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445732/hdfs-811-4.patch
against trunk revision 952861.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 19 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/403/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/403/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/403/console
This message is automatically generated.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HDFS-811:
--------------------------------
Assignee: Eli Collins
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877278#action_12877278 ]
Eli Collins commented on HDFS-811:
----------------------------------
I'll update the patch to add descriptions to the asserts and remove the white-space you listed (arg!). I can't think of a better description for the class, I would put the tests in TestDataNodeVolumeFailure if they could easily share the setup, teardown, etc. but open to renaming if you've got a suggestion.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Status: Patch Available (was: Open)
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: hdfs-811-1.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866067#action_12866067 ]
Eli Collins commented on HDFS-811:
----------------------------------
The automated test covers the following but I also verified by hand on a local cluster that:
# Making a datadir inaccessible causes the volume to be blacklisted, the capacities and failed volume count adjusted accordingly on the web UI stats (using HDFS-556).
# Making the datadir accessible again does not cause it to be used until the datanode is restarted, at which point the failed volume count and capacities are back to their original values
# The above but made *all* datadirs inaccessible, confirmed the datanode is not used and is eventually is reported as dead.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: hdfs-811-1.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876936#action_12876936 ]
Eli Collins commented on HDFS-811:
----------------------------------
Hey Cos, thanks for taking a look! hdfs-811-4.patch applies on trunk for me. Kicked hudson to test it. You probably tried the previous patch. For some reason jira didn't put that one on top even though it's the latest patch.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Attachment: hdfs-811-2.patch
Patch attached. Adds the tests from HDFS-1161 and incorporates Dhruba's suggestions in that jira.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Attachment: hdfs-811-4.patch
Patch attached. Merges with trunk.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877115#action_12877115 ]
Eli Collins commented on HDFS-811:
----------------------------------
Thanks for taking a look Jakob. The latest patch mostly adds new code. Two big blocks of test code got shifted left due to removing the surrounding try blocks, but that's not really a white space change. I moved the whitespace changes in an earlier patch over to HDFS-1160, lemme know if I missed any.
Do you feel it's necessary to add descriptions like "Ensure DN 0 is up" to asserts like the following? I usually only add descriptions where the assert itself isn't clear. Happy to make the change for any asserts you find unclear (always more clear to the author than readers).
{code}
assertTrue(DataNode.isDatanodeUp(dns.get(0)));
{code}
The 2nd test class just adds additional DN volume failure test, but needs different startup and teardown methods which is why it's in a different class, so I used TestLeaseRecovery and TestFileAppend as examples which just sequentially number the test classes. Both test classes cover various aspects of volume failure so hard to come up with a name that's not generic. Open to suggestions of course.
Btw the previous test run was bogus (181 tests failed due to hudson issues).
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-811:
-----------------------------
Status: Open (was: Patch Available)
Canceling patch pending update...
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875825#action_12875825 ]
Konstantin Boudnik commented on HDFS-811:
-----------------------------------------
patch is no longer applicable. Could you please refit the patch and run it through {{test-patch}} if possible?
+1 'cause it looks good otherwise.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787986#action_12787986 ]
dhruba borthakur commented on HDFS-811:
---------------------------------------
another useful thing for a system adminstrator would be:
#5 Test that the WebUI tells the system adminstrator that one volume of a datanode is unavailable.
Thsi is needed so that the administrator can monitor (and fix) failed disks in the cluster.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Priority: Minor
> Fix For: 0.22.0
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Fix Version/s: 0.21.0
Affects Version/s: 0.21.0
Description:
HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
Description of HDFS-457
{quote}
Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
{quote}
I suggest following additional tests for this improvement.
#1 Test successive volume failures ( Minimum 4 volumes )
#2 Test if each volume failure reports reduction in available DFS space and remaining space.
#3 Test if failure of all volumes on a data nodes leads to the data node failure.
#4 Test if correcting failed storage disk brings updates and increments available DFS space.
was:
HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
Description of HDFS-457
{quote}
Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
{quote}
I suggest following additional tests for this improvement.
#1 Test successive volume failures ( Minimum 4 volumes )
#2 Test if each volume failure reports reduction in available DFS space and remaining space.
#3 Test if failure of all volumes on a data nodes leads to the data node failure.
#4 Test if correcting failed storage disk brings updates and increments available DFS space.
This blocks HDFS-1161 and we want that in 21 so updating this fixVersion as well.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872035#action_12872035 ]
Konstantin Boudnik commented on HDFS-811:
-----------------------------------------
A couple of comments:
- JavaDoc for the parameter is missing
+ public void updateRegInfo(DatanodeID nodeReg) {
- same for exception throw in
+ synchronized public void incVolumeFailure(DatanodeID nodeID)
+ throws IOException {
- Some of new tests have JavaDocs and some aren't. It's better be consistent.
- it usually a good style to have meaningful messages for the assertions.
Seems to be good otherwise.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Status: Open (was: Patch Available)
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Attachment: hdfs-811-1.patch
Patch attached.
* Adds a DN metric to track volume failures
* Adds DN descriptor state to track failed volumes (used for HDFS-556)
* Refactors TestDiskError and adds test coverage for individual volume failure behavior, that capacity and liveliness is adjust accordingly, etc.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: hdfs-811-1.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788258#action_12788258 ]
Boris Shkolnik commented on HDFS-811:
-------------------------------------
I think there is already JIRA on this - HDFS-556
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Priority: Minor
> Fix For: 0.22.0
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876948#action_12876948 ]
Jakob Homan commented on HDFS-811:
----------------------------------
Before this goes in, there are still quite a few undescribed asserts and a fair amount of unnecessary white space changes that should be removed. Also, TestDataNodeVolumeFailure2? Oy. It that the sequel? Would a more descriptive name be reasonable?
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877268#action_12877268 ]
Konstantin Boudnik commented on HDFS-811:
-----------------------------------------
+1 on Jacob point. It isn't about how obvious the assertion is. It is about time one needs to spend on a failure analysis. If there's no message one needs to pull up the source code, find the line, read the context, etc. With meaningful diagnostics message it is usually enough to just glance at the log file.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Status: Patch Available (was: Open)
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Ravi Phulari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788252#action_12788252 ]
Ravi Phulari commented on HDFS-811:
-----------------------------------
Dhruba thanks for comment,
bq. #5 Test that the WebUI tells the system adminstrator that one volume of a datanode is unavailable.
AFIK currently WebUI does not support functionality of explicitly indicating volume failure in data node, so we can not include this test case.
This is good to have functionality, opening new Jira for it.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Ravi Phulari
> Priority: Minor
> Fix For: 0.22.0
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Ravi Phulari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872017#action_12872017 ]
Ravi Phulari commented on HDFS-811:
-----------------------------------
Reviewed, Patch looks good.
+1
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Additional tests(Unit tests &
Functional tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872026#action_12872026 ]
Eli Collins commented on HDFS-811:
----------------------------------
Thanks for taking a look Ravi. Konst asked that the patch for HDFS-1161 apply against trunk instead of depending on this one so I moved the test from that jira to this jira, and also made the tests waits indefinitely for the required failures instead of just a computed amount of time which Dhruba suggested in HDFS-1161.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877289#action_12877289 ]
Jakob Homan commented on HDFS-811:
----------------------------------
Also, it looks like the new test takes about 75 sec to run. Since it's going into src/test/hdfs rather than src/test/unit (which is still a loney, vacant neighborhood...), this is ok.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877266#action_12877266 ]
Jakob Homan commented on HDFS-811:
----------------------------------
The test failure lines are most useful when a test fails and you're looking through the output in the test directory, or on Hudson, and may not have the context of the failure, so I'd prefer more than less, but if you're ok with the current level I guess it's fine. I had run a search and saw the TestLeaseRecovery and TestFileAppend files, as well. Don't you think those are poor choices for explaining what's going on in those tests? If there is really no better description, then I guess we'll just have to live with it. As for white-space changes, I'm referring to lines such as 35, 124, 159, 185, 206, 207, 222, 227, 252, 260, etc.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch, hdfs-811-4.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Add metrics, failure reporting and
additional tests for HDFS-457
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Summary: Add metrics, failure reporting and additional tests for HDFS-457 (was: Additional tests(Unit tests & Functional tests) for HDFS-457.)
Updating the summary to reflect that the change also adds volume failure DN metrics and plumbing to report and reset failure counts in the NN.
> Add metrics, failure reporting and additional tests for HDFS-457
> ----------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-811) Additional tests(Unit tests & Functional
tests) for HDFS-457.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-811:
-----------------------------
Attachment: hdfs-811-3.patch
Thanks for taking a look Kos. Attached patch addresses your feedback. I also moved the new tests from TestDiskError to TestDataNodeVolumeFailure2 since they were getting big and to be consistent with TestDataNodeVolumeFailure, which I modified to be a junit4 style test. Doing a full test-core run with HDFS-1161 applied.
> Additional tests(Unit tests & Functional tests) for HDFS-457.
> --------------------------------------------------------------
>
> Key: HDFS-811
> URL: https://issues.apache.org/jira/browse/HDFS-811
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Ravi Phulari
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.21.0, 0.22.0
>
> Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch
>
>
> HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
> Description of HDFS-457
> {quote}
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
> {quote}
> I suggest following additional tests for this improvement.
> #1 Test successive volume failures ( Minimum 4 volumes )
> #2 Test if each volume failure reports reduction in available DFS space and remaining space.
> #3 Test if failure of all volumes on a data nodes leads to the data node failure.
> #4 Test if correcting failed storage disk brings updates and increments available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.