You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2011/04/20 03:39:06 UTC
[jira] [Created] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
DN should transmit absolute failed volume count rather than increments to the NN
--------------------------------------------------------------------------------
Key: HDFS-1850
URL: https://issues.apache.org/jira/browse/HDFS-1850
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, name-node
Reporter: Eli Collins
Assignee: Eli Collins
Fix For: 0.23.0
The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
# DN loses a volume and reports it
# NN restarts
# DN re-registers to the new NN
A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027888#comment-13027888 ]
Hadoop QA commented on HDFS-1850:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477990/hdfs-1850-6.patch
against trunk revision 1098781.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 38 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/443//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/443//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/443//console
This message is automatically generated.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Status: Patch Available (was: Open)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0, 0.23.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027917#comment-13027917 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
+1
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-6.patch
Arg. Right patch this time.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026661#comment-13026661 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
hm, I still see {{getNumFailedVols}} in hdfs-1850-3.patch.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037379#comment-13037379 ]
Hudson commented on HDFS-1850:
------------------------------
Integrated in Hadoop-Hdfs-trunk #673 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/])
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026101#comment-13026101 ]
Hadoop QA commented on HDFS-1850:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477428/hdfs-1850-2.patch
against trunk revision 1097252.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 39 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/425//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/425//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/425//console
This message is automatically generated.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0, 0.23.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Fix Version/s: (was: 0.23.0)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-3.patch
Thanks Todd. Patch attached addresses your comments (renames and ensures each call to errorReport gets only one message).
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031467#comment-13031467 ]
Hudson commented on HDFS-1850:
------------------------------
Integrated in Hadoop-Hdfs-22-branch #41 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/41/])
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-5.patch
Patch attached, rebased on trunk.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-2.patch
Updated patch attached. Breaks out the testing of DFS_DATANODE_FAILED_VOLUMES_TOLERATED to a new test so it's easier to add new tests of this functionality.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0, 0.23.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Status: Patch Available (was: Open)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Status: Open (was: Patch Available)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-7.patch
TestDatanodeBlockScanner passes for me in eclipse and when looped from the command line. I think this is related to an earlier change. The error message indicates we need to bump the number of attempts (ie it did see 1 corrupt replica after 20 attempts, but it also bails on the 20th attempt).
{quote}
java.util.concurrent.TimeoutException: Timed out waiting for corrupt replicas. Waiting for 1, but only found 1
{quote}
I'll bump the # attempts to 50 so we're more tolerant.
Updated patch attached.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Description:
The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
# DN loses a volume and reports it
# NN restarts
# DN re-registers to the new NN
A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
was:
The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
# DN loses a volume and reports it
# NN restarts
# DN re-registers to the new NN
A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
This will likely be an incompatible change since it requires changing the Datanode protocol.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-4.patch
Oops, just renamed the variable, not the method as well. Fixed. Patch attached.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Test failures are unrelated. I've committed this to trunk and branch 22. Thanks Todd!
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Hadoop Flags: (was: [Incompatible change])
Good point. Flag removed.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.23.0
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Attachment: hdfs-1850-1.patch
Patch attached.
1. Modifies FSDataset to track and report volume failures like other capacity etc. Adds the test listed in the description, makes TestDataNodeVolumeFailureReporting more robust.
2. Renames the volumesFailed metric to volumeFailures to accurately reflect what it's tracking. This doesn't break compatibility because this metric (added in HDFS-811) has not yet been released.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.23.0
>
> Attachments: hdfs-1850-1.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Status: Patch Available (was: Open)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027963#comment-13027963 ]
Hadoop QA commented on HDFS-1850:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12478000/hdfs-1850-7.patch
against trunk revision 1098781.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 38 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/444//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/444//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/444//console
This message is automatically generated.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch, hdfs-1850-7.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021902#comment-13021902 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
I don't think changes to DN->NN RPC are considered "incompatible changes" going from eg 0.22 to 0.23, since they don't affect users. Given that we don't purport to allow rolling upgrade between releases like this, it's not particularly an issue, right?
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.23.0
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027908#comment-13027908 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
Is this TestDatanodeBlockScanner failure related to the patch? It has a strange error message:
java.util.concurrent.TimeoutException: Timed out waiting for corrupt replicas. Waiting for 1, but only found 1
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch, hdfs-1850-6.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Status: Open (was: Patch Available)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026809#comment-13026809 ]
Hadoop QA commented on HDFS-1850:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477703/hdfs-1850-4.patch
against trunk revision 1097329.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 39 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.TestLargeBlock
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/432//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/432//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/432//console
This message is automatically generated.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1850:
------------------------------
Fix Version/s: 0.22.0
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0, 0.23.0
>
> Attachments: hdfs-1850-1.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026254#comment-13026254 ]
Hadoop QA commented on HDFS-1850:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12477622/hdfs-1850-3.patch
against trunk revision 1097329.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 39 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.TestLargeBlock
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/428//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/428//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/428//console
This message is automatically generated.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026074#comment-13026074 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
Some small comments:
- {{getNumFailedVols}} - probably best not to abbreviate vols, since most of the time we use the full word: {{getNumFailedVolumes}}
- in {{errorReport(...)}} we seem to log twice in the case that there is a disk error. Maybe the first LOG.info should get moved inside the if statement, and "msg" should be included in the warns?
Also, should make this Patch Available to run through Hudson.
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0, 0.23.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1850) DN should transmit absolute failed
volume count rather than increments to the NN
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027810#comment-13027810 ]
Todd Lipcon commented on HDFS-1850:
-----------------------------------
hdfs-1850-5.patch seems to have some conflict markers stuck in it (line 206 of the patch)
> DN should transmit absolute failed volume count rather than increments to the NN
> --------------------------------------------------------------------------------
>
> Key: HDFS-1850
> URL: https://issues.apache.org/jira/browse/HDFS-1850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 0.22.0
>
> Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, hdfs-1850-4.patch, hdfs-1850-5.patch
>
>
> The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:
> # DN loses a volume and reports it
> # NN restarts
> # DN re-registers to the new NN
> A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
> This will likely be an incompatible change since it requires changing the Datanode protocol.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira