You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2012/07/15 01:42:34 UTC
[jira] [Created] (HDFS-3664)
TestPersistBlocks#TestRestartDfsWithFlush failed with NPE
Eli Collins created HDFS-3664:
---------------------------------
Summary: TestPersistBlocks#TestRestartDfsWithFlush failed with NPE
Key: HDFS-3664
URL: https://issues.apache.org/jira/browse/HDFS-3664
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Eli Collins
Looks like there's a race hit by the test where we try to allocate a block after BlocksMap#close, resulting in an NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453696#comment-13453696 ]
Hudson commented on HDFS-3664:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2786 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2786/])
HDFS-3664. BlockManager race when stopping active services. Contributed by Colin Patrick McCabe (Revision 1383753)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383753
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HDFS-3664:
---------------------------------
Assignee: Colin Patrick McCabe
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453585#comment-13453585 ]
Eli Collins commented on HDFS-3664:
-----------------------------------
Change looks good. And the test failures look unrelated, TestDatanodeBlockScanner is known and looks like its still running java process is locking the storage dir causing TestDatanodeBlockScanner to fail. I verified TestBlockMissingException with your patch for sanity.
Not your change but in BlockManager we should check datanodeManager and blocksMap for null like we do pendingReplications right?
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453656#comment-13453656 ]
Hadoop QA commented on HDFS-3664:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544746/HDFS-3664.002.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3175//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3175//console
This message is automatically generated.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453590#comment-13453590 ]
Colin Patrick McCabe commented on HDFS-3664:
--------------------------------------------
bq. Not your change but in BlockManager we should check datanodeManager and blocksMap for null like we do pendingReplications right?
Hmm. All of those fields are declared {{final}} and unconditionally initialized to something non-null in the constructor, so I don't think there should be any null checks in the {{close}} function. I'll post a patch in a sec.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3664:
---------------------------------------
Attachment: HDFS-3664.001.patch
* Previously, we were setting blocksMap to {{null}} before calling {{datanodeManager#close}}. However, {{DatanodeManager#decommissionThread}} accesses blocksMap at various times while the thread runs. Instead, wait until the decommissionThread has been stopped before closing blocksMap.
* We also previously were not joining {{DatanodeManager#decommissionThread}} or {{DatanodeManager#heartbeatManager}}, so there was no guarantee those threads would have exited before blocksMap was nulled. Let's join these threads.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Attachments: HDFS-3664.001.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454002#comment-13454002 ]
Hudson commented on HDFS-3664:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1194 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1194/])
HDFS-3664. BlockManager race when stopping active services. Contributed by Colin Patrick McCabe (Revision 1383753)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383753
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3664:
---------------------------------------
Status: Patch Available (was: Open)
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Attachments: HDFS-3664.001.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3664.
-------------------------------
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453568#comment-13453568 ]
Hadoop QA commented on HDFS-3664:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544723/HDFS-3664.001.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestBlockMissingException
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3173//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3173//console
This message is automatically generated.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3664:
------------------------------
Target Version/s: 2.2.0-alpha
Affects Version/s: 2.0.0-alpha
Yea, this is a separate issue. Looks like we're racing between BM shutdown and an outstanding request.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453716#comment-13453716 ]
Hudson commented on HDFS-3664:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2747 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2747/])
HDFS-3664. BlockManager race when stopping active services. Contributed by Colin Patrick McCabe (Revision 1383753)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383753
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453955#comment-13453955 ]
Hudson commented on HDFS-3664:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1163 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1163/])
HDFS-3664. BlockManager race when stopping active services. Contributed by Colin Patrick McCabe (Revision 1383753)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383753
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3664:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.2-alpha
Target Version/s: (was: 2.0.2-alpha)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2 and branch-2.0.2-alpha. Thanks Colin.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3664:
---------------------------------------
Attachment: HDFS-3664.002.patch
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664)
TestPersistBlocks#TestRestartDfsWithFlush failed with NPE
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414550#comment-13414550 ]
Eli Collins commented on HDFS-3664:
-----------------------------------
Here's the relevant portion of the log:
2012-07-14 20:28:44,991 INFO namenode.FSNamesystem (FSNamesystem.java:stopActiveServices(673)) - Stopping services started for active state
2012-07-14 20:28:44,991 WARN blockmanagement.DecommissionManager (DecommissionManager.java:run(78)) - Monitor interrupted: java.lang.InterruptedException: sleep interrupted
2012-07-14 20:28:44,991 INFO namenode.FSNamesystem (FSNamesystem.java:stopStandbyServices(731)) - Stopping services started for standby state
2012-07-14 20:28:44,992 WARN ipc.Server (Server.java:run(1706)) - IPC Server handler 2 on 58865, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 127.0.0.1:60713: error: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.getBlockCollection(BlocksMap.java:98)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getBlockCollection(BlockManager.java:2893)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isValidBlock(FSNamesystem.java:4437)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:2409)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:452)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
> TestPersistBlocks#TestRestartDfsWithFlush failed with NPE
> ---------------------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Eli Collins
>
> Looks like there's a race hit by the test where we try to allocate a block after BlocksMap#close, resulting in an NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3664) BlockManager race when stopping active
services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3664:
------------------------------
Description:
Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
was:Looks like there's a race hit by the test where we try to allocate a block after BlocksMap#close, resulting in an NPE.
Summary: BlockManager race when stopping active services (was: TestPersistBlocks#TestRestartDfsWithFlush failed with NPE)
Same NPE in TestListCorruptFileBlocks.testListCorruptFileBlocksInSafeMode.
2012-07-15 21:48:39,685 INFO namenode.FSNamesystem (FSNamesystem.java:stopActiveServices(673)) - Stopping services started for active state
2012-07-15 21:48:39,686 FATAL blockmanagement.BlockManager (BlockManager.java:run(2969)) - ReplicationMonitor thread received Runtime exception.
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.getBlockCollection(BlocksMap.java:98)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1097)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1072)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3000)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:2962)
at java.lang.Thread.run(Thread.java:662)
2012-07-15 21:48:39,685 WARN blockmanagement.DecommissionM
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Eli Collins
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453695#comment-13453695 ]
Hudson commented on HDFS-3664:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2723 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2723/])
HDFS-3664. BlockManager race when stopping active services. Contributed by Colin Patrick McCabe (Revision 1383753)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383753
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453683#comment-13453683 ]
Eli Collins commented on HDFS-3664:
-----------------------------------
Good catch. +1 (test failure is still unrelated)
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3664.001.patch, HDFS-3664.002.patch
>
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3664) BlockManager race when stopping
active services
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439159#comment-13439159 ]
Andy Isaacson commented on HDFS-3664:
-------------------------------------
bq. May be a dupe of HDFS-3048.
We've merged that patch and are still seeing this NPE (for example HDFS-3822), so it's not a dupe.
> BlockManager race when stopping active services
> -----------------------------------------------
>
> Key: HDFS-3664
> URL: https://issues.apache.org/jira/browse/HDFS-3664
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Eli Collins
>
> Looks like there's a race hit by the test where we try to do block management while shutting down (eg allocate a block or compute replication work after BlocksMap#close, resulting in an NPE).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira