You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2012/07/02 02:17:56 UTC
[jira] [Created] (HDFS-3582) Hook System.exit in MiniDFSCluster
Eli Collins created HDFS-3582:
---------------------------------
Summary: Hook System.exit in MiniDFSCluster
Key: HDFS-3582
URL: https://issues.apache.org/jira/browse/HDFS-3582
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Description:
Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
was:
Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
Summary: Hook daemon process exit for testing (was: Hook System.exit in MiniDFSCluster)
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411664#comment-13411664 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535994/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2790//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2790//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2790//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2790//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406065#comment-13406065 ]
Aaron T. Myers commented on HDFS-3582:
--------------------------------------
Instead of mucking with SecurityManager at all, I'm personally in favor of changing all references to System.exit to use Runtime#exit instead, and then mocking that out in MiniDFSCluster and asserting that Runtime#exit was never called on MiniDFSCluster shutdown. This strategy is already used by a few tests (e.g. TestEditLogJournalFailures) to assert that exit *is* called in some cases. These tests could be changed to no longer do their own mocking of Runtime objects, and instead MiniDFSCluster could be made to optionally assert that Runtime#exit *was* called > 0 times. This mocking as I've described will probably only need to be done in a handful of places.
Thoughts?
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Fixed one findbugs warning (findbugs isn't able to figure out one statement after terminate is currently unreachable). The other two are HDFS-3615. Test failures are unrelated.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Thanks Stephen. Updated patch.
- ExitUtil members now volatile since we're catching the multithread case now (exit in one thread and detected by MiniDFSCluster in another)
- Made ExitUtil limited private to HDFS/MR. Once we're happy with the API for these two we can broaden the audience.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Target Version/s: (was: 2.0.1-alpha)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406070#comment-13406070 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
Agree, I'm going to update the patch to introduce/use a single NN method that does a System.exit and mock that out in MiniDFSCluster w/ the check as you describe.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Good point Colin, updated the javadoc to indicate as such.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405244#comment-13405244 ]
Alejandro Abdelnur commented on HDFS-3582:
------------------------------------------
In the case of HDFS testcases it may well be that things just work. The thing is, minicluster runs in the same JVM as the testcases. When used in Oozie, Oozie testcases inject/remove security managers in different testcases (we need to do this because hive/sqoop/pig entry classes do a system.exit() on completion and we need to trap that).
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405213#comment-13405213 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
I should say before it creates the NN, 2NN, DNs, etc since technically these aren't daemons in the mini cluster.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Eclipse missed the BK test in contrib when doing the class move. Updated patch to fix that.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409055#comment-13409055 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535588/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2760//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2760//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2760//console
This message is automatically generated.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410636#comment-13410636 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535881/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
-1 javac. The patch appears to cause the build to fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2774//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Description:
Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
was:
Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Patch attached.
Tested that (1) TestClusterId and TestDFSShell behavior is preserved, and (2) an unmodified test like TestCheckpoint will fail cleanly with an ExitException if eg the NN calls System.exit.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411854#comment-13411854 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2470 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2470/])
HDFS-3582. Hook daemon process exit for testing. Contributed by Eli Collins (Revision 1360329)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360329
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureToReadEdits.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStateTransitionFailure.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405097#comment-13405097 ]
Alejandro Abdelnur commented on HDFS-3582:
------------------------------------------
@Eli, it would be great if the Minicluster code itself does not poke with the SecurityManager. Instead, it should be the testcase the one doing that. Else, this will break other testcases that poke with SecurityManager and use Minicluster (Oozie has a few of those).
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411786#comment-13411786 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2518 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2518/])
HDFS-3582. Hook daemon process exit for testing. Contributed by Eli Collins (Revision 1360329)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360329
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureToReadEdits.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStateTransitionFailure.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Updated patch attached.
- Was missing the Runtime#exit calls, fixed these and updated all the relevant edit log tests to match
- Now passing a message to ExitUtil#terminate so the particular cause of the exist is captured in the exception is available to the tests
- NB: I'm just hooking daemon exits, not eg all the tool exits (balancer, *admin, recovery mode)
- Made logs "fatal" that were "error" but terminated
- Made the NN/DN/2NN exit failure codes consistent (use 1 in places we were using -1)
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410596#comment-13410596 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
ATM, Steve,
I'll update the patch to move ExitUtil to common, I figured we'd do that if/when MR adopts this but I'm fine doing it now.
I think hooking this via a single terminate function is the right approach vs the security manager, aside from the Tucu's above arguments it's made the testing / debugging cleaner. This doesn't preclude hooking with a security manager later, or adding alternative hooks that do something aside from throw ExitException.
Thanks,
Eli
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404893#comment-13404893 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534177/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestConnCache
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2729//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2729//console
This message is automatically generated.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405192#comment-13405192 ]
Alejandro Abdelnur commented on HDFS-3582:
------------------------------------------
Then, shouldn't you setup the the security manager BEFORE ALL TESTS? else you are assuming that the Minicluster will be initilized before somebody does a system.exit(#).
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405155#comment-13405155 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
@Alejandro, those should still work right? An example is TestDFSShell, it sets it's own security manager and overrides the one set by the mini cluster. The reason I didn't modify the tests is that we want to catch System.exit in *all tests* therefore the net result would be equivalent. Per above an alternative is to stub out System.exit in HDFS and mock it in the tests.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Status: Patch Available (was: Open)
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410143#comment-13410143 ]
Steve Loughran commented on HDFS-3582:
--------------------------------------
You don't need to do this for testing because the security manager hook is much less invasive. Indeed, all you need to do is
1. stick ant.jar on the classpath
2. set the security manager to org.apache.tools.ant.util.optional.NoExitSecurityManager:
[http://svn.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/ant/util/optional/NoExitSecurityManager.java?view=markup]
I am supportive of a single exit point and the ability to trap exits -which is is why I field HADOOP-8096 in the first place. But to embed Hadoop properly in OSGi we need this across the board.
* the exit routine and exception should go into hadoop-core
* -hdfs and -mapreduce should adopt it
Having an HDFS-specific variant purely for testing doesn't aid OSGi integration, and will only increase the effort of switching to a unified version downstream.
Why not fix it properly the first time, and apply it in HDFS first?
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409628#comment-13409628 ]
Colin Patrick McCabe commented on HDFS-3582:
--------------------------------------------
ExitUtil#terminate: should we point out in the JavaDoc that this is the *only* "exit the process" method that should be called from the NN or DN? I didn't get that sense from reading the comment that's there now.
Other than that, looks great...
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
The two findbugs failures are unrelated. Slightly updated patch as I missed updating a javadoc that was throwing ServiceFailedException and now throws ExitException.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410233#comment-13410233 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535764/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2766//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2766//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2766//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405408#comment-13405408 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
Right but why does what's done here break that? If you explicitly set a security manager in the Oozie test case that's fine it will just override the one here.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412804#comment-13412804 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1134 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1134/])
HDFS-3582. Hook daemon process exit for testing. Contributed by Eli Collins (Revision 1360329)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360329
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureToReadEdits.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStateTransitionFailure.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Left out ExitUtil from the previous patch.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411792#comment-13411792 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2452 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2452/])
HDFS-3582. Hook daemon process exit for testing. Contributed by Eli Collins (Revision 1360329)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360329
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureToReadEdits.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStateTransitionFailure.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408554#comment-13408554 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535429/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHDFSTrash
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2755//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2755//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2755//console
This message is automatically generated.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Updated patch attached - moves ExitUtil and ExitException to common.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411110#comment-13411110 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535936/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2779//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2779//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2779//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2779//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405211#comment-13405211 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
That's correct, I'm assuming the mini cluster is initialized before it starts the daemons (which may System.exit), which is a valid assumption no?
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411768#comment-13411768 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
Thanks for the review ATM. The TestBackupNode failure is HDFS-3625, the test is System#exit'ing (ie the type of failure we can now catch as a unit test failure).
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Updated patch, per offline feedback from ATM modifies MiniDFSCluster to assert on shutdown that terminate was not called. This will catch cases where tests ignore IOExceptions and therefore ignore a server-side ExitException that is turned into a RemoteExceptions, or a test/code path is swallowing RuntimeExceptions.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410726#comment-13410726 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535878/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2772//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2772//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2772//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411244#comment-13411244 ]
Steve Loughran commented on HDFS-3582:
--------------------------------------
This looks good.
Two changes I'd recommend
# fields in the ExitUtil class are synchronized or volatile for x-thread work.
# packaging skope isn't private so much as internal + unstable. We'd like downstream projects to use it too, right?
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411145#comment-13411145 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
The 1 common and 2 hdfs findbugs warnings are unrelated. The test failures are HADOOP-8537, HDFS-3635 and HDFS-3636.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3582) Hook daemon process exit for testing
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3582.
-------------------------------
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411550#comment-13411550 ]
Aaron T. Myers commented on HDFS-3582:
--------------------------------------
The latest patch looks good to me. +1 pending Jenkins (which I've just kicked.)
I also heard from Steve offline that he's +1 on the latest patch, but won't have access to JIRA today.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410122#comment-13410122 ]
Aaron T. Myers commented on HDFS-3582:
--------------------------------------
bq. I wouldn't justify this fix purely for testing, though HADOOP-8096 shows utility at run time.
Why not? We sometimes have tests that fail that we can't adequately debug because of this. That seems like reason enough to me that we should address this issue.
bq. It is not the Hadoop team's fault that Surefire can't handle process exit calls, especially given that Apache Ant has happily done so for twelve years.
Certainly not our fault, but we still suffer the consequences. Since we have a (pretty simple) way to work around the limitation, I think we should implement it.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411182#comment-13411182 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535936/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2783//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2783//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2783//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2783//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412713#comment-13412713 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1101 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1101/])
HDFS-3582. Hook daemon process exit for testing. Contributed by Eli Collins (Revision 1360329)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360329
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureToReadEdits.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStateTransitionFailure.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410582#comment-13410582 ]
Eli Collins commented on HDFS-3582:
-----------------------------------
Turns out the TestBackupNode failure above was real, filed HDFS-3625, we caught it now because we're getting an exception rather than having the test process exit.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410119#comment-13410119 ]
Steve Loughran commented on HDFS-3582:
--------------------------------------
I wouldn't justify this fix purely for testing, though HADOOP-8096 shows utility at run time.
It is not the Hadoop team's fault that Surefire can't handle process exit calls, especially given that Apache Ant has happily done so for twelve years.
The OSGi-centric patch should apply to the whole codebase, not just HDFS.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405627#comment-13405627 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3582:
----------------------------------------------
I think we should check if there is already a SecurityManager before setting it. Otherwise, we may accidentally override an existing SecurityManager.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3582:
------------------------------
Attachment: hdfs-3582.txt
Patch attached. Introduces a method for process termination that is hooked in MiniDFSCluster and some tests to throw a RuntimeException so we can see why the exit occurred and verify the exit code.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook System.exit in MiniDFSCluster
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406071#comment-13406071 ]
Aaron T. Myers commented on HDFS-3582:
--------------------------------------
Cool. I think you might also need to do a similar thing for the DN as well.
> Hook System.exit in MiniDFSCluster
> ----------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System.exit or Runtime.exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this debuggable, let's hook this in MiniDFSCluster via installing a security manager that overrides checkExit (ala TestClusterId) or mock out System.exit in the code itself. I think the former is preferable though we'll need to keep the door open for tests that want to set their own security manager (should be fine to override this one some times).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410010#comment-13410010 ]
Hadoop QA commented on HDFS-3582:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535764/hdfs-3582.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2764//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2764//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2764//console
This message is automatically generated.
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3582) Hook daemon process exit for testing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477822#comment-13477822 ]
Hudson commented on HDFS-3582:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #407 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/407/])
HDFS-4016. back-port HDFS-3582 to branch-0.23 (Ivan A. Veselovsky via bobby) (Revision 1398901)
Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398901
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
> Hook daemon process exit for testing
> -------------------------------------
>
> Key: HDFS-3582
> URL: https://issues.apache.org/jira/browse/HDFS-3582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt, hdfs-3582.txt
>
>
> Occasionally the tests fail with "java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException:
> Error occurred in starting fork, check output in log" because the NN is exit'ing (via System#exit or Runtime#exit). Unfortunately Surefire doesn't retain the log output (see SUREFIRE-871) so the test log is empty, we don't know which part of the test triggered which exit in HDFS. To make this easier to debug let's hook all daemon process exits when running the tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira