You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2011/08/08 18:04:27 UTC
[jira] [Created] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
MiniMR cluster thread crashes if no hadoop log dir set
------------------------------------------------------
Key: MAPREDUCE-2785
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.203.0
Reporter: Steve Loughran
Priority: Minor
I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
at java.io.File.<init>(File.java:222)
at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2785) MiniMR cluster thread crashes
if no hadoop log dir set
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501526#comment-13501526 ]
Matt Foley commented on MAPREDUCE-2785:
---------------------------------------
moved target version to 1.2.0 upon publishing 1.1.1 RC.
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-2785:
----------------------------------
Target Version/s: 1.2.0 (was: 1.1.1)
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-2785:
--------------------------------------
Assignee: Steve Loughran
Status: Open (was: Patch Available)
revert patch as hudson won't take it; plan to apply to 0.22 then backport
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.20.206.0
>
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-2785:
--------------------------------------
Attachment: MAPREDUCE-2785.patch
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-2785:
--------------------------------------
Fix Version/s: 0.20.206.0
Status: Patch Available (was: Open)
won't apply to trunk -there is no MiniMR cluster there, but it should also apply to 0.22
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.20.206.0
>
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2785) MiniMR cluster thread crashes if
no hadoop log dir set
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated MAPREDUCE-2785:
---------------------------------------
Target Version/s: 1.1.0
Fix Version/s: (was: 1.1.0)
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2785) MiniMR cluster thread crashes
if no hadoop log dir set
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105531#comment-13105531 ]
Hadoop QA commented on MAPREDUCE-2785:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494654/MAPREDUCE-2785.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/752//console
This message is automatically generated.
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.20.206.0
>
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2785) MiniMR cluster thread crashes
if no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453867#comment-13453867 ]
Steve Loughran commented on MAPREDUCE-2785:
-------------------------------------------
(ignore that last comment, it was for a Whirr issue)
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2785) MiniMR cluster thread crashes
if no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453866#comment-13453866 ]
Steve Loughran commented on MAPREDUCE-2785:
-------------------------------------------
log
{code}
running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(hdp1)
Not enough nodes available for template hadoop-namenode+hadoop-jobtracker
*CRITICAL* the cluster failed to launch and the automated node termination option was not selected, there might be orphaned nodes.
java.lang.IndexOutOfBoundsException: toIndex = 1
at java.util.SubList.<init>(AbstractList.java:602)
at java.util.RandomAccessSubList.<init>(AbstractList.java:758)
at java.util.AbstractList.subList(AbstractList.java:468)
at org.apache.whirr.actions.ByonClusterAction.doAction(ByonClusterAction.java:137)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:126)
at org.apache.whirr.ByonClusterController.bootstrapCluster(ByonClusterController.java:83)
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
at org.apache.whirr.cli.Main.run(Main.java:69)
at org.apache.whirr.cli.Main.main(Main.java:102)
Exception in thread "main" java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: toIndex = 1
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:128)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
at org.apache.whirr.cli.Main.run(Main.java:69)
at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: java.lang.IndexOutOfBoundsException: toIndex = 1
at java.util.SubList.<init>(AbstractList.java:602)
at java.util.RandomAccessSubList.<init>(AbstractList.java:758)
at java.util.AbstractList.subList(AbstractList.java:468)
at org.apache.whirr.actions.ByonClusterAction.doAction(ByonClusterAction.java:137)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:126)
at org.apache.whirr.ByonClusterController.bootstrapCluster(ByonClusterController.java:83)
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
... 4 more
{code}
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2785) MiniMR cluster thread crashes
if no hadoop log dir set
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105516#comment-13105516 ]
Steve Loughran commented on MAPREDUCE-2785:
-------------------------------------------
This is surprisingly hard to patch. I tried in TestMiniMRBringup, looking for the exception on creation, but the problem is that it happens in a child thread.
This is the stack trace you get in a separate thread from the test runner:
2011-09-15 18:14:19,382 WARN mapred.JobTracker (JobTracker.java:startTracker(314)) - Error starting tracker: java.io.IOException: No Hadoop Log dir - the configuration property hadoop.job.history.location is undefined, as is the system property hadoop.log.dir
at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:544)
at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:502)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2334)
at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2331)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2331)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
at java.lang.Thread.run(Thread.java:662)
-But as these threads are run from the constructor, the cluster is impossible to shut down cleanly afterwards.
For that reason: no tests
> MiniMR cluster thread crashes if no hadoop log dir set
> ------------------------------------------------------
>
> Key: MAPREDUCE-2785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.203.0
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: MAPREDUCE-2785.patch
>
>
> I'm marking this as minor as it is most obvious in the MiniMRCluster, but the root cause is in the JT.
> If you instantiate an MiniMR Cluster without setting {{hadoop.job.history.location}} in the configuration and the system property {{hadoop.log.dir}} unset, then the JobHistory throws an NPE. In production, that would be picked up as a failure to start the JT. In the MiniMRCluster, all it does is crash the JT thread -which isn't noticed by the MiniMR cluster. You see the logged error, but the tests will just timeout waiting for things to come up
> 2011/08/08 17:46:26:427 CEST [ERROR][Thread-44] org.apache.hadoop.mapred.MiniMRCluster - Job tracker crashed <java.lang.NullPointerException> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.hadoop.mapred.JobHistory.initLogDir(JobHistory.java:531)
> at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:499)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2316)
> at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2313)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2313)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:114)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner$1.run(MiniMRCluster.java:112)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.MiniMRCluster$JobTrackerRunner.run(MiniMRCluster.java:112)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira