You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "srikanth ayalasomayajulu (JIRA)" <ji...@apache.org> on 2012/06/12 08:47:42 UTC
[jira] [Created] (MAPREDUCE-4339) pi example job hangs on when run
on hadoop 0.23.0 when capacity scheduler is included in the setting
environment.
srikanth ayalasomayajulu created MAPREDUCE-4339:
---------------------------------------------------
Summary: pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
Key: MAPREDUCE-4339
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples, job submission, mrv2, scheduler
Affects Versions: 0.23.0
Environment: Ubuntu Server 11.04, Hadoop 0.23.0,
Reporter: srikanth ayalasomayajulu
Fix For: 0.23.0
Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed.
Starting Job
2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030
2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS
2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10
2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10
2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048
2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager
2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002
2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4339) pi example job hangs on when
run on hadoop 0.23.0 when capacity scheduler is included in the setting
environment.
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395964#comment-13395964 ]
Jason Lowe commented on MAPREDUCE-4339:
---------------------------------------
I am unable to reproduce a hang like this on a single-node cluster. Could you examine the ResourceManager logs for issues or post them (after any necessary scrubbing/anonymization)? That would help track down what's going on when the job hangs.
> pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4339
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples, job submission, mrv2, scheduler
> Affects Versions: 0.23.0
> Environment: Ubuntu Server 11.04, Hadoop 0.23.0,
> Reporter: srikanth ayalasomayajulu
> Labels: hadoop
> Fix For: 0.23.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed.
> Starting Job
> 2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10
> 2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10
> 2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048
> 2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager
> 2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002
> 2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4339) pi example job hangs on when
run on hadoop 0.23.0 when capacity scheduler is included in the setting
environment.
Posted by "srikanth ayalasomayajulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294215#comment-13294215 ]
srikanth ayalasomayajulu commented on MAPREDUCE-4339:
-----------------------------------------------------
This example i ran in on a single node where all the daemons are running on the same node successfully. I tried to use the capacity scheduler. then the job is getting hanged.
> pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4339
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples, job submission, mrv2, scheduler
> Affects Versions: 0.23.0
> Environment: Ubuntu Server 11.04, Hadoop 0.23.0,
> Reporter: srikanth ayalasomayajulu
> Labels: hadoop
> Fix For: 0.23.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed.
> Starting Job
> 2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10
> 2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10
> 2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048
> 2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager
> 2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002
> 2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4339) pi example job hangs on when
run on hadoop 0.23.0 when capacity scheduler is included in the setting
environment.
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293734#comment-13293734 ]
Jason Lowe commented on MAPREDUCE-4339:
---------------------------------------
I'm wondering if this is caused by what you reported in MAPREDUCE-4338. Are there any nodemanagers successfully running in the cluster at the time the job hangs? A job submitted to a cluster that has no nodemanagers will hang waiting for nodes to run the job. You can check the RM web UI to see if there are any nodes registered.
> pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4339
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples, job submission, mrv2, scheduler
> Affects Versions: 0.23.0
> Environment: Ubuntu Server 11.04, Hadoop 0.23.0,
> Reporter: srikanth ayalasomayajulu
> Labels: hadoop
> Fix For: 0.23.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed.
> Starting Job
> 2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10
> 2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10
> 2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048
> 2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager
> 2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002
> 2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4339) pi example job hangs on when
run on hadoop 0.23.0 when capacity scheduler is included in the setting
environment.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293630#comment-13293630 ]
Arun C Murthy commented on MAPREDUCE-4339:
------------------------------------------
Srikanth - what do you see in the RM logs?
> pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4339
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples, job submission, mrv2, scheduler
> Affects Versions: 0.23.0
> Environment: Ubuntu Server 11.04, Hadoop 0.23.0,
> Reporter: srikanth ayalasomayajulu
> Labels: hadoop
> Fix For: 0.23.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed.
> Starting Job
> 2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:<init>(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030
> 2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10
> 2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10
> 2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048
> 2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager
> 2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002
> 2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira