You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "shenhong (JIRA)" <ji...@apache.org> on 2014/02/10 06:49:22 UTC
[jira] [Updated] (MAPREDUCE-5749)
TestRMContainerAllocator#testReportedAppProgress Failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
shenhong updated MAPREDUCE-5749:
--------------------------------
Description:
When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed with message:
{code}
Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml does not exist
{code}
If fact, the job.xml exits in:
/home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
See the following code:
{code}
public Job submit(Configuration conf, boolean mapSpeculative,
boolean reduceSpeculative) throws Exception {
String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
.getCurrentUser().getShortUserName());
conf.set(MRJobConfig.USER_NAME, user);
conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
// TODO: fix the bug where the speculator gets events with
// not-fully-constructed objects. For now, disable speculative exec
conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);
init(conf);
start();
DefaultMetricsSystem.shutdown();
Job job = getContext().getAllJobs().values().iterator().next();
if (assignedQueue != null) {
job.setQueueName(assignedQueue);
}
// Write job.xml
String jobFile = MRApps.getJobFile(conf, user,
TypeConverter.fromYarn(job.getID()));
LOG.info("Writing job conf to " + jobFile);
new File(jobFile).getParentFile().mkdirs();
conf.writeXml(new FileOutputStream(jobFile));
return job;
}
{code}
At first, user is "yuling.sh", but the UGI will be set to attemptId at "start();", after that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.
> TestRMContainerAllocator#testReportedAppProgress Failed
> -------------------------------------------------------
>
> Key: MAPREDUCE-5749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5749
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: shenhong
>
> When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed with message:
> {code}
> Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml does not exist
> {code}
> If fact, the job.xml exits in:
> /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
> See the following code:
> {code}
> public Job submit(Configuration conf, boolean mapSpeculative,
> boolean reduceSpeculative) throws Exception {
> String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
> .getCurrentUser().getShortUserName());
> conf.set(MRJobConfig.USER_NAME, user);
> conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
> conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
> // TODO: fix the bug where the speculator gets events with
> // not-fully-constructed objects. For now, disable speculative exec
> conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
> conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);
> init(conf);
> start();
> DefaultMetricsSystem.shutdown();
> Job job = getContext().getAllJobs().values().iterator().next();
> if (assignedQueue != null) {
> job.setQueueName(assignedQueue);
> }
> // Write job.xml
> String jobFile = MRApps.getJobFile(conf, user,
> TypeConverter.fromYarn(job.getID()));
> LOG.info("Writing job conf to " + jobFile);
> new File(jobFile).getParentFile().mkdirs();
> conf.writeXml(new FileOutputStream(jobFile));
> return job;
> }
> {code}
> At first, user is "yuling.sh", but the UGI will be set to attemptId at "start();", after that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)