You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Maysam Yabandeh (JIRA)" <ji...@apache.org> on 2016/01/14 19:55:39 UTC
[jira] [Created] (MAPREDUCE-6607) .staging dir is not cleanup if
mapreduce.task.files.preserve.failedtask or
mapreduce.task.files.preserve.filepattern are set
Maysam Yabandeh created MAPREDUCE-6607:
------------------------------------------
Summary: .staging dir is not cleanup if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
Key: MAPREDUCE-6607
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 2.7.1
Reporter: Maysam Yabandeh
Priority: Minor
if either of the following configs are set, then .staging dir is not cleaned up:
* mapreduce.task.files.preserve.failedtask
* mapreduce.task.files.preserve.filepattern
The former was supposed to keep only .staging of failed tasks and the latter was supposed to be used only if that task name matches against the specified regular expression.
{code}
protected boolean keepJobFiles(JobConf conf) {
return (conf.getKeepTaskFilesPattern() != null || conf
.getKeepFailedTaskFiles());
}
{code}
{code}
public void cleanupStagingDir() throws IOException {
/* make sure we clean the staging files */
String jobTempDir = null;
FileSystem fs = getFileSystem(getConfig());
try {
if (!keepJobFiles(new JobConf(getConfig()))) {
jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
if (jobTempDir == null) {
LOG.warn("Job Staging directory is null");
return;
}
Path jobTempDirPath = new Path(jobTempDir);
LOG.info("Deleting staging directory " + FileSystem.getDefaultUri(getConfig()) +
" " + jobTempDir);
fs.delete(jobTempDirPath, true);
}
} catch(IOException io) {
LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)