[jira] [Created] (MAPREDUCE-7229) Job History Server should ignore continuing move done_intermediate files when rename failed

["Morty Zhong (JIRA)" <ji...@apache.org>]

Morty Zhong created MAPREDUCE-7229:
--------------------------------------

             Summary: Job History Server should ignore continuing move done_intermediate files when rename failed
                 Key: MAPREDUCE-7229
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7229
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 3.1.2
            Reporter: Morty Zhong


{code:java}
private void moveToDoneNow(final Path src, final Path target)
    throws IOException {
  LOG.info("Moving " + src.toString() + " to " + target.toString());
  try {
    intermediateDoneDirFc.rename(src, target, Options.Rename.NONE);
  } catch (FileNotFoundException e) {
    if (doneDirFc.util().exists(target)) {
      LOG.info("Source file " + src.toString() + " not found, but target "
          + "file " + target.toString() + " already exists. Move already "
          + "happened.");
    } else {
      throw e;
    }
  }
}
{code}
if files in done_intermediate dir were already deleted(deleted manually), not move to done dir. Job history will infinitely rename these files later, and throw exp always.This will cause newly created files in done_intermediate dir can not be moved. Later, when scan done_intermediate dir, FileContext.listStatus(done_intermediate) will so slow, and gc problem



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org