[jira] [Created] (MAPREDUCE-6450) mapreduce.job.log4j-properties-file does not work for files on classpath

["Ben Podgursky (JIRA)" <ji...@apache.org>]

Ben Podgursky created MAPREDUCE-6450:
----------------------------------------

             Summary: mapreduce.job.log4j-properties-file does not work for files on classpath
                 Key: MAPREDUCE-6450
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6450
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Ben Podgursky
            Priority: Minor


JobSubmitter.addLog4jToDistributedCache checks whether MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE is set, and tries to put it in the distributed cache if it is set:

{code}
    String log4jPropertyFile =
        conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
    if (!log4jPropertyFile.isEmpty()) {
      short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10);
      copyLog4jPropertyFile(job, jobSubmitDir, replication);
{code}

This throws an exception inside JobSubmitter.validateFilePath if the file does not exist on the local filesystem, but is instead on the classpath.  This is unnecessarily restrictive, because later on in MRApps.addLog4jSystemProperties, the actual configuration of -Dlog4j.configuration works fine if the file is on the classpath and not a physical file:

{code}
    String log4jPropertyFile =
        conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
    if (log4jPropertyFile.isEmpty()) {
      vargs.add("-Dlog4j.configuration=container-log4j.properties");
{code}

(the default file container-log4j.properties is on the classpath anyway)

This is a nuisance because we have a bunch of projects and it's much easier to distribute shared configuration files via jars and dependencies than raw files.  It's not a blocker because I'm able to work around it by setting -Dlog4j.configuration in yarn.app.mapreduce.am.command-opts, but it's certainly hackier. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)