[jira] [Comment Edited] (MAPREDUCE-6052) Support overriding log4j.properties per job

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193562#comment-14193562 ] 

Vinod Kumar Vavilapalli edited comment on MAPREDUCE-6052 at 11/1/14 11:20 PM:
------------------------------------------------------------------------------

I am reopening this JIRA for few issues, please post an addendum patch.

h4. Code
 - Users can use URI fragments in distributed-cache - for e.g. hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we do for regular dist-cache files.
 - MRApps.addLog4jSystemProperties(): If the user passes his own log4j file, setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS and logLevel may not matter at all. But let's set them as you do now, just leave a code comment that setting them may or may not really take effect.
 - job.setWorkingDirectory() getting called repeatedly?
 - For the null-scheme input, "File " + file + " does not exist." -> "File " + file + " does not exist on the local file-system."
 - Delete this log statement that you might have added for your testing: LOG.debug("default FileSystem: " + jtFs.getUri());
 - What happens if the user passes a relative path for MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in the documentation too.

h4. Documentation
 - Document the new config in mapred-default.xml
 - Mention in that documentation that if no-scheme is given in the path, it defaults to a log4j file on the local FS.
 - Modify the documentation of log-level configs to say that if you override to have your own log4j.properties file, the log-level configs may not work


was (Author: vinodkv):
I am reopening this JIRA for few minor issues, please post an addendum patch.

h4. Code
 - Users can use URI fragments in distributed-cache - for e.g. hdfs://a/b/c#symlink. Either we should explicitly reject fragments or support them. See MRApps.parseDistributedCacheArtifacts() for similar changes that we do for regular dist-cache files.
 - MRApps.addLog4jSystemProperties(): If the user passes his own log4j file, setting YARN_APP_CONTAINER_LOG_SIZE, YARN_APP_CONTAINER_LOG_BACKUPS and logLevel may not matter at all. But let's set them as you do now, just leave a code comment that setting them may or may not really take effect.
 - job.setWorkingDirectory() getting called repeatedly?
 - For the null-scheme input, "File " + file + " does not exist." -> "File " + file + " does not exist on the local file-system."
 - Delete this log statement that you might have added for your testing: LOG.debug("default FileSystem: " + jtFs.getUri());
 - What happens if the user passes a relative path for MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE? I think it works. If so, let's put this in the documentation too.

h4. Documentation
 - Document the new config in mapred-default.xml
 - Mention in that documentation that if no-scheme is given in the path, it defaults to a log4j file on the local FS.
 - Modify the documentation of log-level configs to say that if you override to have your own log4j.properties file, the log-level configs may not work

> Support overriding log4j.properties per job
> -------------------------------------------
>
>                 Key: MAPREDUCE-6052
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.0
>            Reporter: Junping Du
>            Assignee: Junping Du
>             Fix For: 2.6.0
>
>         Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch, MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch
>
>
> For current MR application, the "log4j.configuration" is hard coded to container-log4j.properties within each node. We still need flexibility to override it per job like what we do in MRV1.
> {code}
>   public static void addLog4jSystemProperties(
>       String logLevel, long logSize, int numBackups, List<String> vargs) {
>     vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)