[jira] [Commented] (SPARK-2007) Spark on YARN picks up hadoop log4j.properties even if SPARK_LOG4J_CONF is set

["Issac Buenrostro (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017160#comment-14017160 ] 

Issac Buenrostro commented on SPARK-2007:
-----------------------------------------

This can be solved by changing the name of the local resource pointing to the log4j configuration (for example to spark.log4j.properties). In a local version of Spark I have implemented this fix.

Other options would be to name the local resource with the same name as the original file, or make this name configurable.

> Spark on YARN picks up hadoop log4j.properties even if SPARK_LOG4J_CONF is set
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-2007
>                 URL: https://issues.apache.org/jira/browse/SPARK-2007
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Issac Buenrostro
>
> When running Spark on YARN, with SPARK_LOG4J_CONF environment variable set, containers might still pick up a different log4j configuration.
> spark-yarn will create a local resource in YARN called log4j.properties, which is included in the classpath. However, there might be other log4j.properties files in the classpath which might be chosen instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)