Fwd: How to pass system properties in spark ?

[Ashwin Shankar <as...@gmail.com>]

Trying spark-dev mailing list to see if anyone knows.

---------- Forwarded message ----------
From: Ashwin Shankar <as...@gmail.com>
Date: Wed, Jun 3, 2015 at 5:38 PM
Subject: How to pass system properties in spark ?
To: "user@spark.apache.org" <us...@spark.apache.org>


Hi,
I'm trying to use property substitution in my log4j.properties, so that
I can choose where to write spark logs at runtime.
The problem is that, system property passed to spark shell
doesn't seem to getting propagated to log4j.

*Here is log4j.properites(partial) with a parameter 'spark.log.path' :*
log4j.appender.logFile=org.apache.log4j.FileAppender
log4j.appender.logFile.File=*${spark.log.path}*
log4j.appender.logFile.layout=org.apache.log4j.PatternLayout
log4j.appender.logFile.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n

*Here is how I pass the 'spark.log.path' variable on command line :*
$spark-shell --conf
spark.driver.extraJavaOptions="-Dspark.log.path=/tmp/spark.log"

I also tried :
$spark-shell -Dspark.log.path=/tmp/spark.log

*Result : */tmp*/*spark.log not getting created when I run spark.

Any ideas why this is happening ?

*When I enable log4j debug I see that following :*
log4j: Setting property [file] to [].
log4j: setFile called: , true
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:  (No such file or directory)
at java.io.FileOutputStream.open(Native Method)

-- 
Thanks,
Ashwin





-- 
Thanks,
Ashwin