spark submit with logs and kerberos

[Juan Pablo Briganti <ju...@globant.com>]

Hi!

I have a question about logs and have not seen the answer through internet.
I have a spark submit process and I configure a custom log configuration to
it using the next params:
--conf
"spark.executor.extraJavaOptions=-Dlog4j.configuration=customlog4j.properties"

--driver-java-options '-Dlog4j.configuration=customlog4j.properties'
So far is working great, but now I have to add kerberos params like this:
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true
--keytab <keytab path>
--principal <us...@domain>
When I add those params (and in particular, --principal) my custom log file
starts being ignored and the default spark log file is being used insted.
Change the default spark configuration file or path is a little difficult
for us since we can't access that cluster.
Is there any way to keep using my custom log configuration the same way? I
am doing something wrong?

Thanks for the help!





-- 
*Juan Pablo Briganti* | Data Architect
*GLOBANT* | AR: +54 11 4109 1700 ext. 19508 | US: +1 877 215 5230 ext. 19508
|
[image: Facebook] <https://www.facebook.com/Globant> [image: Twitter]
<http://www.twitter.com/globant> [image: Youtube]
<http://www.youtube.com/Globant> [image: Linkedin]
<http://www.linkedin.com/company/globant> [image: Pinterest]
<http://pinterest.com/globant/> [image: Globant] <http://www.globant.com>

-- 


The information contained in this e-mail may be confidential. It has been 
sent for the sole use of the intended recipient(s). If the reader of this 
message is not an intended recipient, you are hereby notified that any 
unauthorized review, use, disclosure, dissemination, distribution or 
copying of this communication, or any of its contents, 
is strictly prohibited. If you have received it by mistake please let us 
know by e-mail immediately and delete it from your system. Many thanks.

 

La información contenida en este mensaje puede ser confidencial. Ha sido 
enviada para el uso exclusivo del destinatario(s) previsto. Si el lector de 
este mensaje no fuera el destinatario previsto, por el presente queda Ud. 
notificado que cualquier lectura, uso, publicación, diseminación, 
distribución o copiado de esta comunicación o su contenido está 
estrictamente prohibido. En caso de que Ud. hubiera recibido este mensaje 
por error le agradeceremos notificarnos por e-mail inmediatamente y 
eliminarlo de su sistema. Muchas gracias.