You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mahesh Ambule (JIRA)" <ji...@apache.org> on 2017/08/11 19:23:00 UTC
[jira] [Comment Edited] (SPARK-21711) spark-submit command should
accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123889#comment-16123889 ]
Mahesh Ambule edited comment on SPARK-21711 at 8/11/17 7:22 PM:
----------------------------------------------------------------
Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command
but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit
command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am
more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file.
I went through spark-submit and related script files and found that spark-submit does not provide the option
to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file.
This 'spark-class' file gets invoked by spark-submit.
_
"$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_
Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit.
There is no way to pass parameters to client java JVM.
Is my understanding correct? And can the option be provided to pass parameters to client Java JVM?
was (Author: mahesh_ambule):
sean owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command
but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit
command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am
more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file.
I went through spark-submit and related script files and found that spark-submit does not provide the option
to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file.
This 'spark-class' file gets invoked by spark-submit.
_
"$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_
Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit.
There is no way to pass parameters to client java JVM.
Is my understanding correct? And can the option be provided to pass parameters to client Java JVM?
> spark-submit command should accept log4j configuration parameters for spark client logging.
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-21711
> URL: https://issues.apache.org/jira/browse/SPARK-21711
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 1.6.0, 2.1.0
> Reporter: Mahesh Ambule
> Priority: Minor
> Attachments: spark-submit client logs.txt
>
>
> Currently, log4j properties can be specified in spark 'conf' directory in log4j.properties file.
> The spark-submit command can override these log4j properties for driver and executors.
> But it can not override these log4j properties for *spark client * application.
> The user should be able to pass log4j properties for spark client using the spark-submit command.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org