[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881773#comment-15881773 ] 

Apache Spark commented on SPARK-19720:
--------------------------------------

User 'markgrover' has created a pull request for this issue:
https://github.com/apache/spark/pull/17047

> Redact sensitive information from SparkSubmit console output
> ------------------------------------------------------------
>
>                 Key: SPARK-19720
>                 URL: https://issues.apache.org/jira/browse/SPARK-19720
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.2.0
>            Reporter: Mark Grover
>
> SPARK-18535 took care of redacting sensitive information from Spark event logs and UI. However, it intentionally didn't bother redacting the same sensitive information from SparkSubmit's console output because it was on the client's machine, which already had the sensitive information on disk (in spark-defaults.conf) or on terminal (spark-submit command line).
> However, it seems now that it's better to redact information from SparkSubmit's console output as well because orchestration software like Oozie usually expose SparkSubmit's console output via a UI. To make matters worse, Oozie, in particular, always sets the {{--verbose}} flag on SparkSubmit invocation, making the sensitive information readily available in its UI (see [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248] here).
> This is a JIRA for tracking redaction of sensitive information from SparkSubmit's console output.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org