[spark] branch branch-2.4 updated: [SPARK-30263][CORE] Don't log potentially sensitive value of non-Spark properties ignored in spark-submit

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new f0d6989  [SPARK-30263][CORE] Don't log potentially sensitive value of non-Spark properties ignored in spark-submit
f0d6989 is described below

commit f0d69890373652ad0e28996b728761757e8dffdd
Author: Sean Owen <sr...@gmail.com>
AuthorDate: Sat Dec 14 13:13:54 2019 -0800

    [SPARK-30263][CORE] Don't log potentially sensitive value of non-Spark properties ignored in spark-submit
    
    ### What changes were proposed in this pull request?
    
    The value of non-Spark config properties ignored in spark-submit is no longer logged.
    
    ### Why are the changes needed?
    
    The value isn't really needed in the logs, and could contain potentially sensitive info. While we can redact the values selectively too, I figured it's more robust to just not log them at all here, as the values aren't important in this log statement.
    
    ### Does this PR introduce any user-facing change?
    
    Other than the change to logging above, no.
    
    ### How was this patch tested?
    
    Existing tests
    
    Closes #26893 from srowen/SPARK-30263.
    
    Authored-by: Sean Owen <sr...@gmail.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit 46e950bea883b98cd3beb7bd637bffe522656435)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index 0998757..974a0b7 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -138,10 +138,10 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
    * Remove keys that don't start with "spark." from `sparkProperties`.
    */
   private def ignoreNonSparkProperties(): Unit = {
-    sparkProperties.foreach { case (k, v) =>
+    sparkProperties.keys.foreach { k =>
       if (!k.startsWith("spark.")) {
         sparkProperties -= k
-        logWarning(s"Ignoring non-spark config property: $k=$v")
+        logWarning(s"Ignoring non-Spark config property: $k")
       }
     }
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org