You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by markgrover <gi...@git.apache.org> on 2017/04/22 00:40:40 UTC

[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

GitHub user markgrover opened a pull request:

    https://github.com/apache/spark/pull/17725

    [SPARK-20435][CORE] More thorough redaction of sensitive information

    This change does a more thorough redaction of sensitive information from logs and UI
    Add unit tests that ensure that no regressions happen that leak sensitive information to the logs.
    
    Previously redaction logic was only checking if the key matched the secret regex pattern, it'd redact it's value. That worked for most cases. However, in the above case, the key (sun.java.command) doesn't tell much, so the value needs to be searched. This PR expands the check to check for values as well.
    
    
    ## How was this patch tested?
    
    New unit tests added that ensure that no sensitive information is present in the event logs or the yarn logs.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/markgrover/spark spark-20435

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/17725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17725
    
----
commit 2f5148a2e37d8d36006fb297b28a9e8c21a0026b
Author: Mark Grover <ma...@apache.org>
Date:   2017-04-22T00:24:30Z

    [SPARK-20435][CORE] More thorough redaction of sensitive information from logs/UI, more unit tests

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Few decisions that I made here:
    * I considered if just `sun.java.command` property should be checked and redacted but that seemed to specific and likely a bandaid to the current problem, not a long-term solution, so decided against doing it.
    * Redaction for the `SparkListenerEnvironmentUpdate` event was solely being done on `Spark Properties`, while `sun.java.command` is a part of `System Properties`. I considered doing redaction for `System Properties` in addition to `Spark Properties` (that would have gone somewhere around [here](https://github.com/apache/spark/pull/17725/files#diff-e4a5a68c15eed95d038acfed84b0b66aL258)) but decided against it because that would have even more hardcoding and I didn't see why these 2 special kinds of properties are special enough to be redacted but the rest of them. So, decided to redact information from all kinds of properties.
    * One way to redact the property value would have been to redact the minimum possible set from the value while keeping the rest of the value intact. For example, if the following were the unredacted case:
    `"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password --conf spark.other.property=2"`
    One option for the redacted output could have been:
    `"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=*********(redacted) --conf spark.other.property=2"`
    However, such a redaction is very hard to maintain. For example, we would had to take the current regex (which is `(?i)secret|password` by default and add matchers to it like so `(?i)secret|password` like `"("+SECRET_REDACTION_DEFAULT+"[^ ]*=)[^ ]*"`. That would allow us to squeeze out and replaced just the matched portion. But this all seemed very fragile and even worse when the user supplies a non-default regex so I decided it was easiest to simply replace the entire value, even though only a small part of it contained `secret` or `password` in it.
    * One thing which I didn't explicitly check was the performance implications of this change. The reason I bring this up is because, previously we were comparing keys with a regex, now if the key 
    doesn't match, we match the value with the regex. So, in the worst case, we are twice as many regex matches as before. Also, before we were simply doing regex matching on `Spark Properties`, now we do them on all properties - `Spark Properties`, `System Properties`, `JVM Properties` and `Classpath Properties`. I don't think this should have a big performance impact so I didn't invest time in it, mentioning here in interest of full disclosure.
    
    Thanks in advance for reviewing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17725#discussion_r113091533
  
    --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
    @@ -404,6 +407,37 @@ class SparkSubmitSuite
         runSparkSubmit(args)
       }
     
    +  test("launch simple application with spark-submit with redaction") {
    +    val testDir = Utils.createTempDir()
    +    testDir.deleteOnExit()
    +    val testDirPath = new Path(testDir.getAbsolutePath())
    +    val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
    +    val fileSystem = Utils.getHadoopFileSystem("/",
    +      SparkHadoopUtil.get.newConfiguration(new SparkConf()))
    +    try {
    +      val args = Seq(
    +        "--class", SimpleApplicationTest.getClass.getName.stripSuffix("$"),
    +        "--name", "testApp",
    +        "--master", "local",
    +        "--conf", "spark.ui.enabled=false",
    +        "--conf", "spark.master.rest.enabled=false",
    +        "--conf", "spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password",
    +        "--conf", "spark.eventLog.enabled=true",
    +        "--conf", "spark.eventLog.testing=true",
    +        "--conf", s"spark.eventLog.dir=${testDirPath.toUri.toString}",
    +        "--conf", "spark.hadoop.fs.defaultFS=unsupported://example.com",
    +        unusedJar.toString)
    +      runSparkSubmit(args)
    +      val listStatuses = fileSystem.listStatus(testDirPath)
    --- End diff --
    
    s/listStatuses/something else.
    
    Use list, statuses, statusList, but "listStatuses" doesn't parse for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76143/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76051/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merging to master / 2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76156/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76054/testReport)** for PR 17725 at commit [`0603686`](https://github.com/apache/spark/commit/06036867d96350a51e180565782ffee1515fbea4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17725#discussion_r113091235
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
    @@ -252,11 +252,17 @@ private[spark] class EventLoggingListener(
     
       private[spark] def redactEvent(
           event: SparkListenerEnvironmentUpdate): SparkListenerEnvironmentUpdate = {
    -    // "Spark Properties" entry will always exist because the map is always populated with it.
    -    val redactedProps = Utils.redact(sparkConf, event.environmentDetails("Spark Properties"))
    -    val redactedEnvironmentDetails = event.environmentDetails +
    -      ("Spark Properties" -> redactedProps)
    -    SparkListenerEnvironmentUpdate(redactedEnvironmentDetails)
    +    // environmentDetails maps a string descriptor to a set of properties
    +    // Similar to:
    +    // "JVM Information" -> jvmInformation,
    +    // "Spark Properties" -> sparkProperties,
    +    // ...
    +    // where jvmInformation, sparkProperties, etc. are sequence of tuples.
    +    // We go through the various  of properties and redact sensitive information from them.
    +    val redactedProps = event.environmentDetails.map{
    --- End diff --
    
    `.map { case (name, props) =>`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    This time it finished on regular time (2h 29m) but failed a test. I ran that test locally (org.apache.spark.streaming.ReceiverSuite's 'receiver life cycle') and it passed for me, so for mostly laziness, I am going to issue another run here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76054/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76143/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
     * This patch **fails from timeout after a configured wait of \`250m\`**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76194/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76194/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76055/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17725#discussion_r113091328
  
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -2606,8 +2606,22 @@ private[spark] object Utils extends Logging {
       }
     
       private def redact(redactionPattern: Regex, kvs: Seq[(String, String)]): Seq[(String, String)] = {
    +    // If the sensitive information regex matches with either the key or the value, redact the value
    +    // While the original intent was to only redact the value if the key matched with the regex,
    +    // we've found that especially in verbose mode, the value of the property may contain sensitive
    +    // information like so:
    +    // "sun.java.command":"org.apache.spark.deploy.SparkSubmit ... \
    +    // --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password ...
    +    //
    +    // And, in such cases, simply searching for the sensitive information regex in the key name is
    +    // not sufficient. The values themselves have to be searched as well and redacted if matched.
    +    // This does mean we may be accounting more false positives - for example, if the value of an
    +    // arbitrary property contained the term 'password', we may redact the value from the UI and
    +    // logs. In order to work around it, user would have to make the spark.redaction.regex property
    +    // more specific.
         kvs.map { kv =>
    --- End diff --
    
    Since you're looking at values now...
    
    `.map { case (key, value) =>`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/17725


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76051/testReport)** for PR 17725 at commit [`2f5148a`](https://github.com/apache/spark/commit/2f5148a2e37d8d36006fb297b28a9e8c21a0026b).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Kicking off another run here while I am running a run locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76051/testReport)** for PR 17725 at commit [`2f5148a`](https://github.com/apache/spark/commit/2f5148a2e37d8d36006fb297b28a9e8c21a0026b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76194/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76156/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76156/testReport)** for PR 17725 at commit [`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #17725: [SPARK-20435][CORE] More thorough redaction of se...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17725#discussion_r113091382
  
    --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
    @@ -404,6 +407,37 @@ class SparkSubmitSuite
         runSparkSubmit(args)
       }
     
    +  test("launch simple application with spark-submit with redaction") {
    +    val testDir = Utils.createTempDir()
    +    testDir.deleteOnExit()
    +    val testDirPath = new Path(testDir.getAbsolutePath())
    +    val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
    +    val fileSystem = Utils.getHadoopFileSystem("/",
    +      SparkHadoopUtil.get.newConfiguration(new SparkConf()))
    +    try {
    +      val args = Seq(
    +        "--class", SimpleApplicationTest.getClass.getName.stripSuffix("$"),
    +        "--name", "testApp",
    +        "--master", "local",
    +        "--conf", "spark.ui.enabled=false",
    +        "--conf", "spark.master.rest.enabled=false",
    +        "--conf", "spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password",
    +        "--conf", "spark.eventLog.enabled=true",
    +        "--conf", "spark.eventLog.testing=true",
    +        "--conf", s"spark.eventLog.dir=${testDirPath.toUri.toString}",
    +        "--conf", "spark.hadoop.fs.defaultFS=unsupported://example.com",
    +        unusedJar.toString)
    +      runSparkSubmit(args)
    +      val listStatuses = fileSystem.listStatus(testDirPath)
    +      val logData = EventLoggingListener.openEventLog(listStatuses.last.getPath, fileSystem)
    +      Source.fromInputStream(logData).getLines().foreach {
    --- End diff --
    
    `.foreach { line =>`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76114/testReport)** for PR 17725 at commit [`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76054/testReport)** for PR 17725 at commit [`0603686`](https://github.com/apache/spark/commit/06036867d96350a51e180565782ffee1515fbea4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76114 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76114/testReport)** for PR 17725 at commit [`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Thanks Marcelo!
    
    On Apr 26, 2017 5:06 PM, "Marcelo Vanzin" <no...@github.com> wrote:
    
    > Merging to master / 2.2.
    >
    > \u2014
    > You are receiving this because you authored the thread.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/spark/pull/17725#issuecomment-297575053>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/ABoVi9s2uyakNm-81qWP3aBpRkFHtm3Jks5rz9vugaJpZM4NE66l>
    > .
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76114/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76143/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Hmm, this is weird, it ran for 4h10 mins and got killed due to a timeout. Looking...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by markgrover <gi...@git.apache.org>.
Github user markgrover commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    Finally it's passing!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17725
  
    **[Test build #76055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76055/testReport)** for PR 17725 at commit [`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org