You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by steveloughran <gi...@git.apache.org> on 2016/06/09 16:21:17 UTC
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/13579
[SPARK-15844] [core] HistoryServer doesn't come up if spark.authenticate = true
## What changes were proposed in this pull request?
During history server startup, the spark configuration is examined. If security.authentication is
set, a dummy key is added, so that {{SecurityManager}} can be created.
It'd be more elegant to actually remove the unneeded option, but the environment variable complicates
things: you can't unset/override the env var in the spark configuration.
## How was this patch tested?
New Unit test sets the `spark.authenticate` property to true, tries to create a security manager via a new package-private method `HistoryServer.createSecurityManager(SparkConf)`. This method is also used in `HistoryServer.main`, so the patching will also take place there.
Ideally, the code inside the main() method would be split into instantiate and "spin awaiting completion". That would guarantee that the codepath used in the production history server launch is being followed. If people would prefer that, I'll happily do it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark history/SPARK-15844-security
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13579.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13579
----
commit 3c1840ee0d0e0e45e08c4ef1cc9b8b84075b271e
Author: Steve Loughran <st...@apache.org>
Date: 2016-06-09T16:09:54Z
SPARK-15844 HistoryServer doesn't come up if spark.authenticate = true
commit 26d1ad243c9c6db227a13303c3781546634794db
Author: Steve Loughran <st...@apache.org>
Date: 2016-06-09T16:17:41Z
[SPARK-15844] HistoryServer doesn't come up if spark.authenticate = true; making new (visible for test) method private to the package
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91546615
--- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
}
+ /**
+ * Verify that the security manager needed for the history server can be instantiated
+ * even if spark.authenticate is set
+ */
+ test("SecurityManagerStartsWithSecureShuffle") {
+ val conf = new SparkConf()
+ .set("spark.testing", "true")
--- End diff --
fixed, along with IDE settings
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70029/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #66657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66657/consoleFull)** for PR 13579 at commit [`6701922`](https://github.com/apache/spark/commit/67019225dac688d882817af92cbee05d5ddaf90e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #60238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60238/consoleFull)** for PR 13579 at commit [`26d1ad2`](https://github.com/apache/spark/commit/26d1ad243c9c6db227a13303c3781546634794db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #70023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70023/consoleFull)** for PR 13579 at commit [`033f21e`](https://github.com/apache/spark/commit/033f21e8799584c89d5bbac8692765624695b5b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69864/consoleFull)** for PR 13579 at commit [`ad35cd9`](https://github.com/apache/spark/commit/ad35cd90f269b973478b074a6d49dd2f2cc90a69).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69824/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
LGTM, will merge once tests finish. (The interesting test is passing anyway.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
stylecheck; unexpected, as I thought I'd run them in the `mvn install` of the module.
```
[error] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala:352:2:
Insert a space after the start of the comment
```
Comment is
```scala
/**
* Verify that the security manager needed for the history server can be instantiated
* when `spark.authenticate` is `true`, rather than raise an `IllegalArgumentException`.
**/
```
...the trailing **/ is the problem. Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91427789
--- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
}
+ /**
+ * Verify that the security manager needed for the history server can be instantiated
+ * even if spark.authenticate is set
+ */
+ test("SecurityManagerStartsWithSecureShuffle") {
+ val conf = new SparkConf()
+ .set("spark.testing", "true")
+ .set(SecurityManager.SPARK_AUTH_CONF, "true")
+ HistoryServer.createSecurityManager(conf)
--- End diff --
Can you add a comment explaining that the SecurityManager would throw an exception if spark.authenticate was true? (I think that'd be better than the comment you have before the test.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
I see: you want the HS to set it? Yeah, that would work. I'll change this patch accordingly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69868/consoleFull)** for PR 13579 at commit [`f375682`](https://github.com/apache/spark/commit/f3756820e9c530be4be7ef3cc44e90adae66b598).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91427614
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala ---
@@ -289,6 +289,30 @@ object HistoryServer extends Logging {
while(true) { Thread.sleep(Int.MaxValue) }
}
+ /**
+ * Create a security manager.
+ * This includes any fixup of the configurations needed to produce a security manager
+ * capable of starting the History Server.
+ * @param config configuration for the SecurityManager constructor
+ * @return
+ */
+ private[history] def createSecurityManager(config: SparkConf): SecurityManager = {
+ patchSecuritySettings(config)
+ new SecurityManager(config)
+ }
+
+ /**
+ * Fix up the configuration of a spark configuration so that the security manager will
+ * always start.
+ * @param config configuration to be used in a SecurityManager constructor
+ */
+ private def patchSecuritySettings(config: SparkConf): Unit = {
--- End diff --
Could you just inline this method? That avoids adding a method with a single call, and pretty much a copy of the whole scaladoc of the other method you're adding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69868/consoleFull)** for PR 13579 at commit [`f375682`](https://github.com/apache/spark/commit/f3756820e9c530be4be7ef3cc44e90adae66b598).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #70023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70023/consoleFull)** for PR 13579 at commit [`033f21e`](https://github.com/apache/spark/commit/033f21e8799584c89d5bbac8692765624695b5b3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69823/consoleFull)** for PR 13579 at commit [`01a0220`](https://github.com/apache/spark/commit/01a02200568696dd467a35fb584496dd0cd900cb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #70029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70029/consoleFull)** for PR 13579 at commit [`033f21e`](https://github.com/apache/spark/commit/033f21e8799584c89d5bbac8692765624695b5b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69824/consoleFull)** for PR 13579 at commit [`3b3db69`](https://github.com/apache/spark/commit/3b3db69b464fdf700f271e751e2e03244c01cda3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
Test failure is pretty unlikely to be related. Looks more like a timing or timeout problem.
```
org.apache.spark.rdd.AsyncRDDActionsSuite.async failure handling
Failing for the past 1 build (Since Failed#69868 )
Took 10 sec.
Error Message
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to failAfter did not complete within 10 seconds.
Stacktrace
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to failAfter did not complete within 10 seconds.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
The main problem here is in a cluster where auth is turned on globally, the HS gets really confused: it's enabled but doesn't have any secrets. This patch sets things up so that even in a cluster-wide security enabled, the history server doesn't fall over
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
That doesn't answer my question, though. Why can't the history server explicitly set that value to false after all the settings having been read from wherever?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69823/consoleFull)** for PR 13579 at commit [`01a0220`](https://github.com/apache/spark/commit/01a02200568696dd467a35fb584496dd0cd900cb).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #60238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60238/consoleFull)** for PR 13579 at commit [`26d1ad2`](https://github.com/apache/spark/commit/26d1ad243c9c6db227a13303c3781546634794db).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69868/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69864/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
While tests are running, can you update the description? It mentions env vars, which is misleading since they don't have anything to do with this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #66657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66657/consoleFull)** for PR 13579 at commit [`6701922`](https://github.com/apache/spark/commit/67019225dac688d882817af92cbee05d5ddaf90e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
@steveloughran are you going to update this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91546575
--- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
}
+ /**
+ * Verify that the security manager needed for the history server can be instantiated
+ * even if spark.authenticate is set
+ */
+ test("SecurityManagerStartsWithSecureShuffle") {
+ val conf = new SparkConf()
+ .set("spark.testing", "true")
+ .set(SecurityManager.SPARK_AUTH_CONF, "true")
+ HistoryServer.createSecurityManager(conf)
--- End diff --
done. You know, this test could be merged into `"incomplete apps get refreshed"`, just by setting the security option and mentioning why it matters in a comment. I've patched that test to also use `HistoryServer.createSecurityManager` for consistency. it would save a fractional amount of test time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
I might be missing something, but why doesn't the following work?
sparkConf.set(SecurityManager.SPARK_AUTH_CONF, "false")
That value is never read from the environment as far as I can tell.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69864/consoleFull)** for PR 13579 at commit [`ad35cd9`](https://github.com/apache/spark/commit/ad35cd90f269b973478b074a6d49dd2f2cc90a69).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #69824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69824/consoleFull)** for PR 13579 at commit [`3b3db69`](https://github.com/apache/spark/commit/3b3db69b464fdf700f271e751e2e03244c01cda3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70023/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
yeah, I've just got so many other distractions. Let me do it again while tests run in different windows
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60238/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13579
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91427795
--- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
}
+ /**
+ * Verify that the security manager needed for the history server can be instantiated
+ * even if spark.authenticate is set
+ */
+ test("SecurityManagerStartsWithSecureShuffle") {
+ val conf = new SparkConf()
+ .set("spark.testing", "true")
--- End diff --
nit: indented too far
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91548419
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala ---
@@ -289,6 +289,30 @@ object HistoryServer extends Logging {
while(true) { Thread.sleep(Int.MaxValue) }
}
+ /**
+ * Create a security manager.
+ * This includes any fixup of the configurations needed to produce a security manager
+ * capable of starting the History Server.
+ * @param config configuration for the SecurityManager constructor
+ * @return
+ */
+ private[history] def createSecurityManager(config: SparkConf): SecurityManager = {
+ patchSecuritySettings(config)
+ new SecurityManager(config)
+ }
+
+ /**
+ * Fix up the configuration of a spark configuration so that the security manager will
+ * always start.
+ * @param config configuration to be used in a SecurityManager constructor
+ */
+ private def patchSecuritySettings(config: SparkConf): Unit = {
--- End diff --
I've inlined it, which highlights something else. if the debug message is viewed as unimportant, the if() clause can be pulled completely, and the security flag *always* set to false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66657/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13579
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13579
**[Test build #70029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70029/consoleFull)** for PR 13579 at commit [`033f21e`](https://github.com/apache/spark/commit/033f21e8799584c89d5bbac8692765624695b5b3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/13579
thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13579#discussion_r91587654
--- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
}
+ /**
+ * Verify that the security manager needed for the history server can be instantiated
+ * when `spark.authenticate` is `true`, rather than raise an `IllegalArgumentException`.
+ */
+ test("SecurityManagerStartsWithSecureShuffle") {
--- End diff --
nit: can you write a test description like the others in the class, instead of this camel-cased string?
Also spark.authenticate is not (just) for secure shuffle, so the string itself is not really accurate. Just mention spark.authenticate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69823/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org