You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by scwf <gi...@git.apache.org> on 2014/08/16 05:15:52 UTC
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
GitHub user scwf opened a pull request:
https://github.com/apache/spark/pull/1980
[SPARK-2750] support https in spark web ui
https://issues.apache.org/jira/browse/SPARK-2750
enable spark web ui to support https, also compatible with old web ui(http)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwf/spark https
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1980.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1980
----
commit e8256e5278da8a8b4aaefb7638a2683e2102ab4a
Author: scwf <wa...@huawei.com>
Date: 2014-08-16T00:20:44Z
support https in spark web ui
commit 8f7cc967e6fd1b9d0b3768543afe06211e2178fa
Author: scwf <wa...@huawei.com>
Date: 2014-08-16T02:27:15Z
add unit test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493406
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
--- End diff --
nit: these lines look indented too much. They should be 4 spaces from where the "@Override" currently is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432586
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ url.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ url.append(scheme).append("://").append(server)
+ }
+ if (port > 0) {
+ url.append(':').append(port)
+ }
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ if (securePort > 0) {
--- End diff --
Hmm, yes, it a no use condition :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432532
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ url.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ url.append(scheme).append("://").append(server)
+ }
+ if (port > 0) {
+ url.append(':').append(port)
+ }
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ if (securePort > 0) {
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
+ }else {
--- End diff --
Style nit: space after the `{`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569562
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -30,7 +30,7 @@ private[spark] class WorkerInfo(
val cores: Int,
val memory: Int,
val actor: ActorRef,
- val webUiPort: Int,
+ val webUiAddress: String,
val publicAddress: String)
--- End diff --
It looks like this `publicAddress` field is never read anymore, so we should remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57796375
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21244/consoleFull) for PR 1980 at commit [`e5c87cb`](https://github.com/apache/spark/commit/e5c87cb596ab3ae16ae52458c8e1edf6d1d3647a).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56303934
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20627/consoleFull) for PR 1980 at commit [`967950f`](https://github.com/apache/spark/commit/967950f26d612383fd3e121af3284d4a2ce013c8).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18641539
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Hi @vanzin, actually here i just refer to the code of jetty 9 see(https://github.com/eclipse/jetty.project/blob/master/jetty-util/src/main/java/org/eclipse/jetty/util/URIUtil.java#L726-L733) since there is no ```newURI```method in spark jetty version(spark use jetty 8).
And L238 is for the case to handle IPv6 address, here we can remove it if unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570651
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala ---
@@ -53,6 +53,9 @@ private[spark] class WorkerArguments(args: Array[String], conf: SparkConf) {
if (System.getenv("SPARK_WORKER_DIR") != null) {
workDir = System.getenv("SPARK_WORKER_DIR")
}
+ if (conf.contains("worker.ui.port")) {
--- End diff --
This should be `spark.worker.ui.port`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570077
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
--- End diff --
Shouldn't this be configurable instead of hardcoded?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57493281
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21119/consoleFull) for PR 1980 at commit [`6c31dc7`](https://github.com/apache/spark/commit/6c31dc71a932083e158ef9c6737f33ebe9de1348).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432535
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ url.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ url.append(scheme).append("://").append(server)
+ }
+ if (port > 0) {
+ url.append(':').append(port)
+ }
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ if (securePort > 0) {
--- End diff --
Why would this port ever be negative or zero?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58636685
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21584/consoleFull)** for PR 1980 at commit [`3b01d3a`](https://github.com/apache/spark/commit/3b01d3a66d21854b5e053de89f3d301fee58b315) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432489
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ""): ServerInfo = {
val collection = new ContextHandlerCollection
- collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get("spark.ui.https.enabled", "false").toBoolean) {
+ // / If the new port wraps around, do not try a privilege port
+ val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
What if this port isn't free? It's dangerous to make assumptions about whether a particular ports is free, especially since this can cause problems in Jenkins, where we might have multiple builds running concurrently on the same machine that are contending for ports.
Could we use `Utils.startServiceOnPort` for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55345554
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20175/consoleFull) for PR 1980 at commit [`8f7cc96`](https://github.com/apache/spark/commit/8f7cc967e6fd1b9d0b3768543afe06211e2178fa).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57505464
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21129/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432549
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
--- End diff --
Similarly, this function is only called from one place and it's kind of confusing, so I'd prefer to inline this code in `newURI` (or define it as a nested block / function there).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57923018
I'd like to try this out with view ACLs / authentication enabled to see how that interacts with HTTPs. Do you have a set of instructions that I can follow to test this out? It would be great to test this on YARN, since I think that's the typical environment where people would use view ACLs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432556
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ""): ServerInfo = {
val collection = new ContextHandlerCollection
- collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get("spark.ui.https.enabled", "false").toBoolean) {
+ // / If the new port wraps around, do not try a privilege port
+ val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
Hey, note ```connect``` will be passed to ```Utils.startServiceOnPort```, so in ```Utils.startServiceOnPort``` it will try to start http service on currentPort and https service on securePort. If securePort is not free, it will try again until the maximum number of retries
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58681651
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21595/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57905663
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21294/consoleFull) for PR 1980 at commit [`baaa1ce`](https://github.com/apache/spark/commit/baaa1ce05fcb426de7d3002a5cc30f18ae119d34).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57731222
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56304928
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20627/consoleFull) for PR 1980 at commit [`967950f`](https://github.com/apache/spark/commit/967950f26d612383fd3e121af3284d4a2ce013c8).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57725678
Have added redirect from http to https and updated docs. Some directions as follows
1 There is no changes for web ui when not set ```spark.ui.https.enabled```
2 When https enabled, users no need input https url to get the right ui. Actually it's the same as http situation, users can input http url as they do in http case and the http url will redirect to a https one automatically
3 If set ```spark.ui.https.enabled```, user should(must) also config the ssl config opts
spark.ui.ssl.server.keystore.keypassword
spark.ui.ssl.server.keystore.location
spark.ui.ssl.server.keystore.password.
4 Config ```spark.ui.ssl.client.https.needAuth``` is optional, used to enable client authenticates. When set true users should config client ssl config
```spark.ui.ssl.server.truststore.location``` and ```spark.ui.ssl.server.truststore.password```.
5 ```spark.ui.ssl.server.keystore.type``` and ```spark.ui.ssl.server.truststore.type``` has default value JKS
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56306346
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20628/consoleFull) for PR 1980 at commit [`9591c9c`](https://github.com/apache/spark/commit/9591c9c5b90e0d1d65b4a48b08b53f6a3ed3e49d).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55340427
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57924007
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21303/consoleFull) for PR 1980 at commit [`a48c6fc`](https://github.com/apache/spark/commit/a48c6fc585c7fdb9c859b00f9b7d7d46836706b8).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57570323
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21165/consoleFull) for PR 1980 at commit [`a29ec86`](https://github.com/apache/spark/commit/a29ec8632cce8cb29c38d8e9e3aee2334400130b).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` println(s"Failed to load main class $childMainClass.")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58681640
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21595/consoleFull) for PR 1980 at commit [`4ae834b`](https://github.com/apache/spark/commit/4ae834b6c5aafc36d44adba4c4dbb934a63ea3ed).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by jacek-lewandowski <gi...@git.apache.org>.
Github user jacek-lewandowski commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-60174311
Btw. I was trying to do SSL for UI as well. One thing I'm afraid of is that this solution would break any recovery information for Spark Master (different signature of WorkerInfo). Also, it doesn't allow to run the cluster in mixed mode during upgrade - some nodes running older Spark and some nodes running newer Spark, because it changes the signature of messages. Anyway, this is very minor I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57479925
jenkins down...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57922537
Thanks for posting those configuration instructions. I think the problem was that I had been specifying an invalid truststore. After following your instructions, I was able to browse to the HTTP and HTTPs urls as expected and the redirect seemed to work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf closed the pull request at:
https://github.com/apache/spark/pull/1980
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493360
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
--- End diff --
Can you add a comment explaining what this method is doing? It's not obvious from reading the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569719
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -78,6 +78,7 @@ private[spark] class Worker(
var activeMasterUrl: String = ""
var activeMasterWebUiUrl : String = ""
val akkaUrl = "akka.tcp://%s@%s:%s/user/%s".format(actorSystemName, host, port, actorName)
+ var workerWebUiUrl: String = _
--- End diff --
For stylistic consistency with the surrounding code, I'd prefer if this was initialized with an empty string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18692553
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Ok, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58624892
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21584/consoleFull) for PR 1980 at commit [`3b01d3a`](https://github.com/apache/spark/commit/3b01d3a66d21854b5e053de89f3d301fee58b315).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493480
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
--- End diff --
Might be a good idea to call ` response.encodeRedirectURL()` first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56750923
`spark.ui.https.enabled` is repeated in several places; maybe a helper method somewhere to retrieve that property would be better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432538
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
--- End diff --
I'd remove this function and fold it's functionality into the above function, since it's only two lines of straightforward code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569799
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -35,6 +35,8 @@ import org.json4s.jackson.JsonMethods.{pretty, render}
import org.apache.spark.{Logging, SecurityManager, SparkConf}
import org.apache.spark.util.Utils
+import org.eclipse.jetty.server.nio.SelectChannelConnector
--- End diff --
These imports should be placed alongside the other `jetty` imports (e.g. line 32).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18006638
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
--- End diff --
thanks, pretty good advice
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-95577333
i am closing this in favor of #5664
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56301209
@JoshRosen, here i named the configuration options refer to how hadoop do(actually just use ```spark``` instead ```hadoop```). ```spark.client``` is the browser which connect to UIServer.
Now browse to ```http```ui when configure ```http``` will get error, and i will check that how to set up automatic redirect(maybe configure this in jetty )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56301104
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20622/consoleFull) for PR 1980 at commit [`c90d84e`](https://github.com/apache/spark/commit/c90d84e0c70a88742feb453c6f1517163878c003).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57796385
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21244/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57789135
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21244/consoleFull) for PR 1980 at commit [`e5c87cb`](https://github.com/apache/spark/commit/e5c87cb596ab3ae16ae52458c8e1edf6d1d3647a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55662724
Is `spark.http.policy` the best name for this configuration option? Do you think that this can be a boolean option, or are there cases for wanting to have values other than `http` and `https`? It looks like [Hadoop has a `dfs.http.policy`](http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml) option that accepts three values, HTTP_ONLY, HTTPS_ONLY, and HTTP_AND_HTTPS.
Whatever configuration options we end up using, we should add documentation for them to the [Spark Security](http://spark.apache.org/docs/latest/security.html) and [Configuration](http://spark.apache.org/docs/latest/configuration.html#spark-ui) guides.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55665149
What happens if I've configured the web UI to use `https` then attempt to browse to the `http` URL? Is it easy to set up an automatic redirect?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17569959
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
+ context.setKeyStorePath(conf.get("spark.ssl.server.keystore.location"))
+ context.setKeyStorePassword(conf.get("spark.ssl.server.keystore.password", "123456"))
+ context.setKeyStoreType(conf.get("spark.ssl.server.keystore.type", "jks"))
+
+ if (needAuth && conf.contains("spark.ssl.server.truststore.location")) {
+ context.setTrustStore(conf.get("spark.ssl.server.truststore.location"))
+ context.setTrustStorePassword(conf.get("spark.ssl.server.truststore.password"))
+ context.setTrustStoreType(conf.get("spark.ssl.server.truststore.type", "jks"))
+ }
+ connector
+ }
+
+ def getHttpPolicy(conf: SparkConf): Boolean = {
--- End diff --
This is a confusing name, since this function returns a boolean but sounds like it will return a "http policy". `isHttpsEnabled` would be a better name, although I'm not sure that we need this:
We should just be able to do
```
val https = conf.get("spark.http.policy", "") == "https"
```
instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58132338
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21360/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18656947
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Could you put that reference in the code itself, so we know where it comes from? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432581
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ""): ServerInfo = {
val collection = new ContextHandlerCollection
- collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get("spark.ui.https.enabled", "false").toBoolean) {
+ // / If the new port wraps around, do not try a privilege port
+ val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
--- End diff --
Ah, I see. Thanks for explaining.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57731094
The table in the documentation should include `spark.ui.https.enabled`, too; looks like it's missing now, which was confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493470
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ val url = newURI(schema, baseRequest.getServerName, securePort,
--- End diff --
nit: URL or URI?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18006403
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -183,7 +185,8 @@ private[spark] object JettyUtils extends Logging {
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
--- End diff --
here should only ```setHost()```, port has been set in ```getConnector(currentPort, conf)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57909017
Updated and fix the confilct, @JoshRosen you can have a test for this refer to my last comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17992049
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -183,7 +185,8 @@ private[spark] object JettyUtils extends Logging {
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
--- End diff --
It looks like we used to bind Jetty to a specific hostname / interface, but it doesn't look like the new code uses `hostName` anywhere. To preserve the same behavior, maybe we should call `connector.setHost()` after `connector.setPort()` in the code below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by jacek-lewandowski <gi...@git.apache.org>.
Github user jacek-lewandowski commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-60173980
@scwf can you take a look at https://github.com/apache/spark/issues/2739? I created PR which aim is to add SSL support for Akka based connections and for HttpServer which in turn bases on Jetty. I use SSLOptions object to configure SSL. All the settings are being kept in a separate file `ssl.conf`. You can use different name spaces to separate configurations for say UI, control and data. Would you take a look? It would be nice to have a single place to configure SSL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57924008
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21303/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57922856
Updated, thanks for all your comments, it's very helpful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57495153
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21129/consoleFull) for PR 1980 at commit [`7a898fb`](https://github.com/apache/spark/commit/7a898fbbb566172ee33f462357bb4fb02cced237).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493363
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
--- End diff --
override
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57730857
Yes, here i did not change it, because "http://joshs-mbp:4041" will redirect to https url
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-72580055
in that PR now no https support for web ui right? if so i will rework this to support it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18497130
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ val url = newURI(schema, baseRequest.getServerName, securePort,
--- End diff --
```newURL``` will return a redirected https url string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57570330
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21165/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55663526
Also, it looks like this adds configuration options under several (new) namespaces:
- `spark.http.policy`
- `spark.client.https.need-auth`
- `spark.ssl.server.keystore.*`
- `spark.ssl.server.truststore.*`
Are these the best names? What if we decide to add HTTPS / SSL to other components?
What is `spark.client`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56302482
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20623/consoleFull) for PR 1980 at commit [`de8d1bd`](https://github.com/apache/spark/commit/de8d1bd5cc3c840230d6d5dd4a4866bf0f0f7575).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005554
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
--- End diff --
Something I'd recommend here instead is to just bypass options to Jetty. e.g., some untested code based on similar Java code I have around:
val ctxFactory = new SslContextFactory();
conf.getAll.
.filter { case (k, v) => k.startsWith("spark.ui.ssl.") }
.foreach { case (k, v) => // set property in ctxFactory }
connector = new SslSelectChannelConnector(ctxFactory);
The trick is setting the property; in my code I use `BeanUtils.setProperty()` for that. My main concern here is that Jetty's SSL connector has a bunch of configurable options, and having a bypass mechanism would avoid having to plumb all of them manually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57787121
Hi, @JoshRosen, refer to http://wiki.eclipse.org/Jetty/Howto/Configure_SSL
you can generating keys and certificates by openssl or keytool
1 Generating Keys and Certificates with OpenSSL keytool use command as follows
openssl genrsa -des3 -out spark.key
openssl req -new -x509 -key spark.key -out spark.crt
openssl pkcs12 -inkey spark.key -in spark.crt -export -out spark.pkcs12
keytool -importkeystore -srckeystore spark.pkcs12 -srcstoretype PKCS12 -destkeystore spark.keystore
2 Generating Keys and Certificates with JDK keytool use
keytool -keystore spark.keystore -alias spark -genkey -keyalg RSA
This two way both ok in my machine and here is my configs in spark
```spark-env.sh```
export SPARK_MASTER_OPTS="-Dspark.ui.https.enabled=true -Dspark.ui.ssl.server.keystore.location=/home/wf/code/https/ssl/spark.keystore -Dspark.ui.ssl.server.keystore.keypassword=123456 -Dspark.ui.ssl.server.keystore.password=123456"
export SPARK_WORKER_OPTS="-Dspark.ui.https.enabled=true -Dspark.ui.ssl.server.keystore.location=/home/wf/code/https/ssl/spark.keystore -Dspark.ui.ssl.server.keystore.keypassword=123456 -Dspark.ui.ssl.server.keystore.password=123456"
```spark-defaults.conf```
spark.ui.https.enabled true
spark.ui.ssl.server.keystore.location /home/wf/code/https/ssl/spark.keystore
spark.ui.ssl.server.keystore.keypassword 123456
spark.ui.ssl.server.keystore.password 123456
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57922888
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21303/consoleFull) for PR 1980 at commit [`a48c6fc`](https://github.com/apache/spark/commit/a48c6fc585c7fdb9c859b00f9b7d7d46836706b8).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56304208
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20628/consoleFull) for PR 1980 at commit [`9591c9c`](https://github.com/apache/spark/commit/9591c9c5b90e0d1d65b4a48b08b53f6a3ed3e49d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-52381192
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18493516
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = new StringBuilder
+
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ builder.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ builder.append(scheme).append("://").append(server)
+ }
+ builder.append(':').append(port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
+ baseRequest.setHandled(true)
+ }
+ })
+ redirectHandler
+ }
+
/** Attach a prefix to the given path, but avoid returning an empty path */
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector = {
+ val ctxFactory = new SslContextFactory()
+ conf.getAll
+ .filter { case (k, v) => k.startsWith("spark.ui.ssl.") }
+ .foreach { case (k, v) => setSslContextFactoryProps(k, v, ctxFactory) }
+
+ val connector = new SslSelectChannelConnector(ctxFactory)
+ connector.setPort(port)
+ connector
+ }
+
+ private def setSslContextFactoryProps(
+ key: String, value: String, ctxFactory: SslContextFactory) = {
+ key match {
+ case "spark.ui.ssl.client.https.needAuth" => ctxFactory.setNeedClientAuth(value.toBoolean)
+ case "spark.ui.ssl.server.keystore.keypassword" => ctxFactory.setKeyManagerPassword(value)
+ case "spark.ui.ssl.server.keystore.location" => ctxFactory.setKeyStorePath(value)
+ case "spark.ui.ssl.server.keystore.password" => ctxFactory.setKeyStorePassword(value)
+ case "spark.ui.ssl.server.keystore.type" => ctxFactory.setKeyStoreType(value)
+ case "spark.ui.ssl.server.truststore.location" => ctxFactory.setTrustStore(value)
+ case "spark.ui.ssl.server.truststore.password" => ctxFactory.setTrustStorePassword(value)
+ case "spark.ui.ssl.server.truststore.type" => ctxFactory.setTrustStoreType(value)
+ }
+
+ ctxFactory
--- End diff --
Not needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57468355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21113/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57725962
Hey, sorry for the delay; I've been swamped with other work. I'm going to take a look right now...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55341033
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20175/consoleFull) for PR 1980 at commit [`8f7cc96`](https://github.com/apache/spark/commit/8f7cc967e6fd1b9d0b3768543afe06211e2178fa).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57730551
I'm testing this out locally with my own keystore and truststore. Still debugging some issues (maybe I've misconfigured my keys), but I noticed that even with HTTPS enabled, the logs still refer to the `http://` URL:
```
4/10/02 17:01:19 INFO Utils: start SparkUI at tryport: 4040
14/10/02 17:01:19 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
14/10/02 17:01:19 INFO Utils: start SparkUI at tryport: 4041
14/10/02 17:01:19 INFO Utils: Successfully started service 'SparkUI' on port 4041.
14/10/02 17:01:19 INFO SparkUI: Started SparkUI at http://joshs-mbp:4041
```
I think this is because that log message is generated from https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L103 which doesn't know about SSL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57474002
Have added redirect from http to https and updated docs. Some directions as follows
1 there is no changes for web ui when not set ```spark.ui.https.enabled```
2 when https enabled, users no need input https url to get the right ui. Aactually it's the same as http situation, users can input http url as they do in old case and the http will redirect to a https one automatically
3 if set ```spark.ui.https.enabled```, user should(must) also config the ssl config opts
```spark.ui.ssl.server.keystore.keypassword```,
```spark.ui.ssl.server.keystore.location``,
```spark.ui.ssl.server.keystore.password```.
4 config ```spark.ui.ssl.client.https.needAuth``` is optional, used to enable client authenticates. When set true users should config client ssl config
```spark.ui.ssl.server.truststore.location```
```spark.ui.ssl.server.truststore.password```.
5 ```spark.ui.ssl.server.keystore.type``` and ```spark.ui.ssl.server.truststore.type``` has default value JKS
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57505455
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21129/consoleFull) for PR 1980 at commit [`7a898fb`](https://github.com/apache/spark/commit/7a898fbbb566172ee33f462357bb4fb02cced237).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-72575398
Hi @scwf,
I just merged #3571, which added SSL support for Akka and the HTTPServer. That PR added a common configuration mechanism for SSL, plus some useful utilities for testing this functionality. If you're interested, it would be great to revisit HTTPS for the web UI now that we've merged that other patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57907414
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21294/consoleFull) for PR 1980 at commit [`baaa1ce`](https://github.com/apache/spark/commit/baaa1ce05fcb426de7d3002a5cc30f18ae119d34).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570027
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
--- End diff --
Mutating the configuration object seems like an anti-pattern to me; can we avoid this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57734206
I followed the instructions at http://stackoverflow.com/a/14408381/590203 to generate a self-signed certificate, but unfortunately I see errors when I try to browse to my web UI:
```
javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1290)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1177)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1149)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.eclipse.jetty.io.nio.SslConnection.wrap(SslConnection.java:460)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:386)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266)
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:375)
... 6 more
```
followed by
```
javax.net.ssl.SSLProtocolException: handshake alert: no_certificate
at sun.security.ssl.ServerHandshaker.handshakeAlert(ServerHandshaker.java:1586)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1747)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1060)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:884)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:536)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:401)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
```
I'm not too familiar with Java SSL setup; any idea what I'm doing wrong here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18486828
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -180,12 +186,32 @@ private[spark] object JettyUtils extends Logging {
serverName: String = ""): ServerInfo = {
val collection = new ContextHandlerCollection
- collection.setHandlers(handlers.toArray)
addFilters(handlers, conf)
// Bind to the given port, or throw a java.net.BindException if the port is occupied
def connect(currentPort: Int): (Server, Int) = {
- val server = new Server(new InetSocketAddress(hostName, currentPort))
+ val server = new Server
+ // Create a connector on port currentPort to listen for HTTP requests
+ val httpConnector = new SelectChannelConnector()
+ httpConnector.setPort(currentPort)
+ httpConnector.setHost(hostName)
+
+ if (conf.get("spark.ui.https.enabled", "false").toBoolean) {
+ // / If the new port wraps around, do not try a privilege port
+ val securePort = (currentPort + 1 - 1024) % (65536 - 1024) + 1024
+ val schema = "https"
--- End diff --
nit: scheme, not schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57907416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21294/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57923191
I have not tested it on yarn and i think we can refer the docs to config ACLs / authentication. Yeah, this test is necessary and important, i will also test it :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56303535
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20626/consoleFull) for PR 1980 at commit [`35074fd`](https://github.com/apache/spark/commit/35074fd465dee6c3dae8e0a43f2787aed50f4537).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56301129
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20622/consoleFull) for PR 1980 at commit [`c90d84e`](https://github.com/apache/spark/commit/c90d84e0c70a88742feb453c6f1517163878c003).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55356016
I did not make very much stuty of SecurityManager, but it only does authentication while not the encryption in communication.
I am not sure if it is appropriate using https to ensure the security in Spark ui communication, and not considering consistency of encryption in different underlying mechanisms.
BTW, @scwf is my workmate. We wanna encrypt the Web ui communiation now, maybe in other protocols in future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17992230
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
--- End diff --
It looks like we don't specify a default for `spark.ssl.server.keystore.location` or for `spark.ssl.server.truststore.password`, so we probably shouldn't set one for `spark.ssl.server.keystore.keypassword` either. Maybe we should log a warning if a user attempts to enable HTTPS without having properly configured their keystore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18532610
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Hi @scwf,
The reason I asked for a comment is that it seems like the method is doing a little more than just that. For example, L238 seems to be doing some sort of parsing of the `server` string, so it's more than just concatenating the different arguments into a URL.
It would be nice if the comment explained exactly what the relationship between the input and the output is. A unit test wouldn't hurt either.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-60180284
@jacek-lewandowski, you mean in ```ssl.conf``` to set ssl configs for ui( and for akka, httpserver)? then ssl.con f is the single place to configure ssl, right? if so, i think it's ok.
For the second one "some nodes running older Spark and some nodes running newer Spark", i don't think there is this situation.
@JoshRosen @vanzin is this ok to go? I think we can merge this first then in #2739 to unify the ssl configs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57480425
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21119/consoleFull) for PR 1980 at commit [`6c31dc7`](https://github.com/apache/spark/commit/6c31dc71a932083e158ef9c6737f33ebe9de1348).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432542
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ url.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ url.append(scheme).append("://").append(server)
+ }
+ if (port > 0) {
--- End diff --
I find this confusing: why would `port` ever be zero or negative?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57564475
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21165/consoleFull) for PR 1980 at commit [`a29ec86`](https://github.com/apache/spark/commit/a29ec8632cce8cb29c38d8e9e3aee2334400130b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17570692
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala ---
@@ -53,6 +53,9 @@ private[spark] class WorkerArguments(args: Array[String], conf: SparkConf) {
if (System.getenv("SPARK_WORKER_DIR") != null) {
workDir = System.getenv("SPARK_WORKER_DIR")
}
+ if (conf.contains("worker.ui.port")) {
--- End diff --
In fact, `spark.worker.ui.port` is already read on line 51, so this is unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58128776
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21360/consoleFull) for PR 1980 at commit [`8b32853`](https://github.com/apache/spark/commit/8b32853f4d138027e0319196978d3de9bad32b1b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18432553
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,88 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ private def newURI(scheme: String, server: String, port: Int, path: String, query: String) = {
+ val builder = newURIBuilder(scheme, server, port)
+ builder.append(path)
+ if (query != null && query.length > 0) builder.append('?').append(query)
+ builder.toString
+ }
+
+ private def newURIBuilder(scheme: String, server: String, port: Int) = {
+ val builder = new StringBuilder
+ appendSchemeHostPort(builder, scheme, server, port)
+ builder
+ }
+
+ private def appendSchemeHostPort(url: StringBuilder, scheme: String, server: String, port: Int) {
+ if (server.indexOf(':') >= 0 && server.charAt(0) != '[') {
+ url.append(scheme).append("://").append('[').append(server).append(']')
+ } else {
+ url.append(scheme).append("://").append(server)
+ }
+ if (port > 0) {
+ url.append(':').append(port)
+ }
+ }
+
+ private def createRedirectHttpsHandler(securePort: Int, schema: String): ContextHandler = {
+ val redirectHandler: ContextHandler = new ContextHandler
+ redirectHandler.setContextPath("/")
+ redirectHandler.setHandler(new AbstractHandler {
+ @Override def handle(
+ target: String,
+ baseRequest: Request,
+ request: HttpServletRequest,
+ response: HttpServletResponse): Unit = {
+ if (baseRequest.isSecure) {
+ return
+ }
+ if (securePort > 0) {
+ val url = newURI(schema, baseRequest.getServerName, securePort,
+ baseRequest.getRequestURI, baseRequest.getQueryString)
+ response.setContentLength(0)
+ response.sendRedirect(url)
+ }else {
+ response.sendError(HttpStatus.FORBIDDEN_403, "!Secure")
+ }
+ baseRequest.setHandled(true)
+ }
+ })
+ redirectHandler
+ }
+
/** Attach a prefix to the given path, but avoid returning an empty path */
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector = {
+ val ctxFactory = new SslContextFactory()
+ conf.getAll
+ .filter { case (k, v) => k.startsWith("spark.ui.ssl.") }
+ .foreach { case (k, v) => setSslContextFactoryProps(k, v, ctxFactory) }
+
+ val connector = new SslSelectChannelConnector(ctxFactory)
+ connector.setPort(port)
+ connector
+ }
+
+ private def setSslContextFactoryProps(
+ key: String, value: String, ctxFactory:SslContextFactory) = {
--- End diff --
Space after `:`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56720228
Regarding `http -> https` redirects, my PR #2474 adds Selenium-based tests to the Web UI; that should make it much easier to test this behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58118253
For some reason I downloaded the patch but it doesn't apply cleanly to my copy of the repo... I'll try again later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56301425
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20623/consoleFull) for PR 1980 at commit [`de8d1bd`](https://github.com/apache/spark/commit/de8d1bd5cc3c840230d6d5dd4a4866bf0f0f7575).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58636687
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21584/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-72580281
@scwf Yep, that PR only added SSL support to Akka and the HTTP FileServer, not the web UI or the shuffle service.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58672706
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21595/consoleFull) for PR 1980 at commit [`4ae834b`](https://github.com/apache/spark/commit/4ae834b6c5aafc36d44adba4c4dbb934a63ea3ed).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005297
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -137,6 +138,12 @@ private[spark] class Worker(
context.system.eventStream.subscribe(self, classOf[RemotingLifecycleEvent])
webUi = new WorkerWebUI(this, workDir, webUiPort)
webUi.bind()
+ val workerWebUiUrlPrefix = if( conf.get("spark.ui.https.enabled", "false").toBoolean) {
--- End diff --
nit: `if (conf...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r17571251
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -207,6 +210,48 @@ private[spark] object JettyUtils extends Logging {
private def attachPrefix(basePath: String, relativePath: String): String = {
if (basePath == "") relativePath else (basePath + relativePath).stripSuffix("/")
}
+
+ private def getConnector(port: Int, conf: SparkConf): Connector = {
+ val https = getHttpPolicy(conf)
+ if (https) {
+ buildSslSelectChannelConnector(port, conf)
+ } else {
+ conf.set("spark.http.policy", "http")
+ val connector = new SelectChannelConnector
+ connector.setPort(port)
+ connector
+ }
+ }
+
+ private def buildSslSelectChannelConnector(port: Int, conf: SparkConf): Connector =
+ {
+ val connector = new SslSelectChannelConnector
+ connector.setPort(port)
+
+ val context = connector.getSslContextFactory
+ val needAuth = conf.getBoolean("spark.client.https.need-auth", false)
+
+ context.setNeedClientAuth(needAuth)
+ context.setKeyManagerPassword(conf.get("spark.ssl.server.keystore.keypassword", "123456"))
--- End diff --
Also, we should include easy-to-follow instructions to help users properly configure this.
http://wiki.eclipse.org/Jetty/Howto/Configure_SSL ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18005569
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -92,6 +92,12 @@ private[spark] abstract class WebUI(
}
}
+ def appUiAddressPrefix = if(conf.get("spark.ui.https.enabled", "false").toBoolean) {
--- End diff --
nit: `if (`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57652480
@JoshRosen @andrewor14 @vanzin can you take a look at this to see whether it is ok to go~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #1980: [SPARK-2750] support https in spark web ui
Posted by pritpalm <gi...@git.apache.org>.
Github user pritpalm commented on the issue:
https://github.com/apache/spark/pull/1980
I want to enable https on spark UI. I added following config to spark-defaults.config, but when we access spark ui via https::/<ip>:8080 or https://<ip>:443 or https://<ip>:8480, it's not able to connect.
spark.ui.https.enabled true
spark.ssl.keyPassword abcd
spark.ssl.keyStore rtmqa-clientid.jks
spark.ssl.keyStorePassword changeit
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-55340547
Looks like this is a duplicate of #1714, which is now closed, however. @WangTaoTheTonic you originally filed the issue to support this. Is there a particular motivation? How is this related to the view ACLs we already have (http://spark.apache.org/docs/latest/security.html)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-54694463
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-57493293
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21119/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-56302384
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20626/consoleFull) for PR 1980 at commit [`35074fd`](https://github.com/apache/spark/commit/35074fd465dee6c3dae8e0a43f2787aed50f4537).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1980#issuecomment-58132332
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21360/consoleFull) for PR 1980 at commit [`8b32853`](https://github.com/apache/spark/commit/8b32853f4d138027e0319196978d3de9bad32b1b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org