You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by attilapiros <gi...@git.apache.org> on 2018/11/19 20:45:00 UTC
[GitHub] spark pull request #23090: [SPARK-26118][UI] Introducing spark.ui.requestHea...
GitHub user attilapiros opened a pull request:
https://github.com/apache/spark/pull/23090
[SPARK-26118][UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize
## What changes were proposed in this pull request?
Introducing spark.ui.requestHeaderSize for configuring Jetty's HTTP requestHeaderSize.
This way long authorization field does not lead to HTTP 413.
## How was this patch tested?
Manually with curl (which version must be at least 7.55).
With the original default value (8k limit):
```bash
# Starting history server with default requestHeaderSize
$ ./sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# Creating huge header
$ echo -n "X-Custom-Header: " > cookie
$ printf 'A%.0s' {1..9500} >> cookie
# HTTP GET with huge header fails with 431
$ curl -H @cookie http://458@apiros-MBP.lan:18080/
<h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre>
# The log contains the error
$ tail -1 /Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
18/11/19 21:24:28 WARN HttpParser: Header is too large 8193>8192
```
After:
```bash
# Creating the history properties file with the increased requestHeaderSize
$ echo spark.ui.requestHeaderSize=10000 > history.properties
# Starting Spark History Server with the settings
$ ./sbin/start-history-server.sh --properties-file history.properties
starting org.apache.spark.deploy.history.HistoryServer, logging to /Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# HTTP GET with huge header gives back HTML5 (I have added here only just a part of the response)
$ curl -H @cookie http://458@apiros-MBP.lan:18080/
<!DOCTYPE html><html>
<head>...
<link rel="shortcut icon" href="/static/spark-logo-77x50px-hd.png"></link>
<title>History Server</title>
</head>
<body>
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/attilapiros/spark JettyHeaderSize
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23090.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23090
----
commit c6278becc29f5605254786ad5070ab2666ffad47
Author: “attilapiros” <pi...@...>
Date: 2018-11-19T19:55:28Z
Introducing spark.ui.requestHeaderSize
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99023/testReport)** for PR 23090 at commit [`a343f93`](https://github.com/apache/spark/commit/a343f933d15aacc1e0407a0b23eceb0b4da7a5bc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/23090
Thanks @dongjoon-hyun, of course I will create those PRs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/23090
I would prefer to override this 8k limit when it is really necessary (and only with the extent which justified by the production system).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][UI] Introducing spark.ui.requestHeaderSize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99019/testReport)** for PR 23090 at commit [`81334f4`](https://github.com/apache/spark/commit/81334f45e158fe5edd42a1297caa91d208c9c112).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23090
Thank you, @squito , @attilapiros , @srowen .
Then, @attilapiros , could you send backporting PRs against `branch-2.3` and `branch-2.2` please?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23090: [SPARK-26118][Web UI] Introducing spark.ui.reques...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23090#discussion_r234799817
--- Diff: docs/configuration.md ---
@@ -973,6 +973,14 @@ Apart from these, the following properties are also available, and may be useful
<br /><code>spark.com.test.filter1.param.name2=bar</code>
</td>
</tr>
+<tr>
+ <td><code>spark.ui.requestHeaderSize</code></td>
+ <td>8k</td>
+ <td>
+ The HTTP request header size, in bytes unless otherwise specified.
+ This setting applied for the Spark History Server too.
--- End diff --
I'd say "The maximum allowed size for a HTTP request header".
Also applied -> applies
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23090: [SPARK-26118][Web UI] Introducing spark.ui.reques...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/23090
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99018/testReport)** for PR 23090 at commit [`c6278be`](https://github.com/apache/spark/commit/c6278becc29f5605254786ad5070ab2666ffad47).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/23090
Backport PR for 2.2: https://github.com/apache/spark/pull/23115
Both 2.2 and 2.3 are retested.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #4434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4434/testReport)** for PR 23090 at commit [`a343f93`](https://github.com/apache/spark/commit/a343f933d15aacc1e0407a0b23eceb0b4da7a5bc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/23090
Backport PR for 2.3: https://github.com/apache/spark/pull/23114
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][UI] Introducing spark.ui.requestHeaderSize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #4434 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4434/testReport)** for PR 23090 at commit [`a343f93`](https://github.com/apache/spark/commit/a343f933d15aacc1e0407a0b23eceb0b4da7a5bc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99019/testReport)** for PR 23090 at commit [`81334f4`](https://github.com/apache/spark/commit/81334f45e158fe5edd42a1297caa91d208c9c112).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99023/testReport)** for PR 23090 at commit [`a343f93`](https://github.com/apache/spark/commit/a343f933d15aacc1e0407a0b23eceb0b4da7a5bc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23090
Why not just hard-code a much higher limit? what's the consequence? in a generic public web server I can see that this defends against malicious or malformed requests, but that's much less an issue for an internal Spark server.
That is, would a user really need to vary this up and down and know enough to find this config?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99019/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/23090
@dongjoon-hyun, @squito it is my bad, this problem is present in earlier releases as well. If I can change the issue type I modify it to Bug.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23090
Hi, @squito .
SPARK-26118 is marked as 'Improvement', we don't backport 'Improvement'.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99018/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
merged to master / 2.4
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
btw I agree this was bad judgement on my part to only backport to 2.4, sorry abotu that and thanks for catching @dongjoon-hyun . I do think this fixes a bug (the ability to use the UI when you are a member of many user groups) and so should be backported further, though I would really be fine either way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][UI] Introducing spark.ui.requestHeaderSize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99023/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
I think the default is pretty reasonable in most cases, in that this is the first time we've heard of someone hitting this limit. I'm not sure how high we would make it to get around this problem in general. And from the jetty docs: https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/server/HttpConfiguration.html#setRequestHeaderSize-int-
> Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL. However, larger headers consume more memory and can make a server more vulnerable to denial of service attacks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][Web UI] Introducing spark.ui.requestHeader...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23090
I understand the situation. However, according to the content of the patch, it's an improvement to add a missing configuration, @attilapiros .
```
httpConfig.setRequestHeaderSize(conf.get(UI_REQUEST_HEADER_SIZE).toInt)
```
@rxin , @gatorsmile , @srowen . How do you think about this? If PMC allows this, I'd like to have this in older branches, too. Otherwise, we need to revert this in `branch-2.4`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23090: [SPARK-26118][UI] Introducing spark.ui.requestHeaderSize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23090
**[Test build #99018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99018/testReport)** for PR 23090 at commit [`c6278be`](https://github.com/apache/spark/commit/c6278becc29f5605254786ad5070ab2666ffad47).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org