You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by carlosfuertes <gi...@git.apache.org> on 2014/07/31 07:42:48 UTC
[GitHub] spark pull request: Spark 2017
GitHub user carlosfuertes opened a pull request:
https://github.com/apache/spark/pull/1682
Spark 2017
I address here issues SPARK-2017 and SPARK-2016 by serving the data for the tables as JSON further rendering under Spark UI web interface.
Main addition is exposing paths with the JSON data as:
/stages/stage/tasks/json/?id=nnn
/storage/json
/storage/rdd/workers/json?id=nnn
/storage/rdd/blocks/json?id=nnn
and using javascript to built the tables from an ajax request of those JSON.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/carlosfuertes/spark SPARK-2017
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1682.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1682
----
commit 458cae4ab065a0e173e0f9dd2c075addafabd9a7
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-28T05:24:41Z
json output for storage/rdd
commit 40a49c36516a261936fd8d591ec864646ec3f1ae
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-28T06:12:11Z
small refactoring to keep code dry
commit 7e6ce37a601f7b58ab25351c366f213a7e159cb7
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-29T05:54:18Z
storage tab is using completely JSON to render all data
commit 3d5f1ddf210d07821e2ba6177a04f338972e7109
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-29T05:55:08Z
Merge remote-tracking branch 'upstream/master' into SPARK-2017
commit 469320603e1aca5c10fc59ff3fe3443ad9f4c9ce
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-29T06:08:56Z
made code pass style guidelines
commit f4ebcdbcca0ecefb4f4c990c35810a388c8e7bc0
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-29T12:27:41Z
added expanding arrays in JSON for html tables into javascript function
commit f20560041468f1c0a59051cc960a94c58ebc138d
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-30T02:29:56Z
Merge remote-tracking branch 'upstream/master' into SPARK-2017
commit 84d8628df35f01bc9ff818e0026a8cda6140bdf4
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-30T05:11:21Z
adding tasks table under stage Spark UI from JSON. First pass
commit 6e787f60c670db77a2e1c7be3980ca7493b9ede9
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-30T06:52:25Z
added sorttable custom key to json
commit 11e33836bc89932b1d5a4919394170e030329b57
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-30T13:09:30Z
fixed sorttable to sort tables again
commit 7b0bde38371fe0e12d4cee52336ead6dff819d37
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-31T02:31:47Z
Merge remote-tracking branch 'upstream/master' into SPARK-2017
commit 5aa4b283010a1eaaf48d09399ed16bf5aafb4726
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-31T05:19:38Z
added a proper JSON handling of sorttable_custom key
commit 384ff1a567eadeefa72d28d5fedd9260a1874d3f
Author: carlosfuertes <ca...@gmail.com>
Date: 2014-07-31T05:22:27Z
deleted commented code
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark 2017
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-50931690
Identified the bottleneck in the rendering of the page containing the data tables. It was related to the css styling used (bootstrap with some nth-child(odd)). Added a custom style instead and now all rendering is much faster. See Jira SPARK-2017 for more details
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Spark 2017
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54398223
Also, do you mind editing the title of this PR so that it's tracked correctly by our review tools?
Something like `[SPARK-2017] [SPARK-2016] Web UI responsiveness with large numbers of tasks/partitions` would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54398122
It seems like having two different table rendering techniques, server-side HTML and client-side Javascript, could become a maintenance / complexity burden. Do you think it's important for the UI to work without Javascript? It could be important for tools that scrape the web UI, but it would be better if those tools consumed JSON data instead.
Personally, I'd be a fan of delivering a stable JSON API and rewriting the web UI as a Javascript application that consumes data from those endpoints, but I'm open to other opinions.
/cc @pwendell @mateiz @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54750881
Hi,
I have updated the title of the pull request and make sure it is mergable
after latest master updates. Last I have dropped the usage of a custom
table css since as I explained in
https://issues.apache.org/jira/browse/SPARK-2017 it is not reallly the
bottleneck and it may simplify things at first (that's a later tweak).
The reason I added the env variable ""spark.ui.jsRenderingEnabled" and
retain html server rendering in this PR, is to ensure folks that rely on
not having to use javascript can still operate. They would just to launch
spark with "spark.ui.jsRenderingEnabled" to false.
Later, anybody that relies on not using javascript, should access the info
using the JSON interface. But till people are using the JSON interface, we
should still have the current minimal html form.
Right now I see this Pull Request as a working proof of concept of what the
JSON interface and javascript can look like. There are still some points to
discuss and agree among everybody:
1) What is the JSON format that we want?
Current JSON is very verbose, in the sense that is very inefficient
since it sends everything as key value for every line: It repeats
field unnecessarily
[ {
"Index" : 0,
"ID" : 0,
"Attempt" : {
"value" : "0",
"sorttable_customkey" : "0"
},
"Status" : "SUCCESS",
"Locality Level" : "PROCESS_LOCAL",
"Executor" : "localhost",
"Launch Time" : "2014/09/07 15:25:24",
"Duration" : {
"value" : "0.8 s",
"sorttable_customkey" : "780"
},
"GC Time" : "",
"Errors" : ""
}, ...
We could use a much compact format where the first line are the names
of the fields and every new line is just an array of the values of the
fields (no repetition of keys).
Also we need to include tests for the JSON interface.
2) If we want to deal and render really big tables, I think we should
include pagination and update the web UI with it.
In the JSON interface, we should include a parameter that tells you
how many rows to return. Something like ex.
"/storage/rdd/blocks/json/?id=0&nrows=1000" and if you want to get
everything say it explicitly, ex.
"/storage/rdd/blocks/json/?id=0&nrows=all"
3) The current method to sort the output in the browser using the
"sorttable" js package does not work for large tables. It is too slow.
When we request the data, the server should do the sorting.
That is, the JSON api should receive a parameter telling the server
which column should be used to do the sorting: something like ex.
"/storage/rdd/blocks/json/?id=0&sortby=0"
Let me know what you think about the points above.
On Sat, Sep 6, 2014 at 3:44 PM, Josh Rosen <no...@github.com> wrote:
> @ash211 <https://github.com/ash211> This weekend, I'm actually working on
> writing a design document for web UI improvements in Spark 1.2. SSL
> encryption, authentication, and ACLs are all features that I'm planning to
> put on the roadmap.
>
> Do you have SSH to your EC2 machines? One option is to use a SSH proxy to
> view the full web UI in your browser. Once you've set up the proxy, you can
> use a browser plugin like FoxyProxy <http://getfoxyproxy.org/> to
> seamlessly proxy requests for the UI.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/1682#issuecomment-54726062>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-50715124
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54774014
@JoshRosen I can imagine that avoiding duplication of keys can save easily 50% or more. If we are aiming at big data sizes that can matter a lot. In the JIRA post I explain how I was seeing already 15Mb data sizes with just 50,000 jobs. Gzip can make sense but if you start with something that is already 50% smaller that's even better.
In any case that is very simple to test and benchmark (with this PR for example is very simple) and an optimization at the end of the day.
To be very concrete, something like
[ {"key1": row1_value1, "key2": row1_value2},
{"key1": row2_value1, "key2": row2_value2},
{"key1": row3_value1, "key2": row3_value2} ]
versus using
{ meta: { keys: [ "key1", "key2" ] },
data: [ ["row1_value1", "row1_value1"],
["row2_value1", "row2_value1"],
["row3_value1", "row3_value1"] ] }
I would stick to using JSON and not csv or tsv, since JSON interacts extremely well with javascript (it was design to), and pretty much anything, and you have the flexibility to add other meta information and parse it.
I have seen some HTTP APIs that incorporate pagination and so forth (ex. steam web page) but I do not have a particular one in mind freely available in the wild... I'm thinking that talking about JSON API is a bit confusing and it would be better to refer to the HTTP API (which returns JSON for some calls). I think it may make sense in the doc that you are working on to delineate the full (RESTful) HTTP API for the WebUI rather than just the JSON part so that the global picture and design is clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54702827
FWIW I run Spark in AWS and my ops team requires that all web interfaces exposed through the proxy into the enclave have both SSL and user auth. Spark doesn't support those, so for now I'm making heavy use of the webui via the links command line browser.
All that to say, yes there are users accessing the interface via no-Javascript browsers. I can understand though if you make the web interface use JS and expose a REST API since that's a more generally attractive setup for checking status on a cluster.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54754415
@JoshRosen You mentioned above that you are working on a design document for web UI improvements. Are you going to post it anywhere? I would like to take all the above and help make it happen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-56478100
I've opened [SPARK-3644](https://issues.apache.org/jira/browse/SPARK-3644) as a forum for discussing the design of a REST API; sorry for the delay (got busy with other work / bug fixing).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes closed the pull request at:
https://github.com/apache/spark/pull/1682
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by carlosfuertes <gi...@git.apache.org>.
Github user carlosfuertes commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-50980868
I added a configuration property "spark.ui.jsRenderingEnabled" that controls whether the rendering of the tables happens using Javascript or not. It is enable by default. This ensures that people that cannot or do not want to run javascript to do the rendering, they can use the web ui as before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2017] [SPARK-2016] Web UI responsivenes...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54755433
@carlosfuertes I'm working on a draft this weekend; I'll make a post to the developers mailing list sometime next week once I've gotten the basics fleshed out.
Is the duplication of the keys a significant problem in practice? If we configure the API's HTTP server to use gzip compression, I think the API responses still shouldn't be too big. If we do choose to remove this redundancy, would it make more sense to return the data in CSV or TSV format?
You've raised some good questions about sorting, pagination, etc. I think we should address these in a consistent way throughout the API. Are there some existing JSON APIs that you think made good decisions for these features?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54694542
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 2017
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1682#issuecomment-54726062
@ash211 This weekend, I'm actually working on writing a design document for web UI improvements in Spark 1.2. SSL encryption, authentication, and ACLs are all features that I'm planning to put on the roadmap.
Do you have SSH to your EC2 machines? One option is to use a SSH proxy to view the full web UI in your browser. Once you've set up the proxy, you can use a browser plugin like [FoxyProxy](http://getfoxyproxy.org/) to seamlessly proxy requests for the UI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org