You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by LantaoJin <gi...@git.apache.org> on 2018/02/11 07:03:40 UTC
[GitHub] spark pull request #20574: [SPARK-23385][CORE] Allow SparkUITab to be custom...
GitHub user LantaoJin opened a pull request:
https://github.com/apache/spark/pull/20574
[SPARK-23385][CORE] Allow SparkUITab to be customized adding in Spark…
…Conf and loaded when creating SparkUI
## What changes were proposed in this pull request?
It would be nice if there was a mechanism to allow to add customized SparkUITab (embedded like Jobs, Stages, Storage, Environment, Executors,...) to be registered through SparkConf settings. This would be more flexible when we need display some special information in UI rather than adding the embedded one by one and wait community to merge.
I propose to introduce a new configuration option, spark.extraUITabs, that allows customized WebUITab to be specified in SparkConf and registered when SparkUI is created. Here is the proposed documentation for the new option:
> A comma-separated list of classes that implement SparkUITab; when initializing SparkUI, instances of these classes will be created and registered to the tabs array in SparkUI. If a class has a two-argument constructor that accepts a SparkUI and AppStatusStore, that constructor will be called; If a class has a single-argument constructor that accepts a SparkUI; otherwise, a zero-argument constructor will be called. If no valid constructor can be found, the SparkUI creation will fail with an exception.
## How was this patch tested?
1. Offerred a unit test.
2. Check the WebUI to see a new tab called "Test" via
`bin/spark-shell` --master local --driver-class-path /path/spark-core_2.11-*-tests.jar --conf `spark.extraUITabs=org.apache.spark.ui.TestUITab
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/LantaoJin/spark SPARK-23385
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20574.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20574
----
commit fb9a8a1be7fc515848b0906af8af31c4c8081807
Author: LantaoJin <ji...@...>
Date: 2018-02-11T06:56:01Z
[SPARK-23385][CORE] Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20574
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20574
@jerryshao Could you have a time to help to review?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20574
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20574
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20574
Hi @abellina, I think all the tabs were configured this way couldn't go through in community. Even opening an interface to add customized tabs, @srowen thinks it isn't worth to do that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20574
This seems like a simple nice-to-have, but opens up a lot of questions. What is the SPI contract between Spark and these tabs? I am not sure it's worth it. What's an example of something that must be in a custom tab and not in your user code? I find these requests usually come in as a workaround for some other concern outside Spark.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on the issue:
https://github.com/apache/spark/pull/20574
@LantaoLin what if all the tabs were configured this way? E.g. there is a default order/set of tabs, but one can move them around, or mix and match with custom tabs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20574
As an actual case, we analyzed memory waste, insuited configuration, data skew in shuffle, failed job reason and so on profiling and suggestion information at runtime. User can get these information in original Spark UI in a new tag. We know it can’t be merge to Spark code. But we still know lots of companies do the similar things. Without hack the code, adding an interface for extending the UITab seams can help it well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20574
cc @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20574
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...
Posted by LantaoJin <gi...@git.apache.org>.
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20574
Thank you Sean for the comments. There are three reasons to do that:
1. Help developer to separate their code from Spark code. I see many tickets and requests which want to add new pages and finally could not be accepted, and if we open this extra tab interface, they can just leave their pages code in a third-part library with a configuration to enable it. No need merge or hack the Spark code.
2. Another common way is starting a jetty in extra listener and display pages in this web server. This can work but need driver to launch another jetty. Also the URL is different with self UI.
3. Use a standalone web server to show spark information. In eBay we did this things, but we found that almost users still prefer to get information from origin Spark UI. Although we can hack the code, we still holp Spark can support this request officially. That’s the original purpose of this jira.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org