You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/27 08:32:05 UTC

[GitHub] [spark] kuwii opened a new pull request #30158: [SPARK-33249][CORE][UI] Add status plugin for live application

kuwii opened a new pull request #30158:
URL: https://github.com/apache/spark/pull/30158

### What changes were proposed in this pull request?

- Add an interface called `AppLiveStatusPlugin` for live application to create extra listeners that can write a given store.
- Add a configuration called `spark.appLiveStatusPlugins`, to specify what `AppLiveStatusPlugin` should be loaded.
- In SparkContext, load `AppLiveStatusPlugin` using `Utils.loadExtensions`, then create and register listeners.

### Why are the changes needed?

There are cases that developers may want to extend the current REST API of Web UI. In most of these cases, adding external module is a better option than directly editing the original Spark code.

For an external module, to extend the REST API of the Web UI, 2 things may need to be done:
* Add extra API to provide extra status info. This can be simply done by implementing another `ApiRequestContext` which will be automatically loaded.
* If the info can not be calculated from the original data in the store, add extra listeners to generate them.

For history server, there is an interface called `AppHistoryServerPlugin`, which is loaded based on SPI, providing a method to create listeners. In live application, the only way is `spark.extraListeners` based on `Utils.loadExtensions`. But this is not enough for the cases.

To let the API get the status info, the data need to be written to the `AppStatusStore`, which is the only store that an API can get by accessing `ui.store` or `ui.sc.statusStore`. But listeners created by `Utils.loadExtensions` only get a `SparkConf` in construction, and are unable to write the `AppStatusStore`.

So I think we still need plugin like `AppHistorySever` for live UI. For concerns like [SPARK-22786](https://github.com/apache/spark/pull/19981), the plugin for live app can be separated from the history server one, and also loaded using Utils.loadExtensions with an extra configurations. So by default, nothing will be loaded.

### Does this PR introduce _any_ user-facing change?

Yes, a new configuration property called `spark.appLiveStatusPlugins`, indicating all `AppLiveStatusPlugin` to load.

### How was this patch tested?

Tested using existed UT.

Also add tests for `AppLiveStatusPlugin` to make sure plugins in `spark.appLiveStatusPlugins` will be loaded and listeners will be created and registered.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org