You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/10/24 23:36:58 UTC
[jira] [Updated] (SPARK-18085) Scalability enhancements for the
History Server
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin updated SPARK-18085:
-----------------------------------
Attachment: spark_hs_next_gen.pdf
Here's an initial document to bootstrap the discussion of how to fix the History Server. It discusses the current issues and proposes solutions to them, providing an implementation path that will hopefully allow for finer-grained reviews with as little disruption to functionality as possible while the work is under review.
I could also share a google doc link if people prefer (although I believe the ASF likes documents to be stored on their servers).
I think there are two points that might be a little controversial, so I'll list them up front:
- use of a JNI library for data storage: this is not a requirement, but there were no good pure Java alternatives I could find. The functionality is not dependent on the actual library, though, so we could switch the library if desired.
- move of Spark UI code to a separate module: I think this is good both as a way to isolate these changes, and for future maintainability; it makes it clearer that the UI is sort of a separate component, and that there should be a clearer separation between the code in core and the ui.
> Scalability enhancements for the History Server
> -----------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues. I'll be attaching a document shortly describing the issues and suggesting a path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org