[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

["Tathagata Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109316#comment-15109316 ] 

Tathagata Das commented on SPARK-12140:
---------------------------------------

BTW, a major concern in supporting streaming stuff in history server is that the event logs can be extremely huge, and reading all the events to recreate the UI requires reading a huge log which can take looooong time. This needs to be solved. One idea is to allow chunking/rotations of the event logs, so that only the latest 1 hour can be read to recreate the streaming UI. It would have to handle errors, missing data, etc. 

My two cents.

> Support Streaming UI in HistoryServer
> -------------------------------------
>
>                 Key: SPARK-12140
>                 URL: https://issues.apache.org/jira/browse/SPARK-12140
>             Project: Spark
>          Issue Type: New Feature
>          Components: Streaming
>    Affects Versions: 1.6.0
>            Reporter: Marcelo Vanzin
>
> SPARK-11206 added infrastructure that would allow the streaming UI to be shown in the History Server. We should add the necessary code to make that happen, although it requires some changes to how events and listeners are used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org