Spree: a live-updating web UI for Spark

[Ryan Williams <ry...@gmail.com>]

Probably relevant to people on this list: on Friday I released a clone of
the Spark web UI built using Meteor <https://www.meteor.com/> so that
everything updates in real-time, saving you from endlessly refreshing the
page while jobs are running :) It can also serve as the UI for running as
well as completed applications, so you don't have to mess with the separate
history-server process if you don't want to.

*This blog post*
<http://www.hammerlab.org/2015/07/25/spree-58-a-live-updating-web-ui-for-spark/>
and
*the github repo* <https://github.com/hammerlab/spree> have lots of
information on how to use it.

It has two sub-components, JsonRelay
<https://github.com/hammerlab/spark-json-relay> and Slim
<https://github.com/hammerlab/slim>; the former sends SparkListenerEvent
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L33>s
out of Spark via a network socket, while the latter receives those events
and writes various stats about them to Mongo (like an external
JobProgressListener
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala>
that persists info to a database). You might find them to offer a better
way of storing information about running and completed Spark applications
than the event log files that Spark uses, and they can be used with or
without the real-time web UI.

Give them a try if they sound useful to you, and let me know if you have
questions or comments!

-Ryan