Re: Question about Storm UI

[Alessandro Bellina <ab...@yahoo-inc.com.INVALID>]

Sorry about that, attached on this email. The chart is pretty simple/bare bones. 

    On Sunday, July 31, 2016 5:25 PM, Jungtaek Lim <ka...@gmail.com> wrote:
 

 Inline image seems broken. Could you make it available?
Regarding internal storage, below is what I observed from other projects while working with metrics.JStorm is using RocksDB in Nimbus to store metrics internally (only given time frame), and provide plugin to store metrics to external storages. That said, all metrics are in result sent to Nimbus and handled, which is different from Storm metrics feature.And Heron runs separate process, Metrics Manager, which stores topology's metrics via in memory, and expose REST APIs to query. But it seems limited to serve only recent 3 hours. For longer data Heron also exposes metrics sink (like a metrics consumer) to store metrics to external storages. 
For external storages I think it should be selectable since it requires users to set up. We just have to think about how to store metrics without installing external things (having limitations are fine), and how to represent them.
For UI perspective, linking to Grafana is what Heron is providing. I didn't give it a try but it might be better to take a look at. 
Btw, IMO, if we can show dashboards without dependencies that would be amazing, and having some dependencies which are easy to install and maintain would be still nice.
Thanks,Jungtaek Lim (HeartSaVioR)

2016년 8월 1일 (월) 오전 1:43, Alessandro Bellina <ab...@yahoo-inc.com.invalid>님이 작성:


HeartSaVIoR
This is different than STORM-1994. By the way added some screenshots there to clarify.
I agree it would be good to reorganize the statistics shown in the UI. Perhaps having two parts to this, one goes to a dashboard like Grafana, while the simpler metrics (snapshots, or aggregations) go to the regular Storm UI. The reason for some of them going to the Storm UI is that I think that having some stats next to the place where actions can be performed on a topology is beneficial (as they currently are).
With regards to Jerry's suggestion, we tried a simple test using MySQL just as a PoC and c3/d3 as the charting library. The choices were made purely because we wanted to see the charts quickly, so not suggesting we use MySQL for time series data. We know probably storing in Druid or another time series database would be ideal, but harder to set up. Here is a screenshot showing average user cpu time (ms) per minute for each worker:


We could probably then turn around and request data from Druid (or more generally, the "metrics store") and show in the Storm UI. Additionally, Grafana looks like it can read from Druid, so folks could use that as a front end for detailed dashboards. That said, perhaps some version of this in the Topology page at a coarser resolution would be beneficial for a quick glimpse, linking to the detailed dashboard.
Thanks,
Alessandro
 

    On Sunday, July 31, 2016 1:23 AM, Jungtaek Lim <ka...@gmail.com> wrote:
 

 Hi Boyang,

AFAIK, there's no discussion on Storm UI yet. Could you share your sketched
UI for this?
And is it on top of STORM-1994
<https://issues.apache.org/jira/browse/STORM-1994>? I mean what Alessandro
submitted.

A random thought from myself is that we shows too many numbers even
topology info page, which is not intuitive at a glance. It might be better
to show them to graph or at least more abstracted from overview page, and
show details on other pages. (component detail page, or another)

It might be also better to show graph of values for time-series manner (I
think dashboard like Grafana is the best way to show numbers as intuitive
way), but it requires metrics to be stored and served to time-series
manner, so it's not a near-future goal.

Thanks,
Jungtaek Lim (HeartSaVIoR)

2016년 7월 30일 (토) 오전 12:19, Boyang(Jerry) Peng
<je...@yahoo-inc.com.invalid>님이 작성:

> Hello Everyone,
> I would like to make enhancements on the UI by adding dynamic graphs to
> display stats of a topology.  For example, it would be nice to see
> performance and resource usage metrics be displayed in a graph on a per
> worker basis.  I know JStorm's UI has graphs but I am not sure to what
> kind.  What are our plans to improve storm UI?  What portions of JStorm's
> UI might we use?
> Best,
> Jerry