You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by harshach <gi...@git.apache.org> on 2014/05/04 06:17:45 UTC
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
GitHub user harshach opened a pull request:
https://github.com/apache/incubator-storm/pull/101
STORM-205. Add REST API to Storm UI.
1) removed html rendering from core.clj
2) added RESTful apis to core.clj
3) moved html rendering to jquery & mustache on top of newly added restful apis
4) added api
1) /api/cluster/configuration
2) /api/cluster/summary
3) /api/supervisor/summary
4) /api/topology/summary
5) /api/topology/:id
6) /api/topology/:id/component/:component
7) /api/topology/:id/activate (POST)
8) /api/topology/:id/deactivate (POST)
9) /api/topology/:id/rebalance/:wait-time (POST)
10) /api/topology/:id/kill/:wait-time (POST)
all of the above methods returns json response.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/incubator-storm ui-rest-new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/101.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #101
----
commit a06fc90e5ddb15a62c5e8214c94f0d59333fecc7
Author: Sriharsha Chintalapani <ma...@harsha.io>
Date: 2014-05-04T03:31:31Z
STORM-205. Add REST API to Storm UI.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-45014883
+1 that fixed it.
I love how much faster this feels. I like the renames you made too—much clearer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44414882
I get a weird error when trying to compile this:
```
Caused by: java.security.KeyStoreException: problem accessing trust storejava.io.UTFDataFormatException: malformed input around byte 13
```
I am not sure why I need to access a java trust store just to compile this code. Any idea what might be wrong?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13285182
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -414,88 +218,6 @@
(defn bolt-summary? [topology s]
(= :bolt (executor-summary-type topology s)))
-(defn topology-summary-table [^TopologyInfo summ]
- (let [executors (.get_executors summ)
- workers (set (for [^ExecutorSummary e executors] [(.get_host e) (.get_port e)]))]
- (table [{:text "Name" :attr {:class "tip right"
- :title (:name tips)}}
- {:text "Id" :attr {:class "tip right"
- :title (:topo-id tips)}}
- {:text "Status" :attr {:class "tip above"
- :title (:status tips)}}
- {:text "Uptime" :attr {:class "tip above"
- :title (:topo-uptime tips)}}
- {:text "Num workers" :attr {:class "tip above"
- :title (:num-workers tips)}}
- {:text "Num executors" :attr {:class "tip above"
- :title (:num-execs tips)}}
- {:text "Num tasks" :attr {:class "tip above"
- :title (:num-tasks tips)}}]
- [[(escape-html (.get_name summ))
- (escape-html (.get_id summ))
- (.get_status summ)
- (pretty-uptime-sec (.get_uptime_secs summ))
- (count workers)
- (count executors)
- (sum-tasks executors)
- ]]
- )))
-
-(defn total-aggregate-stats [spout-summs bolt-summs include-sys?]
- (let [spout-stats (get-filled-stats spout-summs)
- bolt-stats (get-filled-stats bolt-summs)
- agg-spout-stats (-> spout-stats
- (aggregate-spout-stats include-sys?)
- aggregate-spout-streams)
- agg-bolt-stats (-> bolt-stats
- (aggregate-bolt-stats include-sys?)
- aggregate-bolt-streams)]
- (merge-with
- (fn [s1 s2]
- (merge-with + s1 s2))
- (select-keys agg-bolt-stats [:emitted :transferred])
- agg-spout-stats
- )))
-
-(defn stats-times [stats-map]
- (sort-by #(Integer/parseInt %)
- (-> stats-map
- clojurify-structure
- (dissoc ":all-time")
- keys)))
-
-(defn topology-stats-table [id window stats]
- (let [times (stats-times (:emitted stats))
- display-map (into {} (for [t times] [t pretty-uptime-sec]))
- display-map (assoc display-map ":all-time" (fn [_] "All time"))]
- (sorted-table
- [{:text "Window" :attr {:class "tip right"
- :title (:window tips)}}
- {:text "Emitted" :attr {:class "tip above"
- :title (:emitted tips)}}
- {:text "Transferred" :attr {:class "tip above"
- :title (:transferred tips)}}
- {:text "Complete latency (ms)" :attr {:class "tip above"
- :title (:complete-lat tips)}}
- {:text "Acked" :attr {:class "tip above"
- :title (:spout-acked tips)}}
- {:text "Failed" :attr {:class "tip left"
- :title (:spout-failed tips)}}]
- (for [k (concat times [":all-time"])
- :let [disp ((display-map k) k)]]
- [(link-to (if (= k window) {:class "red"} {})
--- End diff --
The selected time window in the UI no longer appears as red. This could be important since the user would not know whether a window applied to the metrics without looking at the URL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47793022
@Gvain Incase if you not already done it can you clear the browser cache.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13263218
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -644,11 +304,153 @@
[:input {:type "button"
:value action
(if enabled :enabled :disabled) ""
- :onclick (str "confirmAction('"
- (StringEscapeUtils/escapeJavaScript id) "', '"
+ :onclick (str "confirmAction('"
+ (StringEscapeUtils/escapeJavaScript id) "', '"
(StringEscapeUtils/escapeJavaScript name) "', '"
command "', " is-wait ", " default-wait ")")}])
+(defn cluster-configuration []
+ (with-nimbus nimbus
+ (.getNimbusConf ^Nimbus$Client nimbus)))
+
+(defn cluster-summary
+ ([]
+ (with-nimbus nimbus
+ (cluster-summary (.getClusterInfo ^Nimbus$Client nimbus))))
+ ([^ClusterSummary summ]
+ (let [sups (.get_supervisors summ)
+ used-slots (reduce + (map #(.get_num_used_workers ^SupervisorSummary %) sups))
+ total-slots (reduce + (map #(.get_num_workers ^SupervisorSummary %) sups))
+ free-slots (- total-slots used-slots)
+ total-tasks (->> (.get_topologies summ)
+ (map #(.get_num_tasks ^TopologySummary %))
+ (reduce +))
+ total-executors (->> (.get_topologies summ)
+ (map #(.get_num_executors ^TopologySummary %))
+ (reduce +))]
+ { "stormVersion" (read-storm-version)
+ "nimbusUptime" (pretty-uptime-sec (.get_nimbus_uptime_secs summ))
+ "supervisors" (count sups)
+ "topologies" ""
--- End diff --
Looks like topologies is always blank with /api/v1/cluster/summary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by shell0dh <gi...@git.apache.org>.
Github user shell0dh commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47884912
jquery.tablesorter.min.js issue.
Set your browser encoding utf-8
or all *.html add <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13134361
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -48,92 +48,6 @@
(map #(.get_stats ^ExecutorSummary %))
(filter not-nil?)))
-(def tips
--- End diff --
I thought about this since I removed all html rendering from core.clj this made it inconvenient. I will look into mustache templates if it this can be in one place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44849862
@d2r component errors page is working fine. I guess its my template naming thats not good. There is div "error" which get update if the API responded with any error and it doesn't return any proper json.
errors-template is part of component-page-templates.html which get updated if there are any errors for the bolts or spouts returned by component api. I'll change the naming to be more appropriate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by Gvain <gi...@git.apache.org>.
Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47892650
@shell0dh Yeap, that's it Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by Gvain <gi...@git.apache.org>.
Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-48032683
@shell0dh , Thanks for your remind. I will try it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by Gvain <gi...@git.apache.org>.
Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47873698
@harshach I am using chrome, and latest storm from master branch, so, STORM-370 has already merged in. it should not related to it as my problem page is topology page while STORM-370 is component page. I also cleared my chrome cache, it doesn't help. My ui topology page looks like this:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by shell0dh <gi...@git.apache.org>.
Github user shell0dh commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47910576
@Gvain
When there is an exception, the exception time is formatted with multiple languages. Local language will be replaced by question marks.
You can apply STORM-364.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44558458
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44759676
@d2r Thanks for catching that. Topologies info can be retried from /api/v1/topology/summary . I am not sure about including it as part of cluster/summary. If it makes sense I'll add there otherwise I'll remove the "topologies" element from the response.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44579538
@d2r @revans2 Would you guys be okay with opening a JIRA for consolidating the tool tips? As far as documentation, it should probably be added to the docs on the storm website (outside this pull request).
@harshach Can you update the resource paths so the begin with `/api/v1/`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47796027
There seems to javascript syntax error : jquery.tablesorter.min.js
There're some related issues and post on mailing list, and actually I've pull a request to fix it.
STORM-381 and #174
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-45043476
Thanks for the feedback on this. Working on documentation and other recommendations that made earlier will push those as part of new JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13283946
--- Diff: storm-core/src/ui/public/component.html ---
@@ -0,0 +1,88 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
+<html><head>
+<title>Storm UI</title>
+<link href="/css/bootstrap-1.4.0.css" rel="stylesheet" type="text/css">
+<link href="/css/style.css" rel="stylesheet" type="text/css">
+<script src="/js/jquery-1.6.2.min.js" type="text/javascript"></script>
+<script src="/js/jquery.tablesorter.min.js" type="text/javascript"></script>
+<script src="/js/jquery.cookies.2.2.0.min.js" type="text/javascript"></script>
+<script src="/js/jquery.mustache.js" type="text/javascript"></script>
+<script src="/js/purl.js" type="text/javascript"></script>
+<script src="/js/bootstrap-twipsy.js" type="text/javascript"></script>
+<script src="/js/script.js" type="text/javascript"></script>
+</head>
+<body>
+<h1><a href="/">Storm UI</a></h1>
+<div id="component-summary">
+</div>
+<div id="component-stats-detail">
+</div>
+<div id="component-input-stats">
+</div>
+<div id="component-output-stats">
+</div>
+<div id="component-executor-stats">
+</div>
+<div id="component-errors">
+</div>
+<div id="error">
+</div>
+<p id="toggle-switch" style="display: block;" class="js-only"></p>
+<script>
+$(document).ready(function() {
+ var componentId = $.url().param("id");
+ var topologyId = $.url().param("topology_id");
+ var window = $.url().param("window");
+ var sys = $.cookies.get("sys") || "false";
+ var url = "/api/v1/topology/"+topologyId+"/component/"+componentId+"?sys="+sys;
+ if(window) url += "&window="+window;
+ renderToggleSys($("#toggle-switch"));
+ $.ajaxSetup({
+ "error":function(jqXHR,textStatus,response) {
+ var errorJson = jQuery.parseJSON(jqXHR.responseText);
+ $.get("/templates/error-template.html", function(template) {
+ $("#error").append(Mustache.render($(template).filter("#error-template").html(),errorJson));
+ });
+ }
+ });
+
+ $.getJSON(url,function(response,status,jqXHR) {
+ var componentSummary = $("#component-summary");
+ var componentStatsDetail = $("#component-stats-detail")
+ var inputStats = $("#component-input-stats");
+ var outputStats = $("#component-output-stats");
+ var executorStats = $("#component-executor-stats");
+ var errors = $("#component-errors");
+ $.get("/templates/component-page-template.html", function(template) {
+ componentSummary.append(Mustache.render($(template).filter("#component-summary-template").html(),response));
+ if(response["componentType"] == "spout") {
+ componentStatsDetail.append(Mustache.render($(template).filter("#spout-stats-detail-template").html(),response));
+ $("#spout-stats-table").tablesorter({ sortList: [[0,0]], headers: {0: { sorter: "stormtimestr"}}});
+ outputStats.append(Mustache.render($(template).filter("#output-stats-template").html(),response));
+ $("#output-stats-table").tablesorter({ sortList: [[0,0]], headers: {0: { sorter: "stormtimestr"}}});
+ executorStats.append(Mustache.render($(template).filter("#executor-stats-template").html(),response));
+ $("#executor-stats-table").tablesorter({ sortList: [[0,0]], headers: {1: { sorter: "stormtimestr"}}});
+ } else {
+ componentStatsDetail.append(Mustache.render($(template).filter("#bolt-stats-template").html(),response));
+ $("#bolt-stats-table").tablesorter({ sortList: [[0,0]], headers: {0: { sorter: "stormtimestr"}}});
+ inputStats.append(Mustache.render($(template).filter("#bolt-input-stats-template").html(),response));
+ if (response["inputStats"].length > 0) {
+ $("#bolt-input-stats-table").tablesorter({ sortList: [[0,0]], headers: {}});
+ }
+ outputStats.append(Mustache.render($(template).filter("#bolt-output-stats-template").html(),response));
+ $("#bolt-output-stats-table").tablesorter({ sortList: [[0,0]], headers: {}});
+ executorStats.append(Mustache.render($(template).filter("#bolt-executor-template").html(),response));
+ if(response["outputStats"].length > 0) {
+ $("#bolt-executor-table").tablesorter({ sortList: [[0,0]], headers: {}});
+ }
+ }
+ errors.append(Mustache.render($(template).filter("#errors-template").html(),response));
--- End diff --
I am seeing a table of errors where every cell is blank. Should this line have `#error-template` (singular) instead of `#errors-template`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by kishorvpatil <gi...@git.apache.org>.
Github user kishorvpatil commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44439587
Looks fine to me. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47792786
@Gvain I just downloaded apache-storm-0.9.2-incubating and ran a wordcount topology. Here is my UI
http://grab.by/yg8y
http://grab.by/yg8u
http://grab.by/yg8q.
Can you tell me which browser you are using. Also check if its not related to
https://issues.apache.org/jira/browse/STORM-370
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44758853
> I get a weird error when trying to compile this:
Never mind. Rebooted and did not see the issue again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13284947
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -509,26 +231,23 @@
reverse
first)]
(if error
- [:span (if (< (time-delta (.get_error_time_secs ^ErrorInfo error))
- (* 60 30))
- {:class "red"}
--- End diff --
Recent errors are no longer red in the UI. On the topology page, where the times of errors are not listed, this is useful to show that an error is recent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13133826
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -48,92 +48,6 @@
(map #(.get_stats ^ExecutorSummary %))
(filter not-nil?)))
-(def tips
--- End diff --
I noticed that the help text is now replicated possibly to several different HTML templates. Is there a way to consolidate these so that changes only need to be made in one place in the future, or is that really difficult to do?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44444089
@ptgoetz I remember fixing that let me check again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44584774
> @d2r @revans2 Would you guys be okay with opening a JIRA for consolidating the tool tips?
Sure that is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-45039505
looks good to me +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by msukmanowsky <gi...@git.apache.org>.
Github user msukmanowsky commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-46576742
Will do @harshach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44845150
It will be great to have an API with JSON responses coming back. I would like to resolve at least the issue with the errors table on the component page, and then I will be +1. The other issues can go in a follow-on JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44867544
yeah error is in returning the json here https://github.com/harshach/incubator-storm/blob/ui-rest-new/storm-core/src/clj/backtype/storm/ui/core.clj#L514 . updating PR with fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-46575748
@msukmanowsky right now all that api returns is a json response. Could you please file a jira for jsonp support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44599761
@harshach it looks like this commit reverts the previous commit to fix the complete latency metrics... Can you double check?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by PradeepBadiger <gi...@git.apache.org>.
Github user PradeepBadiger commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-46261079
Can we get metrics for a supplied time window?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-storm/pull/101
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by Gvain <gi...@git.apache.org>.
Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-47789948
@harshach Strange, i don't see any spout-stats section or bolt-stats-section in topology page. And thus i can't follow into component page or logviewer page. Am i missing something ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-42307854
Thanks for the feedback @ptgoetz. I will fix the above mentioned issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44860047
This is the rendered HTML I see:
```
<table id="errors-table" class="zebra-striped">
<thead></thead>
<tbody>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr></tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
</tbody>
</table>
```
It looks like 9 rows with two blank cells each, and one row without any cells.
I see this on Firefox and Safari on OSX. It seems to know there are errors there, but they are not getting populated for some reason...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44437112
I was able to build this as well without issue. I'm testing now in a clustered environment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by kishorvpatil <gi...@git.apache.org>.
Github user kishorvpatil commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44439658
Tested api methods and looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44489043
@ptgoetz Added complete-latency to the stats. Can you please try it once.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44583912
I am fine with doing documentation and cleanup later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44442875
+1
At some point in the 0.9.x line a regression was introduced such that the complete latency always displays as 0.
If that can be fixed as part of this pull request, that would be great. If not, then we should create a JIRA for it. I would like to see that fixed for the next release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by msukmanowsky <gi...@git.apache.org>.
Github user msukmanowsky commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-46571969
First off, thanks very much for this guys, having a RESTful JSON API for Storm is awesome. Just curious, is JSON-P supported by the API endpoints? This would allow users (like myself) to build something like [elasticsearch-head](https://github.com/mobz/elasticsearch-head) and avoid issues with same-origin policy in browsers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/101#discussion_r13285186
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -509,26 +231,23 @@
reverse
first)]
(if error
- [:span (if (< (time-delta (.get_error_time_secs ^ErrorInfo error))
- (* 60 30))
- {:class "red"}
- {})
- (error-subset (.get_error ^ErrorInfo error))]
- )))
+ (error-subset (.get_error ^ErrorInfo error))
+ "")))
-(defn component-link [storm-id id]
- (link-to (url-format "/topology/%s/component/%s" storm-id id) (escape-html id)))
+(defn component-task-summs [^TopologyInfo summ topology id]
+ (let [spout-summs (filter (partial spout-summary? topology) (.get_executors summ))
+ bolt-summs (filter (partial bolt-summary? topology) (.get_executors summ))
+ spout-comp-summs (group-by-comp spout-summs)
+ bolt-comp-summs (group-by-comp bolt-summs)
+ ret (if (contains? spout-comp-summs id)
+ (spout-comp-summs id)
+ (bolt-comp-summs id))]
+ (sort-by #(-> ^ExecutorSummary % .get_executor_info .get_task_start) ret)
+ ))
(defn worker-log-link [host port]
- (link-to (url-format "http://%s:%s/log?file=worker-%s.log"
- host (*STORM-CONF* LOGVIEWER-PORT) port) (str port)))
-
-(defn render-capacity [capacity]
- (let [capacity (nil-to-zero capacity)]
- [:span (if (> capacity 0.9)
- {:class "red"}
--- End diff --
Capacities may no longer be highlighted read if they are above 90%. I have not tested this, but I could not find where we were applying this style with the new templates. I do not think this is a very big deal, but it is a regression of a feature that some users might want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44862608
I know what it is. The errors are doubly-nested in lists. So you have `response["errors"][0][0]["time"]` to get the time of the first error, instead of `response["errors"][0]["time"]`. It's noticeable in the JSON output.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-42307190
I think this is a pretty good start. Switching over to a REST API will enable external systems to do monitoring, visualizations, etc.
However, there are a couple of issues that need to be addressed:
Main Screen:
* Topology Summary: uptime is not formatted Xm Xs
Topology Detail:
* Topology stats: # acked is always 0
* Spouts: Transferred and Acked fields are blank
* Bolts: Transferred, Execute Latency, and Process Latency are blank
* Capacity values are off on all screens
In `core.clj`:
* URL decoding should use the `url-decode` function from `backtype.storm.util.clj` to enforce the use of `UTF-8` encoding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44577769
I agree with d2r that I would like to see the tool tips centralized if possible, I would also like to see a version on the REST apis, something like api/v1/... And it would be nice to have some documentation on the APIs somewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44861386
> It looks like 9 rows with two blank cells each, and one row without any cells.
Correction: It is 10 rows of two blank cells each.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44874202
@d2r can you please check the latest update. I've tested on a small cluster here is a screenshot
http://grab.by/xoP4
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44416822
d2r thats werid. I just did a rebase against master and ran the build . It went through fine with all the tests passing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44861584
@d2r thanks for the info. I am testing few things I'll update the PR with the fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-46270173
@PradeepBadiger Yes you can pass a window param (in secs) to
/api/v1/topology/:id and /api/v1/topology/:id/component/:component . It only support time windows that UI shows. There is no arbitrary time windows since metrics collection in storm doesn't support that yet.
I am finishing up documentation on rest api that will give more detailed description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-205. Add REST API to Storm UI.
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/101#issuecomment-44662020
@revans2 @d2r Are you okay with merging this as-is with a follow up JIRA for cleanup/doc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---