You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by zhuoliu <gi...@git.apache.org> on 2015/11/03 18:19:59 UTC
[GitHub] storm pull request: [STORM-902] Simple Log Search
GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/848
[STORM-902] Simple Log Search
This feature is aimed for greatly improving the debugging of Storm.
Simple Log Search supports following important functionalities:
1. String search:
In UI, user can search certain string, e.g., "Exception", in for a certain worker log.
This search can happen for both normal text log or rolled zip log files. In addition, user can specify the number of matches to show in the search result.
2. Deep search:
User can search for a string for a certain topology, which means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology.
Then the matched results can be shown on the UI with url links, directing user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
Also, the topology history REST API is added to give hints for the previously launched topology IDs for a certain user. And the html files, images files, js file, test resource files and unit tests are added for log search and its tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 902
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/848.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #848
----
commit 20db09cc5da62bd1b13e8c563e55f9e8e5103a4a
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-30T19:35:32Z
[YSTORM-902] log search, init commit
commit ad64c5d94607ebaa696171c2f67cf7d3a621f5a1
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T02:06:00Z
Continue on logviewer search
commit 04577758baee3258d2be4401f009e95658a17405
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T02:14:48Z
lv compile passed, to do html and then manual test, unit test
commit 13194c612eb55a4d4254860172139878e46dee67
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T03:03:33Z
Add html image files
commit dfb5e90852e221af6027df35d61c0951a6973ecc
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T19:41:21Z
Now able to search, except the magnifying lens stuff (related to topo history)
commit ca24ea096adc37375e83febb105df5532a720d17
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T19:58:38Z
Support TopologyHistory (just a couple of topoId for a user)
commit 4d1acbf7fb60d7a09fbb861fd5ef309638eca864
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-02T23:31:37Z
Debug, thrift, topo history, now working well for deep search.
commit 90ef371aadc14585f6db785fc0058f0790136eb6
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-11-03T16:52:26Z
Unit test and add related resurces files
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154136334
@zhuoliu Thank you for this feature. This is really great and going to help lot of users in debugging topologies. I'll add my reviews.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r43794932
--- Diff: storm-core/src/clj/backtype/storm/ui/core.clj ---
@@ -843,14 +870,21 @@
(populate-context! servlet-request)
(assert-authorized-user "getClusterInfo")
(json-response (nimbus-summary) (:callback m)))
+ (GET "/api/v1/history/summary" [:as {:keys [cookies servlet-request]} & m]
+ (let [user (.getUserName http-creds-handler servlet-request)]
+ (json-response (topology-history-info user) (:callback m))))
(GET "/api/v1/supervisor/summary" [:as {:keys [cookies servlet-request]} & m]
- (populate-context! servlet-request)
- (assert-authorized-user "getClusterInfo")
- (json-response (supervisor-summary) (:callback m)))
+ (assert-authorized-user servlet-request "getClusterInfo")
--- End diff --
This needs to stay as populate-context! before calling assert-authorized-user, or the code will fail when security is enabled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r43799518
--- Diff: storm-core/src/clj/backtype/storm/daemon/logviewer.clj ---
@@ -476,6 +485,405 @@ Note that if anything goes wrong, this will throw an Error and exit."
(-> (resp/response "Page not found")
(resp/status 404)))))
+(def grep-max-search-size 1024)
+(def grep-buf-size 2048)
+(def grep-context-size 128)
+
+(defn logviewer-port
--- End diff --
We have it have a function mostly for mocking.
logviewer_test.clj
let[expected-port 8888]
(stubbing [local-hostname expected-host
logviewer/logviewer-port expected-port]
.................
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-155136101
Thank you all sincerely for the thorough reviews!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154181484
Thanks @harshach for the review , your comments related to the history API are addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154120525
Hello @harshach @Parth-Brahmbhatt @HeartSaVioR , since this a quite a big feature, more eyes would be very helpful. Could you guys have a look at it? Thanks in advance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-155223589
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153474328
Sure, @revans2 . Comments addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154464749
@zhuoliu Sorry I did partial review and then got distracted, give me today's day and I will add more comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153802271
Upmerged and rebased since the profiling stuff has been merged in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153459000
Thanks @revans2 , commits squashed and nimbus-test fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154287293
+1. @Parth-Brahmbhatt has some reviews.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153816456
+1 looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-155117826
Hi @Parth-Brahmbhatt , really appreciate that if you have some more comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r44042359
--- Diff: docs/documentation/ui-rest-api.md ---
@@ -172,6 +172,27 @@ Sample response:
}
```
+### /api/v1/history/summary (GET)
+
+Returns a list of topology ID submitted by the current user.
--- End diff --
The URL has /history in it but the documentation does not say if it will return all the topologies ever submitted (a true history) or only the current snapshot. If its former we should update the documentation , if its later we should chane the URL to /api/v1/topologIds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153463423
`typeahead.jquery.min.js` needs to be added to LICENSE.txt as it is part of an external project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153463569
@zhuoliu for the most part things look really good, just a few comments that I see need to be addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-155133328
Hi, I am +1 on this patch. Sorry took me longer to review than I expected. Thank a lot for your contributions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r43794150
--- Diff: storm-core/src/clj/backtype/storm/daemon/logviewer.clj ---
@@ -476,6 +485,405 @@ Note that if anything goes wrong, this will throw an Error and exit."
(-> (resp/response "Page not found")
(resp/status 404)))))
+(def grep-max-search-size 1024)
+(def grep-buf-size 2048)
+(def grep-context-size 128)
+
+(defn logviewer-port
--- End diff --
Does this really need to be a function? If so can be memorize this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r44055722
--- Diff: docs/documentation/ui-rest-api.md ---
@@ -172,6 +172,27 @@ Sample response:
}
```
+### /api/v1/history/summary (GET)
+
+Returns a list of topology ID submitted by the current user.
+
+Response fields:
+
+|Field |Value | Description|
+|--- |--- |---
+|topo-history| List| List of Topologies' IDs|
+
+Sample response:
+
+```json
+{
+ "topo-history":[
+ {"host":"wc6-1-1446571009"},
--- End diff --
Yes, nice catch. This is legacy code, actually, we only need to return a list of topo-id, the key "host" does not make sense (it is just a hard-coded string, not real host name). I will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r44042484
--- Diff: docs/documentation/ui-rest-api.md ---
@@ -172,6 +172,27 @@ Sample response:
}
```
+### /api/v1/history/summary (GET)
+
+Returns a list of topology ID submitted by the current user.
+
+Response fields:
+
+|Field |Value | Description|
+|--- |--- |---
+|topo-history| List| List of Topologies' IDs|
+
+Sample response:
+
+```json
+{
+ "topo-history":[
+ {"host":"wc6-1-1446571009"},
--- End diff --
I am confused if this is suppose to return a list of TopologyIDs why does the resposse have a list of map with host name as key?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r43800101
--- Diff: storm-core/src/clj/backtype/storm/daemon/logviewer.clj ---
@@ -476,6 +485,405 @@ Note that if anything goes wrong, this will throw an Error and exit."
(-> (resp/response "Page not found")
(resp/status 404)))))
+(def grep-max-search-size 1024)
+(def grep-buf-size 2048)
+(def grep-context-size 128)
+
+(defn logviewer-port
--- End diff --
Thats right I saw that later on in the tests, and spaced it. looks good then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154464027
Hi @Parth-Brahmbhatt, could I have you some time to check back the fixes? Appreciate that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-154468820
Sure, that would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153842760
+1 OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/848#discussion_r44055022
--- Diff: docs/documentation/ui-rest-api.md ---
@@ -172,6 +172,27 @@ Sample response:
}
```
+### /api/v1/history/summary (GET)
+
+Returns a list of topology ID submitted by the current user.
--- End diff --
Yes, it will return the true history of topology. I will update the doc to clarify that. Also, a history cleaner will added to cleanup the history for topologies that are older than a threshold ago (one week).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/848#issuecomment-153511990
REST API docs added for topology/history and topology-workers/:id
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-902] Simple Log Search
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/848
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---