You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by kn...@apache.org on 2015/12/17 21:45:20 UTC
[1/5] storm git commit: [STORM-1393] Update the storm.log.dir
function, add doc for logs
Repository: storm
Updated Branches:
refs/heads/master 8e708cf27 -> 6cf33a86d
[STORM-1393] Update the storm.log.dir function, add doc for logs
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/288d6054
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/288d6054
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/288d6054
Branch: refs/heads/master
Commit: 288d6054baa0c32f682bfe8ff71f89f3a8f8d54a
Parents: 8b3c208
Author: zhuol <zh...@yahoo-inc.com>
Authored: Tue Dec 15 16:01:39 2015 -0600
Committer: zhuol <zh...@yahoo-inc.com>
Committed: Tue Dec 15 16:01:39 2015 -0600
----------------------------------------------------------------------
docs/documentation/Documentation.md | 2 +-
docs/documentation/Log-Search.md | 14 ---------
docs/documentation/Logs.md | 30 ++++++++++++++++++++
storm-core/src/clj/backtype/storm/config.clj | 10 +++++--
.../clj/backtype/storm/daemon/supervisor.clj | 2 +-
storm-core/src/clj/backtype/storm/util.clj | 4 ---
6 files changed, 40 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/docs/documentation/Documentation.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Documentation.md b/docs/documentation/Documentation.md
index 7044664..ab555c1 100644
--- a/docs/documentation/Documentation.md
+++ b/docs/documentation/Documentation.md
@@ -45,7 +45,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin
* [Metrics](Metrics.html)
* [Lifecycle of a trident tuple]()
* [UI REST API](ui-rest-api.html)
-* [Log Search](Log-Search.html)
+* [Logs](Logs.html)
* [Dynamic Log Level Settings](dynamic-log-level-settings.html)
* [Dynamic Worker Profiling](dynamic-worker-profiling.html)
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/docs/documentation/Log-Search.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Log-Search.md b/docs/documentation/Log-Search.md
deleted file mode 100644
index 03297b7..0000000
--- a/docs/documentation/Log-Search.md
+++ /dev/null
@@ -1,14 +0,0 @@
-Log Search
-==========================
-
-This feature is aimed for improving the debugging of Storm. Log Search supports searching in a certain log file or in a topology's all log files:
-
-Using the Storm UI
--------------
-String search in a log file: In the log page for a worker, user can search a certain string, e.g., "Exception", in for a certain worker log. This search can happen for both normal text log or rolled ziplog files. In the results, the offset and matched lines will be displayed.
-
-
-
-Search in a topology: User can also search a string for a certain topology by clicking the icon of magnifying lens at the top right corner of the UI page. This means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology. The search can happen for both normal text log or rolled zip log files by checking/unchecking the "Search archived logs:" box. Then the matched results can be shown on the UI with url links, directing user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
-
-
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/docs/documentation/Logs.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Logs.md b/docs/documentation/Logs.md
new file mode 100644
index 0000000..0079853
--- /dev/null
+++ b/docs/documentation/Logs.md
@@ -0,0 +1,30 @@
+---
+title: Storm Logs
+layout: documentation
+documentation: true
+---
+Logs in Storm are essential for tracking the status, operations, error messages and debug information for all the
+daemons (nimbus, supervisor, logviewer, drpc, ui) and topologies' workers.
+
+### Location of the Logs
+All the daemon logs are placed under ${storm.log.dir} directory, which user can set in System property or
+in the cluster configuration. By default, ${storm.log.dir} points to ${storm.home}/logs.
+
+All the worker logs are placed under the workers-artifacts directory in a hierarchical manner, e.g.,
+${workers-artifacts}/${topologyId}/${port}/worker.log. Users can set the workers-artifacts directory
+by configuring the variable "storm.workers.artifacts.dir". By default, workers-artifacts directory
+locates at ${storm.log.dir}/logs/workers-artifacts.
+
+### Using the Storm UI for Log View/Download and Log Search
+Daemon and worker logs are allowed to view and download through Storm UI by authorized users.
+
+To improve the debugging of Storm, we provide the Log Search feature.
+Log Search supports searching in a certain log file or in a topology's all log files:
+
+String search in a log file: In the log page for a worker, user can search a certain string, e.g., "Exception", in for a certain worker log. This search can happen for both normal text log or rolled ziplog files. In the results, the offset and matched lines will be displayed.
+
+
+
+Search in a topology: User can also search a string for a certain topology by clicking the icon of magnifying lens at the top right corner of the UI page. This means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology. The search can happen for both normal text log or rolled zip log files by checking/unchecking the "Search archived logs:" box. Then the matched results can be shown on the UI with url links, directing user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
+
+
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/storm-core/src/clj/backtype/storm/config.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/config.clj b/storm-core/src/clj/backtype/storm/config.clj
index ffbbe99..d35dc56 100644
--- a/storm-core/src/clj/backtype/storm/config.clj
+++ b/storm-core/src/clj/backtype/storm/config.clj
@@ -94,6 +94,12 @@
(if (is-absolute-path? path) path (str storm-home file-path-separator path))
(str storm-home file-path-separator "storm-local"))))
+(def LOG-DIR
+ (.getCanonicalPath
+ (clojure.java.io/file (or (System/getProperty "storm.log.dir")
+ (get (read-storm-config) "storm.log.dir")
+ (str (System/getProperty "storm.home") file-path-separator "logs")))))
+
(defn absolute-healthcheck-dir [conf]
(let [storm-home (System/getProperty "storm.home")
path (conf STORM-HEALTH-CHECK-DIR)]
@@ -258,8 +264,8 @@
(if workers-artifacts-dir
(if (is-absolute-path? workers-artifacts-dir)
workers-artifacts-dir
- (str backtype.storm.util/LOG-DIR file-path-separator workers-artifacts-dir))
- (str backtype.storm.util/LOG-DIR file-path-separator "workers-artifacts"))))
+ (str backtype.storm.config/LOG-DIR file-path-separator workers-artifacts-dir))
+ (str backtype.storm.config/LOG-DIR file-path-separator "workers-artifacts"))))
([conf id]
(str (worker-artifacts-root conf) file-path-separator id))
([conf id port]
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 3acc348..3f4c322 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -1022,7 +1022,7 @@
storm-home (System/getProperty "storm.home")
storm-options (System/getProperty "storm.options")
storm-conf-file (System/getProperty "storm.conf.file")
- storm-log-dir (or (System/getProperty "storm.log.dir") (str storm-home file-path-separator "logs"))
+ storm-log-dir backtype.storm.config/LOG-DIR
storm-log-conf-dir (conf STORM-LOG4J2-CONF-DIR)
storm-log4j2-conf-dir (if storm-log-conf-dir
(if (is-absolute-path? storm-log-conf-dir)
http://git-wip-us.apache.org/repos/asf/storm/blob/288d6054/storm-core/src/clj/backtype/storm/util.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/util.clj b/storm-core/src/clj/backtype/storm/util.clj
index fcc8d66..8b61ca8 100644
--- a/storm-core/src/clj/backtype/storm/util.clj
+++ b/storm-core/src/clj/backtype/storm/util.clj
@@ -1040,10 +1040,6 @@
(list form x)))
([x form & more] `(-<> (-<> ~x ~form) ~@more)))
-(def LOG-DIR
- (.getCanonicalPath
- (clojure.java.io/file (or (System/getProperty "storm.log.dir") (str (System/getProperty "storm.home") "logs")))))
-
(defn logs-filename
[storm-id port]
(str storm-id file-path-separator port file-path-separator "worker.log"))
[4/5] storm git commit: Merge branch '1393' of
https://github.com/zhuoliu/storm
Posted by kn...@apache.org.
Merge branch '1393' of https://github.com/zhuoliu/storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/cbb59440
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/cbb59440
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/cbb59440
Branch: refs/heads/master
Commit: cbb594406777ef804d1e17a43ac71ea5012d9796
Parents: 8e708cf b77dc16
Author: Kyle Nusbaum <Ky...@gmail.com>
Authored: Thu Dec 17 14:42:00 2015 -0600
Committer: Kyle Nusbaum <Ky...@gmail.com>
Committed: Thu Dec 17 14:42:00 2015 -0600
----------------------------------------------------------------------
docs/documentation/Documentation.md | 2 +-
docs/documentation/Log-Search.md | 14 ---------
docs/documentation/Logs.md | 30 ++++++++++++++++++++
storm-core/src/clj/backtype/storm/config.clj | 10 +++++--
.../clj/backtype/storm/daemon/supervisor.clj | 2 +-
storm-core/src/clj/backtype/storm/util.clj | 4 ---
6 files changed, 40 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
[5/5] storm git commit: Adding STORM-1393 to CHANGELOG.
Posted by kn...@apache.org.
Adding STORM-1393 to CHANGELOG.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6cf33a86
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6cf33a86
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6cf33a86
Branch: refs/heads/master
Commit: 6cf33a86ddd0c605261097100685f5b4893d40e7
Parents: cbb5944
Author: Kyle Nusbaum <Ky...@gmail.com>
Authored: Thu Dec 17 14:45:07 2015 -0600
Committer: Kyle Nusbaum <Ky...@gmail.com>
Committed: Thu Dec 17 14:45:07 2015 -0600
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/6cf33a86/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1555adc..96e8e41 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 0.11.0
+ * STORM-1393: Update the storm.log.dir function, add doc for logs
* STORM-1377: nimbus_auth_test: very short timeouts causing spurious failures
* STORM-1388: Fix url and email links in README file
* STORM-1389: Removed creation of projection tuples as they are already available
[2/5] storm git commit: Minor
Posted by kn...@apache.org.
Minor
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/a5e4129f
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/a5e4129f
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/a5e4129f
Branch: refs/heads/master
Commit: a5e4129f460cdc19f7bac17fc8702bf6f856917d
Parents: 288d605
Author: zhuol <zh...@yahoo-inc.com>
Authored: Wed Dec 16 09:32:30 2015 -0600
Committer: zhuol <zh...@yahoo-inc.com>
Committed: Wed Dec 16 09:32:30 2015 -0600
----------------------------------------------------------------------
docs/documentation/Logs.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/a5e4129f/docs/documentation/Logs.md
----------------------------------------------------------------------
diff --git a/docs/documentation/Logs.md b/docs/documentation/Logs.md
index 0079853..a0961ea 100644
--- a/docs/documentation/Logs.md
+++ b/docs/documentation/Logs.md
@@ -7,7 +7,7 @@ Logs in Storm are essential for tracking the status, operations, error messages
daemons (nimbus, supervisor, logviewer, drpc, ui) and topologies' workers.
### Location of the Logs
-All the daemon logs are placed under ${storm.log.dir} directory, which user can set in System property or
+All the daemon logs are placed under ${storm.log.dir} directory, which a user can set in the System properties or
in the cluster configuration. By default, ${storm.log.dir} points to ${storm.home}/logs.
All the worker logs are placed under the workers-artifacts directory in a hierarchical manner, e.g.,
@@ -21,10 +21,10 @@ Daemon and worker logs are allowed to view and download through Storm UI by auth
To improve the debugging of Storm, we provide the Log Search feature.
Log Search supports searching in a certain log file or in a topology's all log files:
-String search in a log file: In the log page for a worker, user can search a certain string, e.g., "Exception", in for a certain worker log. This search can happen for both normal text log or rolled ziplog files. In the results, the offset and matched lines will be displayed.
+String search in a log file: In the log page for a worker, a user can search a certain string, e.g., "Exception", in for a certain worker log. This search can happen for both normal text log or rolled ziplog files. In the results, the offset and matched lines will be displayed.
-Search in a topology: User can also search a string for a certain topology by clicking the icon of magnifying lens at the top right corner of the UI page. This means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology. The search can happen for both normal text log or rolled zip log files by checking/unchecking the "Search archived logs:" box. Then the matched results can be shown on the UI with url links, directing user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
+Search in a topology: a user can also search a string for a certain topology by clicking the icon of magnifying lens at the top right corner of the UI page. This means the UI will try to search on all the supervisor nodes in a distributed way to find the matched string in all logs for this topology. The search can happen for both normal text log or rolled zip log files by checking/unchecking the "Search archived logs:" box. Then the matched results can be shown on the UI with url links, directing the user to the certain logs on each supervisor node. This powerful feature is very helpful for users to find certain problematic supervisor nodes running this topology.
[3/5] storm git commit: namespace dir
Posted by kn...@apache.org.
namespace dir
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b77dc162
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b77dc162
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b77dc162
Branch: refs/heads/master
Commit: b77dc16247190845ce6bb9c7bc4fc6eeb4c69d57
Parents: a5e4129
Author: zhuol <zh...@yahoo-inc.com>
Authored: Wed Dec 16 12:14:51 2015 -0600
Committer: zhuol <zh...@yahoo-inc.com>
Committed: Wed Dec 16 12:14:51 2015 -0600
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/config.clj | 4 ++--
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/b77dc162/storm-core/src/clj/backtype/storm/config.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/config.clj b/storm-core/src/clj/backtype/storm/config.clj
index d35dc56..04b16e0 100644
--- a/storm-core/src/clj/backtype/storm/config.clj
+++ b/storm-core/src/clj/backtype/storm/config.clj
@@ -264,8 +264,8 @@
(if workers-artifacts-dir
(if (is-absolute-path? workers-artifacts-dir)
workers-artifacts-dir
- (str backtype.storm.config/LOG-DIR file-path-separator workers-artifacts-dir))
- (str backtype.storm.config/LOG-DIR file-path-separator "workers-artifacts"))))
+ (str LOG-DIR file-path-separator workers-artifacts-dir))
+ (str LOG-DIR file-path-separator "workers-artifacts"))))
([conf id]
(str (worker-artifacts-root conf) file-path-separator id))
([conf id port]
http://git-wip-us.apache.org/repos/asf/storm/blob/b77dc162/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 3f4c322..77bdfd1 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -1022,7 +1022,7 @@
storm-home (System/getProperty "storm.home")
storm-options (System/getProperty "storm.options")
storm-conf-file (System/getProperty "storm.conf.file")
- storm-log-dir backtype.storm.config/LOG-DIR
+ storm-log-dir LOG-DIR
storm-log-conf-dir (conf STORM-LOG4J2-CONF-DIR)
storm-log4j2-conf-dir (if storm-log-conf-dir
(if (is-absolute-path? storm-log-conf-dir)