You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by jessicasco <gi...@git.apache.org> on 2015/11/17 11:25:12 UTC
[GitHub] storm pull request: add storm-id to worker log filename
GitHub user jessicasco opened a pull request:
https://github.com/apache/storm/pull/886
add storm-id to worker log filename
add storm-id to worker log filename, so we can view logs group by the topology-id
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jessicasco/storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/886.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #886
----
commit c4022d233d7d60b4200ee8a8c762ca7f2dea9ee6
Author: 钱龙 <lo...@ele.me>
Date: 2015-11-17T10:21:21Z
add storm-id to worker log filename
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157594251
@wuchong How about 0.9.x version? I checked 0.9.6 still has `worker-{port}.log`, which mix different topology log to one file. Will the `workers-artifacts` log feature merge to 0.9.x version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] storm pull request: add storm-id to worker log filename
Posted by Cody Innowhere <e....@gmail.com>.
@jessicasco,
according to #STORM-901, worker log is already in storm-id directory, so is
it necessary to make this change?
On Wed, Nov 18, 2015 at 5:25 AM, kishorvpatil <gi...@git.apache.org> wrote:
> Github user kishorvpatil commented on the pull request:
>
> https://github.com/apache/storm/pull/886#issuecomment-157511940
>
> @jessicasco change worker log, requires changes to core ui, logviewer,
> worker in order to make sure the links and other search functionality don't
> break.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] storm pull request: add storm-id to worker log filename
Posted by kishorvpatil <gi...@git.apache.org>.
Github user kishorvpatil commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157511940
@jessicasco change worker log, requires changes to core ui, logviewer, worker in order to make sure the links and other search functionality don't break.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163689068
Hi Fang, @hustfxj. Great question! We also met the same problem. Actually our setting for gc log name is "-Xloggc:artifacts/gc.%p.log". Then gc log won't be overwritten due to worker restating.
Then we will also need to https://issues.apache.org/jira/browse/STORM-1206 to avoid too many gc files' names crashing the logviewer JVM (this might happen in a very very extreme and rare case, only once in our cluster).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163807599
very good! thank you @zhuoliu
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by darionyaphet <gi...@git.apache.org>.
Github user darionyaphet commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163610645
@jessicasco sorry I'm using 0.9.4 ..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157972534
@zhuoliu Thank you for your code snippet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157593727
@wuchong I got it. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163528643
@darionyaphet My code rename the log file to `{topology-id}-{port}.log`, but as @zhuoliu says they have a better solution since 0.10.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by darionyaphet <gi...@git.apache.org>.
Github user darionyaphet commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163185408
@jessicasco could be named with `topology-name` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by darionyaphet <gi...@git.apache.org>.
Github user darionyaphet commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163522470
@jessicasco yep
current log's name looks like `worker-{port}.log` rename to 'topo-name-{port}' maybe better ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157761155
Hi jessicasco, for the versions before workers-artifacts, the following code in util.clj might achieve what you want:
```
(defn- logs-rootname [storm-id port]
(str storm-id "-worker-" port))
(defn logs-filename [storm-id port]
(str (logs-rootname storm-id port) ".log"))
(defn logs-metadata-filename [storm-id port]
(str (logs-rootname storm-id port) ".yaml"))
(def worker-log-filename-pattern #"^((.*-\d+-\d+)-worker-(\d+))\.log")
(defn get-log-metadata-file
([fname]
(if-let [[_ _ id port] (re-matches worker-log-filename-pattern fname)]
(get-log-metadata-file id port)))
([id port]
(clojure.java.io/file LOG-DIR "metadata" (logs-metadata-filename id port))))
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157597586
@jessicasco yes, you are right.
The `workers-artifacts` feature is merged by @zhuoliu , you may ask him.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163607879
@zhuoliu Storm current log structures is:
storm-home/logs/
------supervisor.log
------nimbus.log
------workers-artifacts/
-----------topo1/
------------------port1/
------------------port2/
------------------------worker.log
------------------------worker.log.err
------------------------gc.log
worker's gc log's name maybe like {workers-artifacts}-{topo}-{port}-{worker}.gc. Supervisor will restart the worker if some worker dead. Then the worker's gc name is the same as before. I find the dead worker gc log is coverd with new gc log
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157588970
We have topology-id and port in log directory, current log structures under storm-local/workers-artifacts:
```
--topoId0
--topoId1
------port1
------port2
---------worker.log
---------worker.log.err
---------worker.log.out
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163507898
@darionyaphet, in 0.10 and before, worker log file has topology-id and port in its name. In latest version, the topo-name and port no longer exist in its same (please see [storm-901 | https://github.com/apache/storm/pull/822]). However, when user download log file from ui, topo-id and port will be added to the downloaded file's name automatically.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157603085
hi jessicasco, though worker-artifacts changes log directory structure and naming, when you download worker.log from UI, the storm id and port will be automatically append to the file name, which is quite convenient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-157627716
@zhuoliu Thank you for your explanation. Is there a simple way to split topology log in `worker-{port}.log` for storm 0.9.6?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco closed the pull request at:
https://github.com/apache/storm/pull/886
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: add storm-id to worker log filename
Posted by jessicasco <gi...@git.apache.org>.
Github user jessicasco commented on the pull request:
https://github.com/apache/storm/pull/886#issuecomment-163497824
@darionyaphet not `topology-id` but `topology-name`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---