You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by zhuoliu <gi...@git.apache.org> on 2015/10/30 16:16:13 UTC
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/835
[STORM-40] Turn worker garbage collection and heapdump on by default
Since STORM-901 worker artifacts directory just gets merged in, it provides a private place for us to turn on the garbage collection and heapdump for each worker by default.
This will greatly improve the debugging of storm.
Welcome discussion on the setting details (default gc rollover size, number of files, rwhether we want pid in the gc log file name, etc.),
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 40
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/835.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #835
----
commit 24886eec5c45e7fd30cac804fd080360f17599a0
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-30T15:10:32Z
[STORM-40] Turn worker garbage collection and heapdump on by default
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/835#issuecomment-152599615
@harshach the reason we have the gc logs on by default is because we have lots of users who ask why was my worker shot? Most of the time it is due to heavy GC and we wanted to have a smoking gun we could quickly look at to show them. The bigger problem for us is workers that log way too much.
I am +1 on this change, but if @harshach has concerns we can hold off on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/835#issuecomment-152579483
Thanks @harshach . This configuration is similar to what we have in Yahoo cluster.
Different types of workers may diff, mostly generating less than 100B per second. 24 workers per node might generate 2KB/s gc log data.
In terms of total size, it can be rolled over to be limited as 10 X 1MB files for each worker, which will not take too much disk space.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/835#issuecomment-152755528
Thanks for the details @zhuoliu I am +1 as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/835
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-40] Turn worker garbage collection and ...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/835#issuecomment-152553592
@zhuoliu heapdump will be helpful but I am not sure about the gc.log as it can produce lot of logging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---