You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by jerrypeng <gi...@git.apache.org> on 2016/03/15 22:51:48 UTC
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bug when launching...
GitHub user jerrypeng opened a pull request:
https://github.com/apache/storm/pull/1216
[STORM-1631] - Storm CGroup bug when launching workers as the user that submitted the topology
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerrypeng/storm STORM-1631
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1216.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1216
----
commit f679dac1733e5c20fabfda34634d0470466eb539
Author: Boyang Jerry Peng <je...@yahoo-inc.com>
Date: 2016-03-15T21:43:58Z
[STORM-1631] - Storm CGroup bug when launching workers as the user that submitted the topology
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-197209965
It looks good to me, +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-220783029
@yegortokmakov in general, for one topology:
i:woker number
worker number < cluster node number
worker/per node
ii:thread number
this depends on your logic. for example, the parallelism of KafkaSpout should be equal to the number of partitions, in order to get a good performance.
btw, I really recommend you to upgrade your storm to 1.0.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-199978373
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by yimapingchuanto <gi...@git.apache.org>.
Github user yimapingchuanto commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-220782088
i want to know if there is a rule about workers num and threads num. In production, i always set the thread num is 10 times than worker num. oh, i use storm 0.9.5.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by jerrypeng <gi...@git.apache.org>.
Github user jerrypeng commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-220783660
At yahoo, cgroups are used to limit both memory and cpu of every worker so
that topologies can have more resource isolation
On Saturday, May 21, 2016, Xin Wang <no...@github.com> wrote:
> @yegortokmakov <https://github.com/yegortokmakov> in general, for one
> topology:
> i:woker number
> worker number < cluster node number
> worker/per node
> ii:thread number
> this depends on your logic. for example, the parallelism of KafkaSpout
> should be equal to the number of partitions, in order to get a good
> performance.
>
> btw, I really recommend you to upgrade your storm to 1.0.1.
>
> \u2014
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/storm/pull/1216#issuecomment-220783029>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by caishize <gi...@git.apache.org>.
Github user caishize commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-220766704
is any one using cgroup to limit the cpu usage? these days i found the system will be halt when the pressure is growing up, and i found that much of the cpu was wasted by the system, above 25%
CentOS 6.5
64 bit
128 G
32 core
31 workers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launching worke...
Posted by caishize <gi...@git.apache.org>.
Github user caishize commented on the pull request:
https://github.com/apache/storm/pull/1216
I\u2019ve found that the system will be halt when there are some workers are reboot frequently, this may be a bug of the OS\uff0cand i found this https://bugs.centos.org/view.php?id=7538 may be the reason, i hadn\u2019t confirm this yet..
Darwin
On May 21, 2016, at 11:25 PM, Boyang Jerry Peng <no...@github.com>> wrote:
At yahoo, cgroups are used to limit both memory and cpu of every worker so
that topologies can have more resource isolation
On Saturday, May 21, 2016, Xin Wang <no...@github.com>> wrote:
> @yegortokmakov <https://github.com/yegortokmakov> in general, for one
> topology:
> i:woker number
> worker number < cluster node number
> worker/per node
> ii:thread number
> this depends on your logic. for example, the parallelism of KafkaSpout
> should be equal to the number of partitions, in order to get a good
> performance.
>
> btw, I really recommend you to upgrade your storm to 1.0.1.
>
> \u2014
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/storm/pull/1216#issuecomment-220783029>
>
\u2014
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub<https://github.com/apache/storm/pull/1216#issuecomment-220783660>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1216
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1631] - Storm CGroup bugs 1) when launc...
Posted by vesense <gi...@git.apache.org>.
Github user vesense commented on the pull request:
https://github.com/apache/storm/pull/1216#issuecomment-220773192
@caishize too many threads will cause high sys cpu usage. much worse situation, the node will be halt.
I see there are 31 workers in your machine, what's the total thread number? reducing the number of threads should bring a improvement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---