You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Ethanlm <gi...@git.apache.org> on 2017/09/14 20:22:09 UTC
[GitHub] storm pull request #2325: [STORM-2738] The number of ackers should default t...
GitHub user Ethanlm opened a pull request:
https://github.com/apache/storm/pull/2325
[STORM-2738] The number of ackers should default to the number of actual running workers on RAS cluster
**Problem**:
If `topology.acker.executors` is not set, the number of ackers will be equal to `topology.workers`. But on RAS cluster, we don't set `topology.workers` because the number of workers will be determined by the scheduler. So in this case, the number of ackers will always be 1.
For example, there are 5 workers but only 1 acker.
**Analysis**:
The number of ackers has to be computed before scheduling happens, so it knows how to schedule the topology. The number of workers is not set until the topology is scheduled, so it is a bit of a chicken and egg problem.
**Solution**:
We could probably use the total amount of requested memory when the topology is submitted divided by the memory per worker to get an estimate that is better than 1.
JIRA at https://issues.apache.org/jira/browse/STORM-2738
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ethanlm/storm STORM-2738
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2325.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2325
----
commit 9a1f86182fc8c15c3fe170c5da32f15e37463717
Author: Ethan Li <et...@gmail.com>
Date: 2017-09-13T18:55:47Z
[STORM-2738] The number of ackers should default to the number of actual running workers on RAS cluster
----
---
[GitHub] storm issue #2325: [STORM-2738] The number of ackers should default to the n...
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2325
Two tests:
**Confg**
`topology.workers` not set;
`topology.acker.executors` not set.
`topology.component.resources.onheap.memory.mb` is default: 128.
`worker.heap.memory.mb` is default: 768.
**Test1**
numOfAckerExec = Math.ceil(12 * 128 / 768) = 2.
**Test2**
numOfAckerExec =Math.ceil(25 * 128 / 768) = 5.
---
[GitHub] storm pull request #2325: [STORM-2738] The number of ackers should default t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/2325
---
[GitHub] storm issue #2325: [STORM-2738] The number of ackers should default to the n...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2325
The estimation makes sense. +1
@Ethanlm Could you resolve merge conflict? Thanks in advance.
---