You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by agresch <gi...@git.apache.org> on 2018/05/03 18:24:59 UTC
[GitHub] storm pull request #2659: STORM-3042 restore topology.acker.cpu.pcore.percen...
GitHub user agresch opened a pull request:
https://github.com/apache/storm/pull/2659
STORM-3042 restore topology.acker.cpu.pcore.percent
TOPOLOGY_ACKER_CPU_PCORE_PERCENT and other similar settings in adjustResourcesForExec() were orphaned accidentally by a previous pull request. This adds the functionality back.
When we deployed the change to clusters that were near full with some topologies that had this setting, because we were not previously honoring it, the nodes now looked over 100% full and were throwing exceptions during scheduling.
The fix for this was to instead zero out the resources and track that the issue is occurring. We had also logged that the error was occurring, and found that the logging became excessive, which also slowed scheduling. A further change was added by @revans2 to limit the logging.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/agresch/storm agresch_storm3042
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2659.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2659
----
commit 75dc1825252ecbadf3317a141458bee6323404f6
Author: Aaron Gresch <ag...@...>
Date: 2018-05-03T18:09:56Z
STORM-3042 restore topology.acker.cpu.pcore.percent
----
---
[GitHub] storm issue #2659: STORM-3042 restore topology.acker.cpu.pcore.percent
Posted by agresch <gi...@git.apache.org>.
Github user agresch commented on the issue:
https://github.com/apache/storm/pull/2659
@HeartSaVioR - rebased.
---
[GitHub] storm pull request #2659: STORM-3042 restore topology.acker.cpu.pcore.percen...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/2659
---
[GitHub] storm issue #2659: STORM-3042 restore topology.acker.cpu.pcore.percent
Posted by agresch <gi...@git.apache.org>.
Github user agresch commented on the issue:
https://github.com/apache/storm/pull/2659
@Ethanlm - updated the issues you mentioned.
---
[GitHub] storm pull request #2659: STORM-3042 restore topology.acker.cpu.pcore.percen...
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on a diff in the pull request:
https://github.com/apache/storm/pull/2659#discussion_r185929379
--- Diff: storm-server/src/main/java/org/apache/storm/scheduler/TopologyDetails.java ---
@@ -293,7 +295,7 @@ public Double getTotalMemReqTask(ExecutorDetails exec) {
* Gets the total memory resource list for a set of tasks that is part of a topology.
*
* @param executors all executors for a topology
- * @return Map<ExecutorDetails , Double> , a map of the total memory requirement for all tasks in topology topoId.
+ * @return Map<ExecutorDetails> a map of the total memory requirement for all tasks in topology topoId.
--- End diff --
Looks not right to me. `@return Set<SharedMemory>` ?
---
[GitHub] storm issue #2659: STORM-3042 restore topology.acker.cpu.pcore.percent
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2659
@agresch Could you please rebase? After rebase we can review quickly and merge this in.
---
[GitHub] storm pull request #2659: STORM-3042 restore topology.acker.cpu.pcore.percen...
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on a diff in the pull request:
https://github.com/apache/storm/pull/2659#discussion_r185929506
--- Diff: storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -533,11 +533,11 @@ private static StormBase make(TopologyStatus status) {
private static <T extends AutoCloseable> TimeCacheMap<String, T> fileCacheMap(Map<String, Object> conf) {
return new TimeCacheMap<>(ObjectReader.getInt(conf.get(DaemonConfig.NIMBUS_FILE_COPY_EXPIRATION_SECS), 600),
(id, stream) -> {
- try {
- stream.close();
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
+ try {
--- End diff --
The format seems off
---