You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Ethan Li (Jira)" <ji...@apache.org> on 2020/07/07 16:36:00 UTC
[jira] [Updated] (STORM-3668)
Nimbus.validateTopologyWorkerMaxHeapSizeConfigs doesn't work properly when
a topology doesn't set component heap size explicitly
[ https://issues.apache.org/jira/browse/STORM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Li updated STORM-3668:
----------------------------
Description:
This [validateTopologyWorkerMaxHeapSizeConfigs|https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L3150-L3151] takes topoConf to get the default heap size for components. But at this point, the topoConf might not have "topology.component.resources.onheap.memory.mb". And in this case, validateTopologyWorkerMaxHeapSizeConfigs just passes.
For example, submission with
{code:java}
storm jar storm-starter.jar org.apache.storm.starter.WordCountTopology -c topology.worker.max.heap.size.mb=100 wc
{code}
will succeed while the config "topology.component.resources.onheap.memory.mb" is default to 120mb > 100mb.
While submission succeeds, nimbus is not able to schedule the topology.
{code:java}
2020-07-06 21:41:07.376 o.a.s.s.r.s.s.BaseResourceAwareStrategy pool-21-thread-1 [WARN] Not Enough Resources to schedule Task [6, 6] - split Normalized resources: {onheap.memory.mb=128.0, cpu.pcore.percent=10.0, offheap.memory.mb=0.0}
{code}
Note: this is when ResourceAwareScheduler is being used.
was:
This [validateTopologyWorkerMaxHeapSizeConfigs|https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L3150-L3151] takes topoConf to get the default heap size for components. But at this point, the topoConf might not have "topology.component.resources.onheap.memory.mb". And in this case, validateTopologyWorkerMaxHeapSizeConfigs just passes.
For example, submission with
{code:java}
storm jar storm-starter.jar org.apache.storm.starter.WordCountTopology -c topology.worker.max.heap.size.mb=100 wc
{code}
will succeed while the config "topology.component.resources.onheap.memory.mb" is default to 120mb > 100mb.
While submission succeeds, nimbus is not able to schedule the topology.
{code:java}
2020-07-06 21:41:07.376 o.a.s.s.r.s.s.BaseResourceAwareStrategy pool-21-thread-1 [WARN] Not Enough Resources to schedule Task [6, 6] - split Normalized resources: {onheap.memory.mb=128.0, cpu.pcore.percent=10.0, offheap.memory.mb=0.0}
{code}
> Nimbus.validateTopologyWorkerMaxHeapSizeConfigs doesn't work properly when a topology doesn't set component heap size explicitly
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: STORM-3668
> URL: https://issues.apache.org/jira/browse/STORM-3668
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Ethan Li
> Priority: Minor
>
> This [validateTopologyWorkerMaxHeapSizeConfigs|https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L3150-L3151] takes topoConf to get the default heap size for components. But at this point, the topoConf might not have "topology.component.resources.onheap.memory.mb". And in this case, validateTopologyWorkerMaxHeapSizeConfigs just passes.
> For example, submission with
> {code:java}
> storm jar storm-starter.jar org.apache.storm.starter.WordCountTopology -c topology.worker.max.heap.size.mb=100 wc
> {code}
> will succeed while the config "topology.component.resources.onheap.memory.mb" is default to 120mb > 100mb.
> While submission succeeds, nimbus is not able to schedule the topology.
> {code:java}
> 2020-07-06 21:41:07.376 o.a.s.s.r.s.s.BaseResourceAwareStrategy pool-21-thread-1 [WARN] Not Enough Resources to schedule Task [6, 6] - split Normalized resources: {onheap.memory.mb=128.0, cpu.pcore.percent=10.0, offheap.memory.mb=0.0}
> {code}
> Note: this is when ResourceAwareScheduler is being used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)