You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/02/26 15:47:00 UTC

[GitHub] [storm] govind-menon opened a new pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

govind-menon opened a new pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217
 
 
   … prevents starvation and fragmentation

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217#discussion_r402630398
 
 

 ##########
 File path: storm-server/src/main/java/org/apache/storm/scheduler/Topologies.java
 ##########
 @@ -76,6 +76,14 @@ public Topologies(Topologies src) {
         return ret;
     }
 
+    public void addTopology(TopologyDetails details) {
 
 Review comment:
   We shouldn't add methods to regular code while it's only used in unit test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] kishorvpatil commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

Posted by GitBox <gi...@apache.org>.
kishorvpatil commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217#discussion_r387721854
 
 

 ##########
 File path: storm-server/src/test/java/org/apache/storm/scheduler/resource/strategies/scheduling/TestGenericResourceAwareStrategy.java
 ##########
 @@ -300,7 +302,64 @@ public void testGrasRequiringEviction() {
         assertTopologiesFullyScheduled(cluster, noGpu);
         assertTopologiesFullyScheduled(cluster, gpu2);
     }
-    
+
+    /*
+     * test GRAS should not evict other topologies for the sake of satisfying generic resource requirements.
+     * GRAS should also not resort nodes and 'pack' a node instead of spreading it out causing
+     * fragmentation.
+     */
+    @Test
+    public void testGrasStarvation() {
+        double cpuPercent = 200;
+        double memoryOnHeap = 100;
+        double memoryOffHeap = 100;
+
+        TopologyBuilder builder = new TopologyBuilder();
+
+        // non-gpu topology
+        builder.setSpout("spout", new TestSpout(), 2);
+        StormTopology stormTopologyNoGpu = builder.createTopology();
+        String noGpu = "hasNoGpu";
+        List<TopologyDetails> topologyDetails = new ArrayList();
+        Config nonGpuconf = createClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null);
+        nonGpuconf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, DefaultResourceAwareStrategy.class.getName());
+
+        cpuPercent = 100;
+
+        // gpu topology (requires 4 gpu's in total)
+        builder = new TopologyBuilder();
+        builder.setSpout("spout", new TestSpout(), 4).addResource("gpu.count", 1.0);
+        StormTopology stormTopologyWithGpu = builder.createTopology();
+
+
+        Config conf = createGrasClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null, Collections.emptyMap());
+        conf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, GenericResourceAwareStrategy.class.getName());
+
+        String gpu = "hasGpu";
+        Topologies topologies = new Topologies(createTestStormTopology(stormTopologyWithGpu, 9, gpu, conf));
+
+        Map<String, Double> genericResourcesMap = new HashMap<>();
+        genericResourcesMap.put("gpu.count", 2.0);
+        Map<String, SupervisorDetails> supMap = genSupervisors(4, 4, 200, 2000, genericResourcesMap);
+        Cluster cluster = new Cluster(new INimbusTest(), new ResourceMetrics(new StormMetricsRegistry()), supMap, new HashMap<>(), topologies, conf);
+
+        // should schedule gpu and noGpu successfully
+        scheduler = new ResourceAwareScheduler();
+        nonGpuconf.put(DaemonConfig.RESOURCE_AWARE_SCHEDULER_MAX_TOPOLOGY_SCHEDULING_ATTEMPTS, 1); // allows no evictions
+        scheduler.prepare(nonGpuconf);
+        // schedule gpu topology
+        scheduler.schedule(topologies, cluster);
+
+        topologies.addTopology(createTestStormTopology(stormTopologyNoGpu,  10, noGpu, nonGpuconf));
 
 Review comment:
   This is supposed to be immutable class. just create another instance of topologies using constructor.  
   
   try ` Topologies topologies = new Topologies(topo[0], topo[1]);`
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217#discussion_r384816921
 
 

 ##########
 File path: storm-server/src/test/java/org/apache/storm/scheduler/resource/strategies/scheduling/TestGenericResourceAwareStrategy.java
 ##########
 @@ -300,7 +302,63 @@ public void testGrasRequiringEviction() {
         assertTopologiesFullyScheduled(cluster, noGpu);
         assertTopologiesFullyScheduled(cluster, gpu2);
     }
-    
+
+    /*
+     * test GRAS should not evict other topologies for the sake of satisfying generic resource requirements.
+     * GRAS should also not resort nodes and 'pack' a node instead of spreading it out causing
+     * fragmentation.
+     */
+    @Test
+    public void testGrasStarvation() {
+        double cpuPercent = 200;
+        double memoryOnHeap = 100;
+        double memoryOffHeap = 100;
+
+        TopologyBuilder builder = new TopologyBuilder();
+
+        // non-gpu topology
+        builder.setSpout("spout", new TestSpout(), 2);
+        StormTopology stormTopologyNoGpu = builder.createTopology();
+        String noGpu = "hasNoGpu";
+        List<TopologyDetails> topologyDetails = new ArrayList();
+        Config nonGpuconf = createClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null);
+        nonGpuconf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, DefaultResourceAwareStrategy.class.getName());
+
+        cpuPercent = 100;
+
+        // gpu topology (requires 4 gpu's in total)
+        builder = new TopologyBuilder();
+        builder.setSpout("spout", new TestSpout(), 4).addResource("gpu.count", 1.0);
+        StormTopology stormTopologyWithGpu = builder.createTopology();
+
+
+        Config conf = createGrasClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null, Collections.emptyMap());
+        conf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, GenericResourceAwareStrategy.class.getName());
+
+        String gpu = "hasGpu";
+        Topologies topologies = new Topologies(createTestStormTopology(stormTopologyWithGpu, 10, gpu, conf));
+
+        Map<String, Double> genericResourcesMap = new HashMap<>();
+        genericResourcesMap.put("gpu.count", 2.0);
+        Map<String, SupervisorDetails> supMap = genSupervisors(4, 4, 200, 2000, genericResourcesMap);
+        Cluster cluster = new Cluster(new INimbusTest(), new ResourceMetrics(new StormMetricsRegistry()), supMap, new HashMap<>(), topologies, conf);
+
+        // should schedule gpu and noGpu successfully
+        scheduler = new ResourceAwareScheduler();
+        nonGpuconf.put(DaemonConfig.RESOURCE_AWARE_SCHEDULER_MAX_TOPOLOGY_SCHEDULING_ATTEMPTS, 1); // allows no evictions
+        scheduler.prepare(nonGpuconf);
+        // schedule gpu topology
+        scheduler.schedule(topologies, cluster);
+
+        cluster.addTopologyToClusterToBeScheduled(createTestStormTopology(stormTopologyNoGpu,  10, noGpu, nonGpuconf));
 
 Review comment:
    It doesn't look like a good idea to add a new function in the regular code solely for unit test. We don't need to add this function to achieve what we want. We can follow this https://github.com/apache/storm/blob/master/storm-server/src/test/java/org/apache/storm/scheduler/resource/TestResourceAwareScheduler.java#L504-L511

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] kishorvpatil commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

Posted by GitBox <gi...@apache.org>.
kishorvpatil commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217#discussion_r387720778
 
 

 ##########
 File path: storm-server/src/main/java/org/apache/storm/scheduler/Topologies.java
 ##########
 @@ -76,6 +76,14 @@ public Topologies(Topologies src) {
         return ret;
     }
 
+    public void addTopology(TopologyDetails details) {
 
 Review comment:
   This is supposed to be immutable class. just create another instance of topologies using constructor. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…

Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3217: STORM-3590: Adds test to validate that GRAS's node sort is stable and…
URL: https://github.com/apache/storm/pull/3217#discussion_r402631134
 
 

 ##########
 File path: storm-server/src/test/java/org/apache/storm/scheduler/resource/strategies/scheduling/TestGenericResourceAwareStrategy.java
 ##########
 @@ -300,7 +302,64 @@ public void testGrasRequiringEviction() {
         assertTopologiesFullyScheduled(cluster, noGpu);
         assertTopologiesFullyScheduled(cluster, gpu2);
     }
-    
+
+    /*
+     * test GRAS should not evict other topologies for the sake of satisfying generic resource requirements.
+     * GRAS should also not resort nodes and 'pack' a node instead of spreading it out causing
+     * fragmentation.
+     */
+    @Test
+    public void testGrasStarvation() {
+        double cpuPercent = 200;
+        double memoryOnHeap = 100;
+        double memoryOffHeap = 100;
+
+        TopologyBuilder builder = new TopologyBuilder();
+
+        // non-gpu topology
+        builder.setSpout("spout", new TestSpout(), 2);
+        StormTopology stormTopologyNoGpu = builder.createTopology();
+        String noGpu = "hasNoGpu";
+        List<TopologyDetails> topologyDetails = new ArrayList();
+        Config nonGpuconf = createClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null);
+        nonGpuconf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, DefaultResourceAwareStrategy.class.getName());
+
+        cpuPercent = 100;
+
+        // gpu topology (requires 4 gpu's in total)
+        builder = new TopologyBuilder();
+        builder.setSpout("spout", new TestSpout(), 4).addResource("gpu.count", 1.0);
+        StormTopology stormTopologyWithGpu = builder.createTopology();
+
+
+        Config conf = createGrasClusterConfig(cpuPercent, memoryOnHeap, memoryOffHeap, null, Collections.emptyMap());
+        conf.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, GenericResourceAwareStrategy.class.getName());
+
+        String gpu = "hasGpu";
+        Topologies topologies = new Topologies(createTestStormTopology(stormTopologyWithGpu, 9, gpu, conf));
+
+        Map<String, Double> genericResourcesMap = new HashMap<>();
+        genericResourcesMap.put("gpu.count", 2.0);
+        Map<String, SupervisorDetails> supMap = genSupervisors(4, 4, 200, 2000, genericResourcesMap);
+        Cluster cluster = new Cluster(new INimbusTest(), new ResourceMetrics(new StormMetricsRegistry()), supMap, new HashMap<>(), topologies, conf);
+
+        // should schedule gpu and noGpu successfully
+        scheduler = new ResourceAwareScheduler();
+        nonGpuconf.put(DaemonConfig.RESOURCE_AWARE_SCHEDULER_MAX_TOPOLOGY_SCHEDULING_ATTEMPTS, 1); // allows no evictions
 
 Review comment:
   This is not able to indicate if eviction happens or not.
   Waiting on https://github.com/apache/storm/pull/3213 so we can have a right way to detect eviction. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services