You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/04/20 21:05:43 UTC

[GitHub] [storm] agresch opened a new pull request #3254: STORM-3596 use send assignment status in blacklist scheduling

agresch opened a new pull request #3254:
URL: https://github.com/apache/storm/pull/3254


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] agresch commented on pull request #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
agresch commented on pull request #3254:
URL: https://github.com/apache/storm/pull/3254#issuecomment-620241907


   @Ethanlm - updated a couple of your comments.  I'd rather leave the reporter as is for now.  I feel its message is already a bit of a mess.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] Ethanlm commented on a change in pull request #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3254:
URL: https://github.com/apache/storm/pull/3254#discussion_r413975679



##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/strategies/DefaultBlacklistStrategy.java
##########
@@ -72,13 +74,23 @@ public void prepare(Map<String, Object> conf) {
             }
         }
 
+        // update countMap failures for sendAssignments failing
+        for (Map<String, Integer> item : sendAssignmentFailureCount) {
+            for (Map.Entry<String, Integer> entry : item.entrySet()) {
+                String supervisorNode = entry.getKey();
+                int sendAssignmentFailures = entry.getValue() + countMap.getOrDefault(supervisorNode, 0);
+                countMap.put(supervisorNode, sendAssignmentFailures);
+            }
+        }
+
         for (Map.Entry<String, Integer> entry : countMap.entrySet()) {
             String supervisor = entry.getKey();
             int count = entry.getValue();
             if (count >= toleranceCount) {
                 if (!blacklist.containsKey(supervisor)) { // if not in blacklist then add it and set the resume time according to config
                     LOG.debug("Added supervisor {} to blacklist", supervisor);
                     LOG.debug("supervisorsWithFailures : {}", supervisorsWithFailures);
+                    LOG.debug("sendAssignmentFailureCount: {}", sendAssignmentFailureCount);
                     reporter.reportBlacklist(supervisor, supervisorsWithFailures);

Review comment:
       This `reportBlacklist` no longer fits the need since we added sentAssignmentFailure. Maybe need to fix it as a follow-up. Not sure what's the best way to do it

##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/BlacklistScheduler.java
##########
@@ -155,6 +163,18 @@ private void badSupervisors(Map<String, SupervisorDetails> supervisors) {
         badSupervisorsToleranceSlidingWindow.add(badSupervisors);
     }
 
+    private void trackAssignmentFailures() {
+        if (!blacklistSendAssignentFailures) {
+            return;
+        }
+        Map<String, Integer> assignmentFailureWindow = new HashMap<>();
+        assignmentFailureWindow.putAll(this.assignmentFailures);
+        synchronized (assignmentFailures) {

Review comment:
       Since we already `synchronized` on it, maybe should just as well put `assignmentFailureWindow.putAll(this.assignmentFailures);` inside the block and then it doesn't need to be a `concurrentHashMap`.  
   To reduce two synchronization mechanisms to one

##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/BlacklistScheduler.java
##########
@@ -51,12 +52,15 @@
     protected IBlacklistStrategy blacklistStrategy;
     protected int nimbusMonitorFreqSecs;
     protected Map<String, Set<Integer>> cachedSupervisors;
-    //key is supervisor key ,value is supervisor ports
-    protected EvictingQueue<HashMap<String, Set<Integer>>> badSupervisorsToleranceSlidingWindow;
+    // key is supervisor nodeId, value is supervisor ports
+    protected EvictingQueue<Map<String, Set<Integer>>> badSupervisorsToleranceSlidingWindow;
+    protected EvictingQueue<Map<String, Integer>> sendAssignmentFailureCount;
+    private Map<String, Integer> assignmentFailures = new ConcurrentHashMap<>();

Review comment:
       Make this `final` since we are `synchronize` on it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] agresch commented on issue #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
agresch commented on issue #3254:
URL: https://github.com/apache/storm/pull/3254#issuecomment-616808646


   @Ethanlm - reworked my previous feature on a new branch with the changes you requested.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] agresch commented on issue #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
agresch commented on issue #3254:
URL: https://github.com/apache/storm/pull/3254#issuecomment-618558747


   Please don't merge.  Will get back to some of @Ethanlm 's comments


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] Ethanlm commented on a change in pull request #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3254:
URL: https://github.com/apache/storm/pull/3254#discussion_r412415684



##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/BlacklistScheduler.java
##########
@@ -266,4 +286,15 @@ private Object initializeInstance(String className, String representation) {
     public Set<String> getBlacklistSupervisorIds() {
         return Collections.unmodifiableSet(blacklistedSupervisorIds);
     }
+
+    @Override
+    public void nodeAssignmentSent(String node, boolean successful) {
+        if (!blacklistSendAssignentFailures) {
+            return;
+        }
+        if (!successful) {
+            int failCount = assignmentFailures.getOrDefault(node, 0) + 1;
+            assignmentFailures.put(node, failCount);

Review comment:
       Synchronization is needed otherwise there can be a race condition between getting the current count and updating the count (line 296 and line297)
   
   Also between `this.assignmentFailures.clear();` and `assignmentFailures.put(node, failCount);

##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/strategies/IBlacklistStrategy.java
##########
@@ -35,7 +36,9 @@
      *       the `cluster` object.
      * @return blacklisted supervisors' id set
      */
-    Set<String> getBlacklist(List<Map<String, Set<Integer>>> badSupervisorsToleranceSlidingWindow, Cluster cluster, Topologies topologies);
+    Set<String> getBlacklist(List<Map<String, Set<Integer>>> badSupervisorsToleranceSlidingWindow,

Review comment:
       Looks like we need to modify the interface every time if we want to add another kind of failures.
   
   Technically this is a public interface and users can provide their own implementation so backwards compatibility should be preserved as long as possible. If we can make it backwards compatible, that would be fantastic.
   
   But I assume users don't really do that since this is actually tightly coupled and not really easy for users to provide their own implementation; even if they do, it is easy to make the change since the interface change is minimum. I feel this `IBlacklistStrategy` interface itself is not well defined and there are more refactoring to be done with this interface. So if it's hard to maintain backwards compatibility, this change is fine to me. Refactoring can be a follow-up

##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/strategies/DefaultBlacklistStrategy.java
##########
@@ -72,6 +74,15 @@ public void prepare(Map<String, Object> conf) {
             }
         }
 
+        // update countMap failures for sendAssignments failing
+        for (Map<String, Integer> item : sendAssignmentFailureCount) {

Review comment:
       since we have this now, Line92 and Line 93 need to be adjusted
   
   ```
                       LOG.debug("supervisorsWithFailures : {}", supervisorsWithFailures);
                       reporter.reportBlacklist(supervisor, supervisorsWithFailures);
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] agresch commented on a change in pull request #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
agresch commented on a change in pull request #3254:
URL: https://github.com/apache/storm/pull/3254#discussion_r414011898



##########
File path: storm-server/src/main/java/org/apache/storm/scheduler/blacklist/BlacklistScheduler.java
##########
@@ -155,6 +163,18 @@ private void badSupervisors(Map<String, SupervisorDetails> supervisors) {
         badSupervisorsToleranceSlidingWindow.add(badSupervisors);
     }
 
+    private void trackAssignmentFailures() {
+        if (!blacklistSendAssignentFailures) {
+            return;
+        }
+        Map<String, Integer> assignmentFailureWindow = new HashMap<>();
+        assignmentFailureWindow.putAll(this.assignmentFailures);
+        synchronized (assignmentFailures) {

Review comment:
       yes




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [storm] agresch commented on issue #3254: STORM-3596 use send assignment status in blacklist scheduling

Posted by GitBox <gi...@apache.org>.
agresch commented on issue #3254:
URL: https://github.com/apache/storm/pull/3254#issuecomment-617958773


   @Ethanlm - updated except for the interface rework.  Could you file a JIRA with what you have in mind?  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org