You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "mgao0 (via GitHub)" <gi...@apache.org> on 2023/03/02 23:53:49 UTC

[GitHub] [helix] mgao0 opened a new pull request, #2390: Add API for users to provide customized threadpool for different categories of state transition messages

mgao0 opened a new pull request, #2390:
URL: https://github.com/apache/helix/pull/2390

   ### Issues
   
   - [ ] My PR addresses the following Helix issues and references them in the PR description:
   
   Resolves #2389
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI changes:
   
   Currently we have MessageInfo for categorizing messages, this commit adds one more property RebalanceType to MessageInfo so that user can provide threadpool based on this property. 
   
   ### Tests
   
   - [ ] The following tests are written for this issue:
   
   Added a section in TestHelixTaskExecutor. testUpdateAndFindMessageThreadpool().
   
   - The following is the result of the "mvn test" command on the appropriate module:
   
   Running
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   
   (Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124883503


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -290,6 +292,17 @@ private void updateStateTransitionMessageThreadPool(Message message, HelixManage
         manager.getStateMachineEngine().getStateModelFactory(stateModelName, factoryName);
 
     Message.MessageInfo msgInfo = new Message.MessageInfo(message);
+    if (stateModelFactory != null) {
+      StateModelFactory.CustomizedExecutorService customizedExecutorService =
+          stateModelFactory.getExecutorService(msgInfo);
+      if (customizedExecutorService != null) {
+        String msgInfoBasedKey = msgInfo.getMessageIdentifier(customizedExecutorService.getBase());
+        _msgInfoBasedThreadpoolChecked.add(msgInfoBasedKey);

Review Comment:
   Good catch, updated!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124882578


##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -115,6 +116,11 @@ public enum MessageState {
     UNPROCESSABLE // get exception when create handler
   }
 
+  public enum STRebalanceType {
+    LOAD_REBALANCE,
+    RECOVERY_REBALANCE
+  }
+

Review Comment:
   Yes, we have thought about reusing the RebalanceType in StateTransitionThrottleConfig class, but reusing it may cause confusion as this use case is not related to throttling. Separating it out may be an option, but the enum in StateTransitionThrottleConfig is widely used, and it will need a lot of code changes, overall it's not worthy to do it, so it is also not considered right now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124680876


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   Curious why do we iterate in reverse order? Can we change the order of declaration in the Enum?



##########
helix-core/src/main/java/org/apache/helix/participant/statemachine/StateModelFactory.java:
##########
@@ -260,4 +261,42 @@ public ExecutorService getExecutorService(String resourceName) {
   public ExecutorService getExecutorService(String resourceName, String fromState, String toState) {
     return null;
   }
+
+  public static class CustomizedExecutorService {
+    private Message.MessageInfo.MessageIdentifierBase _base;
+    private ExecutorService _executorService;

Review Comment:
   These two can be final



##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -115,6 +116,11 @@ public enum MessageState {
     UNPROCESSABLE // get exception when create handler
   }
 
+  public enum STRebalanceType {
+    LOAD_REBALANCE,
+    RECOVERY_REBALANCE
+  }
+

Review Comment:
   Not an issue, just a thought. I think there is existing `RebalanceType` in a throttling related class. For widely used and fundamental enum like this, maybe it deserves a separate class and reduce duplication



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on PR #2390:
URL: https://github.com/apache/helix/pull/2390#issuecomment-1456665421

   This PR is ready to be merged, approved by @qqu0127 @xyuanlu . Thanks.
   
   Final commit message: Add API for users to provide customized threadpool for different categories of state transition messages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124930300


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   Ok, I'm just saying we can reverse the declaration order, so the code can use normal iteration order with for-each. Not a big deal either way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1125030422


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   +1 on this. And we could use:
   `for (Message.MessageInfo.MessageIdentifierBase value : Message.MessageInfo.MessageIdentifierBase.values())`
   for simpler code and this value can be used in line 380 as well.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1125103828


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   Discussed with @xyuanlu offline. Changing enum order does simplify code, but users will not see this block of code because this is our internal logic. What users see is the enum declaration when they use the API, so making the enum easy to understand for users is more important. Therefore, keeping this order.
   The original comments I put besides each enum declaration are a bit confusing in terms of their hierarchy, updated them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1125029143


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +376,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {

Review Comment:
   nit: 
   `for (Message.MessageInfo.MessageIdentifierBase value : Message.MessageInfo.MessageIdentifierBase.values())`
   and this `value` can be used in line 380 as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124888691


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   In the Enum we declare types generally from coarser to finer, so we search in reverse order. If a finer type has a threadpool assigned, then we use the finer one. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1125004769


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -361,17 +374,13 @@ ExecutorService findExecutorServiceForMsg(Message message) {
         executorService = _batchMessageExecutorService;
       } else {
         Message.MessageInfo msgInfo = new Message.MessageInfo(message);
-        String perResourceTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_RESOURCE);
-        String perStateTransitionTypeKey =
-            msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.PER_STATE_TRANSITION_TYPE);
-        if (perStateTransitionTypeKey != null && _executorMap.containsKey(perStateTransitionTypeKey)) {
-          LOG.info(String.format("Find per state transition type thread pool for resource %s from %s to %s",
-              message.getResourceName(), message.getFromState(), message.getToState()));
-          executorService = _executorMap.get(perStateTransitionTypeKey);
-        } else if (_executorMap.containsKey(perResourceTypeKey)) {
-          LOG.info("Find per-resource thread pool with key: " + perResourceTypeKey);
-          executorService = _executorMap.get(perResourceTypeKey);
+        for (int i = Message.MessageInfo.MessageIdentifierBase.values().length - 1; i >= 0; i--) {
+          String msgIdentifer = msgInfo.getMessageIdentifier(Message.MessageInfo.MessageIdentifierBase.values()[i]);
+          if (msgIdentifer != null && _executorMap.containsKey(msgIdentifer)) {
+            LOG.info(String.format("Find customized threadpool for %s", msgIdentifer));
+            executorService = _executorMap.get(msgIdentifer);
+            break;

Review Comment:
   Oh I see, I think declare from lower level to higher level is more intuitive, so I will keep it this way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] mgao0 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "mgao0 (via GitHub)" <gi...@apache.org>.
mgao0 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124886737


##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -969,23 +987,32 @@ public boolean isValid() {
 
   /**
    * This class is for categorizing state transition messages based on certain properties, and generating
+   * an identifier string for each category
    */
   public static class MessageInfo {
     public String _msgType;
     public String _resourceName;
     public String _fromState;
     public String _toState;
+    public STRebalanceType _sTRebalanceType;
 
+    /**
+     * This class is used for defining what properties are used to categorize messages.
+     * Note that the bases should be arranged in a lower level -> higher level order,
+     * because we do backwards searching when looking for customized threadpool.
+     */
     public enum MessageIdentifierBase {
-      PER_RESOURCE,
-      PER_STATE_TRANSITION_TYPE
+      PER_RESOURCE, // Most basic level (L0), just include message type and resource name
+      PER_REBALANCE_TYPE, // One level above PER_RESOURCE (L1), concatenate rebalance type (load / recovery) after PER_RESOURCE identifier string

Review Comment:
   I put PER_REBALANCE_TYPE before PER_STATE_TRANSITION_TYPE is because currently most users are using  PER_STATE_TRANSITION_TYPE, so when we do reverse order traversal to look up for the threadpool, we search PER_STATE_TRANSITION_TYPE first, so it saves some checks. Since these two are in the same level, it should be okay.
   But this is a good suggestion, I found that the enum and the order in getMessageIdentifier() is not aligned, so I updated it, please take a look, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124893701


##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -115,6 +116,11 @@ public enum MessageState {
     UNPROCESSABLE // get exception when create handler
   }
 
+  public enum STRebalanceType {
+    LOAD_REBALANCE,
+    RECOVERY_REBALANCE
+  }
+

Review Comment:
   Agree, feel free to close this one. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1123962131


##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -969,23 +987,32 @@ public boolean isValid() {
 
   /**
    * This class is for categorizing state transition messages based on certain properties, and generating
+   * an identifier string for each category
    */
   public static class MessageInfo {
     public String _msgType;
     public String _resourceName;
     public String _fromState;
     public String _toState;
+    public STRebalanceType _sTRebalanceType;
 
+    /**
+     * This class is used for defining what properties are used to categorize messages.
+     * Note that the bases should be arranged in a lower level -> higher level order,
+     * because we do backwards searching when looking for customized threadpool.
+     */
     public enum MessageIdentifierBase {
-      PER_RESOURCE,
-      PER_STATE_TRANSITION_TYPE
+      PER_RESOURCE, // Most basic level (L0), just include message type and resource name
+      PER_REBALANCE_TYPE, // One level above PER_RESOURCE (L1), concatenate rebalance type (load / recovery) after PER_RESOURCE identifier string

Review Comment:
   nit: maybe the enum order align with categorize order? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1123958251


##########
helix-core/src/main/java/org/apache/helix/model/Message.java:
##########
@@ -660,6 +666,18 @@ public void setBucketSize(int bucketSize) {
     }
   }
 
+  public void setSTRebalanceType(STRebalanceType stRebalanceType) {
+    setAttribute(Attributes.ST_REBALANCE_TYPE, stRebalanceType.name());
+  }
+
+  public STRebalanceType getSTRebalanceType() {
+    String rebalanceTypeStr = getAttribute(Attributes.ST_REBALANCE_TYPE);
+    if (rebalanceTypeStr == null) {

Review Comment:
   nit: can be inline with 678



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] rahulrane50 commented on a diff in pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2390:
URL: https://github.com/apache/helix/pull/2390#discussion_r1124118485


##########
helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java:
##########
@@ -290,6 +292,17 @@ private void updateStateTransitionMessageThreadPool(Message message, HelixManage
         manager.getStateMachineEngine().getStateModelFactory(stateModelName, factoryName);
 
     Message.MessageInfo msgInfo = new Message.MessageInfo(message);
+    if (stateModelFactory != null) {
+      StateModelFactory.CustomizedExecutorService customizedExecutorService =
+          stateModelFactory.getExecutorService(msgInfo);
+      if (customizedExecutorService != null) {
+        String msgInfoBasedKey = msgInfo.getMessageIdentifier(customizedExecutorService.getBase());
+        _msgInfoBasedThreadpoolChecked.add(msgInfoBasedKey);

Review Comment:
   Curious, could msgInfoBasedKey be null?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu merged pull request #2390: Add API for users to provide customized threadpool for different categories of state transition messages

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2390:
URL: https://github.com/apache/helix/pull/2390


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org