You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/10/30 06:17:28 UTC

[GitHub] [helix] jiajunwang opened a new pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

jiajunwang opened a new pull request #1500:
URL: https://github.com/apache/helix/pull/1500


   ### Issues
   
   - [X] My PR addresses the following Helix issues and references them in the PR description:
   
   Resolve #1488 
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI changes:
   
   This PR enhances the message event processing logic to prevent silent failure and unnecessary retry.
   1. If creating a message handler fails unexpectedly (meaning there is an Exception), then the message will be marked as UNPROCESSABLE unless the message is sent with a retry count larger than 0. When the retry count is configured, then before the message runs out of the retry count, the participant will keep retrying on any message callbacks.
   2. The UNPROCESSABLE message, which is generated due to the previous point, will be left in the participant message folder and not automatically removed. This is to prevent unnecessary retry.
   3. If the message handler fails due to the participant cannot schedule the task, then the message will be discarded. If the message is a state transition message, then the corresponding state model and the partition current state will be set to ERROR. This is also to prevent unnecessary retry.
   
   ### Tests
   
   - [X] The following tests are written for this issue:
   
   TestHelixTaskExecutor
   
   - [X] The following is the result of the "mvn test" command on the appropriate module:
   
   [WARNING] Tests run: 1237, Failures: 0, Errors: 0, Skipped: 1
   [INFO] 
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  02:00 h
   [INFO] Finished at: 2020-10-29T22:59:19-07:00
   [INFO] ------------------------------------------------------------------------
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] narendly commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514902018



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("Message %s has a negative remaining retry count %d. Stop processing it!",

Review comment:
       nit: instead of saying "negative retry count", what you really meant is "this message has been retried for "retryCount" times"
   
   also i'd print the `msgHandler` here. although we could deduce this info from the code, we might want to indicate that msgHandler is null in the log message as well for faster debugging




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang merged pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang merged pull request #1500:
URL: https://github.com/apache/helix/pull/1500


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517594223



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {

Review comment:
       I tried to make the change. But then realize it might be a non-backward compatible change.
   Overall, we only want to change the behavior IF the handler creating fails. But if we change like what @dasahcc mentioned, then the behavior is changed even if the handler creating logic will work fine. Given the send message interface is public, it is possible that the application configures -1 retry now.
   
   In addition, the logic that uses this timeout before this PR is also executing the task before look at this value. So I think we can keep the same behavior here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] zhangmeng916 commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r516897658



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("No available message Handler found!"
+                      + " Stop processing message %s since it has a negative remaining retry count %d!",
+                  message.getMsgId(), message.getRetryCount());
+          updateUnprocessableMessage(message, null, errorMsg, manager);

Review comment:
       Seems even there is no exception, you will still set the message as unprocessable after retry instead of leaving it as NEW?

##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -613,7 +613,12 @@ void reset() {
 
       MsgHandlerFactoryRegistryItem item = _hdlrFtyRegistry.get(msgType);
       if (item.factory() != null) {
-        item.factory().reset();
+        try {
+          item.factory().reset();
+        } catch (Exception ex) {

Review comment:
       I don't quite understand why we swallow exception here. This exception means user's reset has an issue, and can we just let the error go?

##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -993,67 +1034,78 @@ private boolean checkAndProcessNoOpMessage(Message message, String instanceName,
    * Preprocess the state transition message to validate if the request is valid.
    * If no operation needs to be triggered, discard the the message.
    * @param message
-   * @param instanceName
    * @param manager
    * @param stateTransitionHandlers
    * @param createHandler
    * @return True if the requested state transition is valid, and need to schedule the transition.
    *         False if no more operation is required.
    */
-  private boolean validateAndProcessStateTransitionMessage(Message message, String instanceName,
-      HelixManager manager, Map<String, MessageHandler> stateTransitionHandlers,
-      MessageHandler createHandler) {
-    HelixDataAccessor accessor = manager.getHelixDataAccessor();
-
+  private boolean validateAndProcessStateTransitionMessage(Message message, HelixManager manager,
+      Map<String, MessageHandler> stateTransitionHandlers, MessageHandler createHandler) {
     String messageTarget = getMessageTarget(message.getResourceName(), message.getPartitionName());
-    if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
-        && isStateTransitionInProgress(messageTarget)) {
-      String taskId = _messageTaskMap.get(messageTarget);
-      Message msg = _taskMap.get(taskId).getTask().getMessage();
-      // If there is another state transition for same partition is going on,
-      // discard the message. Controller will resend if this is a valid message
-      String errMsg = String.format(
-          "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
-          message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
-          msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
-          manager);
-      return false;
-    }
-    if (createHandler instanceof HelixStateTransitionHandler) {
-      // We only check to state if there is no ST task scheduled/executing.
-      HelixStateTransitionHandler.StaleMessageValidateResult result =
-          ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
-      if (!result.isValid) {
-        handleUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
-            accessor, instanceName, manager);
+
+    try {
+      if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
+          && isStateTransitionInProgress(messageTarget)) {
+        String taskId = _messageTaskMap.get(messageTarget);
+        Message msg = _taskMap.get(taskId).getTask().getMessage();
+        // If there is another state transition for same partition is going on,
+        // discard the message. Controller will resend if this is a valid message
+        String errMsg = String.format(
+            "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
+            message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
+            msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);
         return false;
       }
-    }
-    if (stateTransitionHandlers.containsKey(messageTarget)) {
-      // If there are 2 messages in same batch about same partition's state transition,
-      // the later one is discarded
-      Message duplicatedMessage = stateTransitionHandlers.get(messageTarget)._message;
-      String errMsg = String.format(
-          "Duplicated state transition message: %s. Existing: %s->%s; New (Discarded): %s->%s",
-          message.getMsgId(), duplicatedMessage.getFromState(), duplicatedMessage.getToState(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
+      if (createHandler instanceof HelixStateTransitionHandler) {
+        // We only check to state if there is no ST task scheduled/executing.
+        HelixStateTransitionHandler.StaleMessageValidateResult result =
+            ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
+        if (!result.isValid) {
+          updateUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
+              manager);
+          return false;
+        }
+      }
+      if (stateTransitionHandlers.containsKey(messageTarget)) {
+        // If there are 2 messages in same batch about same partition's state transition,
+        // the later one is discarded
+        Message duplicatedMessage = stateTransitionHandlers.get(messageTarget)._message;
+        String errMsg = String.format(
+            "Duplicated state transition message: %s. Existing: %s->%s; New (Discarded): %s->%s",
+            message.getMsgId(), duplicatedMessage.getFromState(), duplicatedMessage.getToState(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);

Review comment:
       I think we changed the original behavior, and never remove messages in this case. There is a mismatch in the comment. But, is this what we would like to do? not removing duplicated message, but only mark it? Seems we leave many unnecessary messages on ZK.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517773713



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {

Review comment:
       I think the basic reason is we reuse the same entry that used for transition fail timeout. We can keep the logic here but let's have a TODO for additional field, which may have the logic more clear for the new definition of the retry only before execution.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514911447



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);

Review comment:
       It is a good point.
   
   There were some discussions earlier yesterday about whether we want this retry count. So the decision is that we want to allow several retries, but would be a very small number. Note that this PR won't really add retry to state transition messages, since the default value of retry count is 0. We will tackle this later in the other PR.
   
   Now regarding if we want to use an in-memory object instead of ZK, it is totally doable. But I don't prefer doing so for 2 reasons.
   1. This ZK write won't make it worse. The retry is not infinite after this change. Compare with the current behavior that we keep retrying infinitely, the new retry logic is much safer. And we can set 0 to the retry count, so no retry at all. It is purely optional.
   2. In-memory count won't work if the participant instance is reset. If the retry count is really large, then in some corner case (like frequent participant reset) the retry will never stop.
   
   In addition, the in-memory design requires us to track the messages, it does not exist on the participant side today. So definitely, it will increase the complicity of code. There will be more corner cases that we need to consider. Like memory leakage, etc. Given the benefit of doing so is not critical, I prefer not to choose this option.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721933473


   > if the retry counts is large (say people put -1 instead 0), this is effectively retry forever. How can we make sure this is not going to happen?
   
   @kaisun2000 What do you mean? If it is -1, then the current logic will try once and stop retrying. And according to what Junkai mentioned, I would like to check for the count even before the first try. So that should not be the case, right?
   
   > Let say later the participant resolve issue and can create the messageHandler for this partition replica. Then does it mean when they recover, they need to manually (or via rest) to remove this UNPROCESSED message? If so, do we have this REST API?
   
   This is the plan. We have discussed this in the previous standup meeting. That task will be addressed separately. Before it is ready, we will require the application to reset the participant.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r516991349



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("No available message Handler found!"
+                      + " Stop processing message %s since it has a negative remaining retry count %d!",
+                  message.getMsgId(), message.getRetryCount());
+          updateUnprocessableMessage(message, null, errorMsg, manager);

Review comment:
       This is not true. The default retry count is 0. And we only reduce it if an exception happens. So if there is no exception, then it will keep retrying.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] narendly commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514905172



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);

Review comment:
       are we writing this back to zk or is this for in-memory use only? if we're writing back, would that be an added amount of zk writes/updates that we should worry about? 
   
   should be possible to track remaining retry counts purely in-memory per cached message using message id.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r516918363



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -613,7 +613,12 @@ void reset() {
 
       MsgHandlerFactoryRegistryItem item = _hdlrFtyRegistry.get(msgType);
       if (item.factory() != null) {
-        item.factory().reset();
+        try {
+          item.factory().reset();
+        } catch (Exception ex) {

Review comment:
       Just wondering, is it better to log the backtrace as well for better debuggability? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-722230763


   Approved by @dasahcc , I have added the comment as suggested. I will merge the PR shortly.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721959239


   > Let us say if the message have a retry count of INT_MAX, this is not going to work. I think we can be a little bit more conservative that if the message have a retry count larger than a threshold, we just retry the threshold value.
   
   @kaisun2000 , if the user explicitly sets the retry to be infinite, we shall not block it, right? Note that the retry won't cause any Helix controller issue, since it is only retried on the participant side. For ZK servers, yeah, it will lead to many ZK writes potentially. But I think that is a problem at a different level, ZK writes throttling for example.
   In addition, compared with the current behavior, even a retry count == INT_MAX works better. Since every retry in the new logic only generates one write. The older logic will remove the message and creating a new one. So at least 2 write IOs for each retry.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517581608



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -993,67 +1034,78 @@ private boolean checkAndProcessNoOpMessage(Message message, String instanceName,
    * Preprocess the state transition message to validate if the request is valid.
    * If no operation needs to be triggered, discard the the message.
    * @param message
-   * @param instanceName
    * @param manager
    * @param stateTransitionHandlers
    * @param createHandler
    * @return True if the requested state transition is valid, and need to schedule the transition.
    *         False if no more operation is required.
    */
-  private boolean validateAndProcessStateTransitionMessage(Message message, String instanceName,
-      HelixManager manager, Map<String, MessageHandler> stateTransitionHandlers,
-      MessageHandler createHandler) {
-    HelixDataAccessor accessor = manager.getHelixDataAccessor();
-
+  private boolean validateAndProcessStateTransitionMessage(Message message, HelixManager manager,
+      Map<String, MessageHandler> stateTransitionHandlers, MessageHandler createHandler) {
     String messageTarget = getMessageTarget(message.getResourceName(), message.getPartitionName());
-    if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
-        && isStateTransitionInProgress(messageTarget)) {
-      String taskId = _messageTaskMap.get(messageTarget);
-      Message msg = _taskMap.get(taskId).getTask().getMessage();
-      // If there is another state transition for same partition is going on,
-      // discard the message. Controller will resend if this is a valid message
-      String errMsg = String.format(
-          "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
-          message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
-          msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
-          manager);
-      return false;
-    }
-    if (createHandler instanceof HelixStateTransitionHandler) {
-      // We only check to state if there is no ST task scheduled/executing.
-      HelixStateTransitionHandler.StaleMessageValidateResult result =
-          ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
-      if (!result.isValid) {
-        handleUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
-            accessor, instanceName, manager);
+
+    try {
+      if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
+          && isStateTransitionInProgress(messageTarget)) {
+        String taskId = _messageTaskMap.get(messageTarget);
+        Message msg = _taskMap.get(taskId).getTask().getMessage();
+        // If there is another state transition for same partition is going on,
+        // discard the message. Controller will resend if this is a valid message
+        String errMsg = String.format(
+            "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
+            message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
+            msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);

Review comment:
       Please refer to what I reply to Meng earlier for the same code. I just move the remove logic a little bit for reusing the code. The messages will still be removed. Please check line 861.
   
   Let me know if you still think something missed here. I agree this is very critical, and I also want to ensure nothing changed here. So please help to double-check for me : )




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721922670


   >If creating a message handler fails unexpectedly (meaning there is an Exception), then the message will be marked as UNPROCESSABLE unless the message is sent with a retry count larger than 0. When the retry count is configured, then before the message runs out of the retry count, the participant will keep retrying on any message callbacks
   
   if the retry counts is large (say people put -1 instead 0), this is effectively retry forever. How can we make sure this is not going to happen?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517581608



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -993,67 +1034,78 @@ private boolean checkAndProcessNoOpMessage(Message message, String instanceName,
    * Preprocess the state transition message to validate if the request is valid.
    * If no operation needs to be triggered, discard the the message.
    * @param message
-   * @param instanceName
    * @param manager
    * @param stateTransitionHandlers
    * @param createHandler
    * @return True if the requested state transition is valid, and need to schedule the transition.
    *         False if no more operation is required.
    */
-  private boolean validateAndProcessStateTransitionMessage(Message message, String instanceName,
-      HelixManager manager, Map<String, MessageHandler> stateTransitionHandlers,
-      MessageHandler createHandler) {
-    HelixDataAccessor accessor = manager.getHelixDataAccessor();
-
+  private boolean validateAndProcessStateTransitionMessage(Message message, HelixManager manager,
+      Map<String, MessageHandler> stateTransitionHandlers, MessageHandler createHandler) {
     String messageTarget = getMessageTarget(message.getResourceName(), message.getPartitionName());
-    if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
-        && isStateTransitionInProgress(messageTarget)) {
-      String taskId = _messageTaskMap.get(messageTarget);
-      Message msg = _taskMap.get(taskId).getTask().getMessage();
-      // If there is another state transition for same partition is going on,
-      // discard the message. Controller will resend if this is a valid message
-      String errMsg = String.format(
-          "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
-          message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
-          msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
-          manager);
-      return false;
-    }
-    if (createHandler instanceof HelixStateTransitionHandler) {
-      // We only check to state if there is no ST task scheduled/executing.
-      HelixStateTransitionHandler.StaleMessageValidateResult result =
-          ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
-      if (!result.isValid) {
-        handleUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
-            accessor, instanceName, manager);
+
+    try {
+      if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
+          && isStateTransitionInProgress(messageTarget)) {
+        String taskId = _messageTaskMap.get(messageTarget);
+        Message msg = _taskMap.get(taskId).getTask().getMessage();
+        // If there is another state transition for same partition is going on,
+        // discard the message. Controller will resend if this is a valid message
+        String errMsg = String.format(
+            "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
+            message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
+            msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);

Review comment:
       Please refer to what I reply to Meng earlier for the same code. The change I made just moves the remove logic a little bit for reusing the code. The messages will still be removed. Please check line 861.
   
   Let me know if you still think something missed here. I agree this is very critical, and I also want to ensure nothing changed here. So please help to double-check for me : )




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514904986



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("Message %s has a negative remaining retry count %d. Stop processing it!",

Review comment:
       Not true for "this message has been retried for "retryCount" times". Since at this place, we don't know how many times this message has been retried. It is not possible to print this information out.
   
   I can add the information that the handler is not available in the error message to make it self-containing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 edited a comment on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 edited a comment on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-722029935


   > > Let us say if the message have a retry count of INT_MAX, this is not going to work. I think we can be a little bit more conservative that if the message have a retry count larger than a threshold, we just retry the threshold value.
   > 
   > @kaisun2000 , if the user explicitly sets the retry to be infinite, we shall not block it, right? Note that the retry won't cause any Helix controller issue, since it is only retried on the participant side. For ZK servers, yeah, it will lead to many ZK writes potentially. But I think that is a problem at a different level, ZK writes throttling for example.
   > In addition, compared with the current behavior, even a retry count == INT_MAX works better. Since every retry in the new logic only generates one write. The older logic will remove the message and creating a new one. So at least 2 write IOs for each retry.
   
   #1487 only makes it by default not logging to ZK. It can still log to ZK with some configuration. I think if you cap the retry count to some fixed value say 100 would be safer. Just my 2 cents. Up to you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514911447



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);

Review comment:
       It is a good point.
   
   There were some discussions earlier yesterday about whether we want this retry count. So the decision is that we want to allow several retries, but would be a very small number. Note that this PR won't really add retry to state transition messages, since the default value of retry count is 0. We will tackle this later in the other PR.
   
   Now regarding if we want to use an in-memory object instead of ZK, it is totally doable. But I don't prefer doing so for 2 reasons.
   1. This ZK write won't make it worse. The retry is not infinite after this change. Compare with the current behavior that we keep retrying infinitely, the new retry logic is much safer. And we can set 0 to the retry count, so no retry at all. It is purely optional.
   2. In-memory count won't work if the participant instance is reset. If the retry count is really large, then in some corner case (like frequent participant reset) the retry will never stop.
   
   In addition, the in-memory requires us to track the messages, it does not exist on the participant side. So definitely will increase the complicity of code. There will be more corner cases that we need to consider. Like memory leakage, etc. Given the benefit of doing so is not critical, I prefer not to choose this option.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514911447



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);

Review comment:
       It is a good point.
   
   There were some discussions earlier yesterday about whether we want this retry count. So the decision is that we want to allow several retries, but would be a very small number. Note that this PR won't really add retry to state transition messages, since the default value of retry count is 0. We will tackle this later in the other PR.
   
   Now regarding if we want to use an in-memory object instead of ZK, it is totally doable. But I don't prefer doing so for 2 reasons.
   1. ZK write won't make it worse. The retry is not infinite after this change. Compare with the current behavior that we keep retrying infinitely, the new retry logic is much safer. And we can set 0 to the retry count, so no retry at all. It is purely optional.
   2. In-memory count won't work if the participant instance is reset. If the retry count is really large, then in some corner case (like frequent participant reset) the retry will never stop.
   
   In addition, the in-memory requires us to track the messages, it does not exist on the participant side. So definitely will increase the complicity of code. There will be more corner cases that we need to consider. Like memory leakage, etc. Given the benefit of doing so is not critical, I prefer not to choose this option.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721949549


   > > if the retry counts is large (say people put -1 instead 0), this is effectively retry forever. How can we make sure this is not going to happen?
   > 
   > @kaisun2000 What do you mean? If it is -1, then the current logic will try once and stop retrying. And according to what Junkai mentioned, I would like to check for the count even before the first try. So that should not be the case, right?
   > 
   
   Let us say if the message have a retry count of INT_MAX, this is not going to work. I think we can be a little bit more conservative that if the message have a retry count larger than a threshold, we just retry the threshold value. 
   
   
   > > Let say later the participant resolve issue and can create the messageHandler for this partition replica. Then does it mean when they recover, they need to manually (or via rest) to remove this UNPROCESSED message? If so, do we have this REST API?
   > 
   > This is the plan. We have discussed this in the previous standup meeting. That task will be addressed separately. Before it is ready, we will require the application to reset the participant.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514903780



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -788,8 +793,8 @@ public void onMessage(String instanceName, List<Message> messages,
     List<MessageHandler> nonStateTransitionHandlers = new ArrayList<>();
     List<NotificationContext> nonStateTransitionContexts = new ArrayList<>();
 
-    // message read
-    List<Message> readMsgs = new ArrayList<>();
+    // message to be updated in ZK
+    List<Message> updatingMsgs = new ArrayList<>();

Review comment:
       readMsgs is no longer an accurate name with this change.
   How about msgsToBeUpdated ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-722029935


   > > Let us say if the message have a retry count of INT_MAX, this is not going to work. I think we can be a little bit more conservative that if the message have a retry count larger than a threshold, we just retry the threshold value.
   > 
   > @kaisun2000 , if the user explicitly sets the retry to be infinite, we shall not block it, right? Note that the retry won't cause any Helix controller issue, since it is only retried on the participant side. For ZK servers, yeah, it will lead to many ZK writes potentially. But I think that is a problem at a different level, ZK writes throttling for example.
   > In addition, compared with the current behavior, even a retry count == INT_MAX works better. Since every retry in the new logic only generates one write. The older logic will remove the message and creating a new one. So at least 2 write IOs for each retry.
   
   #1487 only makes it be default not logging to ZK. It can still log to ZK with some configuration. I think if you cap the retry count to some fixed value say 100 would be safer. Just my 2 cents. Up to you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] narendly commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514901062



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -788,8 +793,8 @@ public void onMessage(String instanceName, List<Message> messages,
     List<MessageHandler> nonStateTransitionHandlers = new ArrayList<>();
     List<NotificationContext> nonStateTransitionContexts = new ArrayList<>();
 
-    // message read
-    List<Message> readMsgs = new ArrayList<>();
+    // message to be updated in ZK
+    List<Message> updatingMsgs = new ArrayList<>();

Review comment:
       this is a little confusing. what's wrong with leaving it as `readMsgs`? `updatingMsgs` is a confusing name since messages don't update themselves. 
   
   or `msgsToBeHandled` or `msgsToHandle` could also be as equally descriptive




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517580514



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {

Review comment:
       Good catch! I guess I tried to simplify the code too much. Let me fix it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514905730



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("Message %s has a negative remaining retry count %d. Stop processing it!",
+                  message.getMsgId(), message.getRetryCount());
+          updateUnprocessableMessage(message, null, errorMsg, manager);
+          updatingMsgs.add(message);
         } else {
-          // Need future process non state transition messages by triggering the handler
-          nonStateTransitionHandlers.add(msgHandler);
-          nonStateTransitionContexts.add(msgWorkingContext);
+          // Skip processing this message in this callback. The same message process will be retried
+          // in the next round.
+          LOG.warn("There is no existing handler for message {}."
+                  + " Skip processing it for now. Will retry on the next callback.",

Review comment:
       When there is no exception, but the createHandler returns null, we don't have any other output except for this line. So I think it is necessary.
   If we see many of this line, then the application logic in participants might have some serious issue, and we shall fix it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] narendly commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r514901374



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,66 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {
+          // If no more retry count remains, then mark the message to be UNPROCESSABLE.
+          String errorMsg = String
+              .format("Message %s has a negative remaining retry count %d. Stop processing it!",
+                  message.getMsgId(), message.getRetryCount());
+          updateUnprocessableMessage(message, null, errorMsg, manager);
+          updatingMsgs.add(message);
         } else {
-          // Need future process non state transition messages by triggering the handler
-          nonStateTransitionHandlers.add(msgHandler);
-          nonStateTransitionContexts.add(msgWorkingContext);
+          // Skip processing this message in this callback. The same message process will be retried
+          // in the next round.
+          LOG.warn("There is no existing handler for message {}."
+                  + " Skip processing it for now. Will retry on the next callback.",

Review comment:
       is this log absolutely necessary? wouldn't it be sufficient to log when we've used up all the retry counts? i'm afraid this might just end up polluting the log




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r516992216



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -613,7 +613,12 @@ void reset() {
 
       MsgHandlerFactoryRegistryItem item = _hdlrFtyRegistry.get(msgType);
       if (item.factory() != null) {
-        item.factory().reset();
+        try {
+          item.factory().reset();
+        } catch (Exception ex) {

Review comment:
       This is only called when we clean up. All the paths lead to the termination of the executor. The only result of not swallowing the exception is that the later cleanup steps will not happen.
   
   Backtrace is a good point. I think I missed it unintentionally. Thanks for pointing this out.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 commented on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 commented on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721923998


   >The UNPROCESSABLE message, which is generated due to the previous point, will be left in the participant message folder and not automatically removed. This is to prevent unnecessary retry.
   
   Let say later the participant resolve issue and can create the messageHandler for this partition replica. Then does it mean when they recover, they need to manually (or via rest) to remove this UNPROCESSED message?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r516994162



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -993,67 +1034,78 @@ private boolean checkAndProcessNoOpMessage(Message message, String instanceName,
    * Preprocess the state transition message to validate if the request is valid.
    * If no operation needs to be triggered, discard the the message.
    * @param message
-   * @param instanceName
    * @param manager
    * @param stateTransitionHandlers
    * @param createHandler
    * @return True if the requested state transition is valid, and need to schedule the transition.
    *         False if no more operation is required.
    */
-  private boolean validateAndProcessStateTransitionMessage(Message message, String instanceName,
-      HelixManager manager, Map<String, MessageHandler> stateTransitionHandlers,
-      MessageHandler createHandler) {
-    HelixDataAccessor accessor = manager.getHelixDataAccessor();
-
+  private boolean validateAndProcessStateTransitionMessage(Message message, HelixManager manager,
+      Map<String, MessageHandler> stateTransitionHandlers, MessageHandler createHandler) {
     String messageTarget = getMessageTarget(message.getResourceName(), message.getPartitionName());
-    if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
-        && isStateTransitionInProgress(messageTarget)) {
-      String taskId = _messageTaskMap.get(messageTarget);
-      Message msg = _taskMap.get(taskId).getTask().getMessage();
-      // If there is another state transition for same partition is going on,
-      // discard the message. Controller will resend if this is a valid message
-      String errMsg = String.format(
-          "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
-          message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
-          msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
-          manager);
-      return false;
-    }
-    if (createHandler instanceof HelixStateTransitionHandler) {
-      // We only check to state if there is no ST task scheduled/executing.
-      HelixStateTransitionHandler.StaleMessageValidateResult result =
-          ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
-      if (!result.isValid) {
-        handleUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
-            accessor, instanceName, manager);
+
+    try {
+      if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
+          && isStateTransitionInProgress(messageTarget)) {
+        String taskId = _messageTaskMap.get(messageTarget);
+        Message msg = _taskMap.get(taskId).getTask().getMessage();
+        // If there is another state transition for same partition is going on,
+        // discard the message. Controller will resend if this is a valid message
+        String errMsg = String.format(
+            "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
+            message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
+            msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);
         return false;
       }
-    }
-    if (stateTransitionHandlers.containsKey(messageTarget)) {
-      // If there are 2 messages in same batch about same partition's state transition,
-      // the later one is discarded
-      Message duplicatedMessage = stateTransitionHandlers.get(messageTarget)._message;
-      String errMsg = String.format(
-          "Duplicated state transition message: %s. Existing: %s->%s; New (Discarded): %s->%s",
-          message.getMsgId(), duplicatedMessage.getFromState(), duplicatedMessage.getToState(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
+      if (createHandler instanceof HelixStateTransitionHandler) {
+        // We only check to state if there is no ST task scheduled/executing.
+        HelixStateTransitionHandler.StaleMessageValidateResult result =
+            ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
+        if (!result.isValid) {
+          updateUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
+              manager);
+          return false;
+        }
+      }
+      if (stateTransitionHandlers.containsKey(messageTarget)) {
+        // If there are 2 messages in same batch about same partition's state transition,
+        // the later one is discarded
+        Message duplicatedMessage = stateTransitionHandlers.get(messageTarget)._message;
+        String errMsg = String.format(
+            "Duplicated state transition message: %s. Existing: %s->%s; New (Discarded): %s->%s",
+            message.getMsgId(), duplicatedMessage.getFromState(), duplicatedMessage.getToState(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);

Review comment:
       I moved the removing message logic to the caller. So for this specific case, it will keep the original behavior.
   
   This change is for reusing the updateUnprocessableMessage in the other conditions. In addition, I also want to avoid silently removing a message in a very deep private call. So the new logic puts all the message processing (update or remove) in a smaller scope.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] kaisun2000 edited a comment on pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
kaisun2000 edited a comment on pull request #1500:
URL: https://github.com/apache/helix/pull/1500#issuecomment-721923998


   >The UNPROCESSABLE message, which is generated due to the previous point, will be left in the participant message folder and not automatically removed. This is to prevent unnecessary retry.
   
   Let say later the participant resolve issue and can create the messageHandler for this partition replica. Then does it mean when they recover, they need to manually (or via rest) to remove this UNPROCESSED message? If so, do we have this REST API?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517873215



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {

Review comment:
       Sure, TODO might be overkill since I don't think we have a concrete plan yet. But let me comment more on this logic.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] dasahcc commented on a change in pull request #1500: Refine the message handler error handling logic to avoid unnecessary retry.

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1500:
URL: https://github.com/apache/helix/pull/1500#discussion_r517562369



##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -804,40 +809,67 @@ public void onMessage(String instanceName, List<Message> messages,
         // skip the following operations for the no-op messages.
         continue;
       }
-      // create message handlers, if handlers not found, leave its state as NEW
+
       NotificationContext msgWorkingContext = changeContext.clone();
+      MessageHandler msgHandler = null;
       try {
-        MessageHandler msgHandler = createMessageHandler(message, msgWorkingContext);
-        if (msgHandler == null) {
-          // Failed to create message handler, skip processing this message in this callback.
-          // The same message process will be retried in the next round.
-          continue;
-        }
-        if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name()) || message.getMsgType()
-            .equals(MessageType.STATE_TRANSITION_CANCELLATION.name())) {
-          if (validateAndProcessStateTransitionMessage(message, instanceName, manager,
-              stateTransitionHandlers, msgHandler)) {
-            // Need future process by triggering state transition
-            String msgTarget =
-                getMessageTarget(message.getResourceName(), message.getPartitionName());
-            stateTransitionHandlers.put(msgTarget, msgHandler);
-            stateTransitionContexts.put(msgTarget, msgWorkingContext);
-          } else {
-            // skip the following operations for the invalid/expired state transition messages.
-            continue;
-          }
+        // create message handlers, if handlers not found but no exception, leave its state as NEW
+        msgHandler = createMessageHandler(message, msgWorkingContext);
+      } catch (Exception ex) {
+        // Failed to create message handler and there is an Exception.
+        int remainingRetryCount = message.getRetryCount();
+        LOG.error(
+            "Exception happens when creating Message Handler for message {}. Current remaining retry count is {}.",
+            message.getMsgId(), remainingRetryCount);
+        // Set the message retry count to avoid infinite retrying.
+        message.setRetryCount(remainingRetryCount - 1);
+        message.setExecuteSessionId(sessionId);
+        // continue processing in the next section where handler object is double-checked.
+      }
+
+      if (msgHandler == null) {
+        if (message.getRetryCount() < 0) {

Review comment:
       NIT: From the retry count definition, we should let retry count check happens before you try another round of creation. If retry count is less than 0, it means it has one more retry than what we defined for the retry count. I know this could be a minor issue. But logically, it breaks the counting rule. Better to reorganize it.

##########
File path: helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTaskExecutor.java
##########
@@ -993,67 +1034,78 @@ private boolean checkAndProcessNoOpMessage(Message message, String instanceName,
    * Preprocess the state transition message to validate if the request is valid.
    * If no operation needs to be triggered, discard the the message.
    * @param message
-   * @param instanceName
    * @param manager
    * @param stateTransitionHandlers
    * @param createHandler
    * @return True if the requested state transition is valid, and need to schedule the transition.
    *         False if no more operation is required.
    */
-  private boolean validateAndProcessStateTransitionMessage(Message message, String instanceName,
-      HelixManager manager, Map<String, MessageHandler> stateTransitionHandlers,
-      MessageHandler createHandler) {
-    HelixDataAccessor accessor = manager.getHelixDataAccessor();
-
+  private boolean validateAndProcessStateTransitionMessage(Message message, HelixManager manager,
+      Map<String, MessageHandler> stateTransitionHandlers, MessageHandler createHandler) {
     String messageTarget = getMessageTarget(message.getResourceName(), message.getPartitionName());
-    if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
-        && isStateTransitionInProgress(messageTarget)) {
-      String taskId = _messageTaskMap.get(messageTarget);
-      Message msg = _taskMap.get(taskId).getTask().getMessage();
-      // If there is another state transition for same partition is going on,
-      // discard the message. Controller will resend if this is a valid message
-      String errMsg = String.format(
-          "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
-          message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
-          msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
-          message.getFromState(), message.getToState());
-      handleUnprocessableMessage(message, null /* exception */, errMsg, accessor, instanceName,
-          manager);
-      return false;
-    }
-    if (createHandler instanceof HelixStateTransitionHandler) {
-      // We only check to state if there is no ST task scheduled/executing.
-      HelixStateTransitionHandler.StaleMessageValidateResult result =
-          ((HelixStateTransitionHandler) createHandler).staleMessageValidator();
-      if (!result.isValid) {
-        handleUnprocessableMessage(message, null /* exception */, result.exception.getMessage(),
-            accessor, instanceName, manager);
+
+    try {
+      if (message.getMsgType().equals(MessageType.STATE_TRANSITION.name())
+          && isStateTransitionInProgress(messageTarget)) {
+        String taskId = _messageTaskMap.get(messageTarget);
+        Message msg = _taskMap.get(taskId).getTask().getMessage();
+        // If there is another state transition for same partition is going on,
+        // discard the message. Controller will resend if this is a valid message
+        String errMsg = String.format(
+            "Another state transition for %s:%s is in progress with msg: %s, p2p: %s, read: %d, current:%d. Discarding %s->%s message",
+            message.getResourceName(), message.getPartitionName(), msg.getMsgId(),
+            msg.isRelayMessage(), msg.getReadTimeStamp(), System.currentTimeMillis(),
+            message.getFromState(), message.getToState());
+        updateUnprocessableMessage(message, null /* exception */, errMsg, manager);

Review comment:
       This changes the behavior. If the message is not exact state we are targeting, it should be a intermediate state, we should throw away this message instead of let it hanging there. Otherwise, we will face a lot of human involved operations.
   
   For example, for the p2p arrived but timedout, the message will be discarded. But if you hang it as unprocessable, the partition will be stuck.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org