You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/05/29 21:42:19 UTC
[GitHub] [helix] alirezazamani opened a new pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
alirezazamani opened a new pull request #1040:
URL: https://github.com/apache/helix/pull/1040
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR title:
Fixes #1039
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
The processTaskWithPendingMessage method is relying on the previousAssignment.
In this PR, this method has been modified and previousAssignment has been
replaced with currentState.
### Tests
- [x] The following tests are written for this issue:
TestUpdatePreviousAssignedTaskStatusWithPendingMessage
- [x] The following is the result of the "mvn test" command on the appropriate module:
Helix-Core:
```
[ERROR] Tests run: 1146, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5,255.063 s <<< FAILURE! - in TestSuite
[ERROR] testJobQueueAutoCleanUp(org.apache.helix.integration.task.TestJobQueueCleanUp) Time elapsed: 300.012 s <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.testng.internal.TestNGMethod.testJobQueueAutoCleanUp() didn't finish within the time-out 300000
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestJobQueueCleanUp.testJobQueueAutoCleanUp ยป ThreadTimeout Method org.testng....
[INFO]
[ERROR] Tests run: 1146, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:27 h
[INFO] Finished at: 2020-05-29T14:14:16-07:00
[INFO] ------------------------------------------------------------------------
```
The failed tests have passed when I ran them individually.
mvn test -Dtest="TestJobQueueCleanUp"
```
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.521 s - in org.apache.helix.integration.task.TestJobQueueCleanUp
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.301 s
[INFO] Finished at: 2020-05-29T14:38:54-07:00
[INFO] ------------------------------------------------------------------------
```
Helix-rest:
```
[INFO] Tests run: 159, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.274 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 159, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46.810 s
[INFO] Finished at: 2020-05-29T14:40:07-07:00
[INFO] ------------------------------------------------------------------------
```
### Commits
- [x] My commits all reference appropriate Apache Helix GitHub issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- [x] My diff has been formatted using helix-style.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r433004938
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
+ // While job is timing out, if the task is pending on INIT->RUNNING, set it back to INIT,
+ // so that Helix will cancel the transition.
+ paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.INIT.name()));
Review comment:
I'm just brainstorming here:
I think a better state to assign here might be DROPPED? Previously, we would send things to INIT but that would be a long-winded way to cause this to be dropped. And I'm not sure with the recent changes, the tasks that are timing out that are in INIT state will indeed be dropped eventually.
If they don't, then there's a problem - init state takes up a quota in the quota calculation, and we might end up seeing quota leaks.
Have you tried to drop here directly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] dasahcc merged pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
dasahcc merged pull request #1040:
URL: https://github.com/apache/helix/pull/1040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r433463896
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
+ // While job is timing out, if the task is pending on INIT->RUNNING, set it back to INIT,
+ // so that Helix will cancel the transition.
+ paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.INIT.name()));
Review comment:
@narendly Thank you for the suggestion. I think what you are suggesting would also work. However, I think the logic behind this part of code should be untouched as much as possible. If we want to do send DROPPED here (although I think it would work), we need to release the quota and do further logic changes. The reason is once we DROP the the task from instance, we will miss currentState and we will not see that task in other methods in order to release quota. Hence, I prefer to send DROPPED for the task using two level approach (i.e. first send it to INIT and then DROP it).
Also since there is a pending INIT->RUNNING message existed, it is not very safe to release the quota here. Because the task might go to RUNNING state and before we cancel it, the quota information will be mismatched with the actual scenario. So if we first let the task to go to INIT state and then releasing the quota and sending it to DROPPED will be better option in my opinion.
To answer your question, we will actually handle this case for releasing the quota in updatePreviousAssignedTasksStatus method. Here the code that would do that:
```
case INIT: {
// INIT is a temporary state for tasks
// Two possible scenarios for INIT:
// 1. Task is getting scheduled for the first time. In this case, Task's state will go
// from null->INIT->RUNNING, and this INIT state will be transient and very short-lived
// 2. Task is getting scheduled for the first time, but in this case, job is timed out or
// timing out. In this case, it will be sent back to INIT state to be removed. Here we
// ensure that this task then goes from INIT to DROPPED so that it will be released from
// AssignableInstance to prevent resource leak
if (jobState == TaskState.TIMED_OUT || jobState == TaskState.TIMING_OUT
|| jobTgtState == TargetState.DELETE) {
// Job is timed out or timing out or targetState is to be deleted, so its tasks will be
// sent back to INIT
// In this case, tasks' IdealState will be removed, and they will be sent to DROPPED
partitionsToDropFromIs.add(pId);
assignedPartitions.get(instance).add(pId);
paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.DROPPED.name()));
// Also release resources for these tasks
assignableInstanceManager.release(instance, taskConfig, quotaType);
break;
}
```
Your comment actually guided me to add another safety check for INIT case. I added two lines to make sure we DROP the task if job is TIMED_OUT or TIMING_OUT and the task is in INIT state. Thanks for the comments and careful review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r433463896
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
+ // While job is timing out, if the task is pending on INIT->RUNNING, set it back to INIT,
+ // so that Helix will cancel the transition.
+ paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.INIT.name()));
Review comment:
@narendly Thank you for the suggestion. I think what you are suggesting would also work. However, I think the logic behind this part of code should be untouched as much as possible. If we want to do send DROPPED here (although I think it would work), we need to release the quota and do further logic changes. The reason is once we DROP the the task from instance, we will miss currentState and we will not see that task in other methods in order to release quota. Hence, I prefer to send DROPPED for the task using two level approach (i.e. first send it to INIT and then DROP it).
Also since there is a pending INIT->RUNNING message existed, it is not very safe to release the quota here. Because the task might go to RUNNING state and before we cancel it, the quota information will be mismatched with the actual scenario. So if we first let the task to go to INIT state and then releasing the quota and sending it to DROPPED will be better option in my opinion.
To answer your question, we will actually handle this case for releasing the quota in updatePreviousAssignedTasksStatus method. Here the code that would do that:
```
case INIT: {
// INIT is a temporary state for tasks
// Two possible scenarios for INIT:
// 1. Task is getting scheduled for the first time. In this case, Task's state will go
// from null->INIT->RUNNING, and this INIT state will be transient and very short-lived
// 2. Task is getting scheduled for the first time, but in this case, job is timed out or
// timing out. In this case, it will be sent back to INIT state to be removed. Here we
// ensure that this task then goes from INIT to DROPPED so that it will be released from
// AssignableInstance to prevent resource leak
if (jobState == TaskState.TIMED_OUT || jobState == TaskState.TIMING_OUT
|| jobTgtState == TargetState.DELETE) {
// Job is timed out or timing out or targetState is to be deleted, so its tasks will be
// sent back to INIT
// In this case, tasks' IdealState will be removed, and they will be sent to DROPPED
partitionsToDropFromIs.add(pId);
assignedPartitions.get(instance).add(pId);
paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.DROPPED.name()));
// Also release resources for these tasks
**assignableInstanceManager.release(instance, taskConfig, quotaType);**
break;
}
```
Your comment actually guided me to add another safety check that we can added for INIT case. I added two lines to make sure we DROP the task if job is TIMED_OUT or TIMING_OUT and the task is in INIT state. Thanks for the comments and careful review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432771509
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
This part of the code is to issue cause ST cancellation. Right? If there pending message to state is INIT then the if statement will not be satisfied and we will not cancel the cancel message.
Also, this part is basically similar to the previous version of the code.
From the comments and the code: If there is a pending message for this task and this pending message is INIT->RUNNING, if the job goes to TIMING_OUT state, we need to cancel this state transition. Hence we would set the task back to INIT state in this case. Then if Helix can cancel the ST (if enabled) this ST will be cancelled by controller.
Also without this change we are in the same situation: We decided to send INIT->RUNNING message for this task (hence in previous pipeline we set previousAssignment to RUNNING and send INIT-> RUNNING message), in next pipelines while the message has not been processed yet, job goes to TIMING_OUT state, hence this message is not needed we can cancel it (if enabled). In this case we set the paMap back to INIT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432771509
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
This part of the code is to issue ST cancellation message. Right? If there is a pending message which its toState is INIT then this if statement will not be satisfied and we will not cancel the cancel message.
Also, this part is basically similar to the previous version of the code (So I don't think there is a logic change in this PR)
From the comments and the code: If there is a pending message for this task and this pending message is INIT->RUNNING, if the job goes to TIMING_OUT state, we need to cancel this state transition. Hence we would set the task back to INIT state in this case. Then if Helix can cancel the ST (if enabled) this ST will be cancelled by controller.
Also without this change we are in the same situation: We decided to send INIT->RUNNING message for this task (hence in previous pipeline we set previousAssignment to RUNNING and send INIT-> RUNNING message), in next pipelines while the message has not been processed yet, job goes to TIMING_OUT state, hence this message is not needed we can cancel it (if enabled). In this case we set the paMap back to INIT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] dasahcc commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432767224
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
Could this cause problem? If there is a cancel message, so the toState is not RUNNING, it should INIT? Then are we cancelling the CANCEL message?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432771509
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
This part of the code is to issue ST cancellation message. Right? If there is a pending message which its toState is INIT then the if statement will not be satisfied and we will not cancel the cancel message.
Also, this part is basically similar to the previous version of the code.
From the comments and the code: If there is a pending message for this task and this pending message is INIT->RUNNING, if the job goes to TIMING_OUT state, we need to cancel this state transition. Hence we would set the task back to INIT state in this case. Then if Helix can cancel the ST (if enabled) this ST will be cancelled by controller.
Also without this change we are in the same situation: We decided to send INIT->RUNNING message for this task (hence in previous pipeline we set previousAssignment to RUNNING and send INIT-> RUNNING message), in next pipelines while the message has not been processed yet, job goes to TIMING_OUT state, hence this message is not needed we can cancel it (if enabled). In this case we set the paMap back to INIT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r433004938
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
+ // While job is timing out, if the task is pending on INIT->RUNNING, set it back to INIT,
+ // so that Helix will cancel the transition.
+ paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.INIT.name()));
Review comment:
I think a better state to assign here might be DROPPED? Previously, we would send things to INIT but that would be a long-winded way to cause this to be dropped. And I'm not sure with the recent changes, the tasks that are timing out that are in INIT state will indeed be dropped eventually.
If they don't, then there's a problem - init state takes up a quota in the quota calculation, and we might end up seeing quota leaks.
Have you tried to drop here directly? We can try to run some tests to see what the consequences are for dropping directly here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on pull request #1040:
URL: https://github.com/apache/helix/pull/1040#issuecomment-638962527
This PR is ready to be merged, approved by @dasahcc
Final commit message:
Remove previousAssignment in processTaskWithPendingMessage method
The processTaskWithPendingMessage method is relying on the previousAssignment.
In this commit, this method has been modified and previousAssignment has been
replaced with currentState.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r433463896
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
+ // While job is timing out, if the task is pending on INIT->RUNNING, set it back to INIT,
+ // so that Helix will cancel the transition.
+ paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.INIT.name()));
Review comment:
@narendly Thank you for the suggestion. I think what you are suggesting would also work. However, I think the logic behind this part of code should be untouched as much as possible. If we want to do send DROPPED here (although I think it would work), we need to release the quota and do further logic changes. The reason is once we DROP the the task from instance, we will miss currentState and we will not see that task in other methods in order to release quota. Hence, I prefer to send DROPPED for the task using two level approach (i.e. first send it to INIT and then DROP it).
Also since there is a pending INIT->RUNNING message existed, it is not very safe to release the quota here. Because the task might go to RUNNING state and before we cancel it, the quota information will be mismatched with the actual scenario. So if we first let the task to go to INIT state and then releasing the quota and sending it to DROPPED will be better option in my opinion.
To answer your question, we will actually handle this case for releasing the quota in updatePreviousAssignedTasksStatus method. Here the code that would do that:
```
case INIT: {
// INIT is a temporary state for tasks
// Two possible scenarios for INIT:
// 1. Task is getting scheduled for the first time. In this case, Task's state will go
// from null->INIT->RUNNING, and this INIT state will be transient and very short-lived
// 2. Task is getting scheduled for the first time, but in this case, job is timed out or
// timing out. In this case, it will be sent back to INIT state to be removed. Here we
// ensure that this task then goes from INIT to DROPPED so that it will be released from
// AssignableInstance to prevent resource leak
if (jobState == TaskState.TIMED_OUT || jobState == TaskState.TIMING_OUT
|| jobTgtState == TargetState.DELETE) {
// Job is timed out or timing out or targetState is to be deleted, so its tasks will be
// sent back to INIT
// In this case, tasks' IdealState will be removed, and they will be sent to DROPPED
partitionsToDropFromIs.add(pId);
assignedPartitions.get(instance).add(pId);
paMap.put(pId, new PartitionAssignment(instance, TaskPartitionState.DROPPED.name()));
// Also release resources for these tasks
assignableInstanceManager.release(instance, taskConfig, quotaType);
break;
}
```
Your comment actually guided me to add another safety check that we can added for INIT case. I added two lines to make sure we DROP the task if job is TIMED_OUT or TIMING_OUT and the task is in INIT state. Thanks for the comments and careful review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432771509
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
This part of the code is to issue ST cancellation message. Right? If there is a pending message which its toState is INIT then this if statement will not be satisfied and we will not cancel the cancel message.
Also, this part is basically similar to the previous version of the code.
From the comments and the code: If there is a pending message for this task and this pending message is INIT->RUNNING, if the job goes to TIMING_OUT state, we need to cancel this state transition. Hence we would set the task back to INIT state in this case. Then if Helix can cancel the ST (if enabled) this ST will be cancelled by controller.
Also without this change we are in the same situation: We decided to send INIT->RUNNING message for this task (hence in previous pipeline we set previousAssignment to RUNNING and send INIT-> RUNNING message), in next pipelines while the message has not been processed yet, job goes to TIMING_OUT state, hence this message is not needed we can cancel it (if enabled). In this case we set the paMap back to INIT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] alirezazamani commented on a change in pull request #1040: Remove previousAssignment in processTaskWithPendingMessage method
Posted by GitBox <gi...@apache.org>.
alirezazamani commented on a change in pull request #1040:
URL: https://github.com/apache/helix/pull/1040#discussion_r432771509
##########
File path: helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -449,43 +448,31 @@ private void updatePartitionInformationInJobContext(CurrentStateOutput currentSt
* @param paMap
* @param assignedPartitions
*/
- private void processTaskWithPendingMessage(ResourceAssignment prevAssignment, Integer pId,
- String pName, String instance, Message pendingMessage, TaskState jobState,
- TaskPartitionState currState, Map<Integer, PartitionAssignment> paMap,
- Map<String, Set<Integer>> assignedPartitions) {
-
- // stateMap is a mapping of Instance -> TaskPartitionState (String)
- Map<String, String> stateMap = prevAssignment.getReplicaMap(new Partition(pName));
- if (stateMap != null) {
- String prevState = stateMap.get(instance);
- if (!pendingMessage.getToState().equals(prevState)) {
- LOG.warn(String.format(
- "Task pending to-state is %s while previous assigned state is %s. This should not"
- + "happen.",
- pendingMessage.getToState(), prevState));
+ private void processTaskWithPendingMessage(Integer pId, String pName, String instance,
+ Message pendingMessage, TaskState jobState, TaskPartitionState currState,
+ Map<Integer, PartitionAssignment> paMap, Map<String, Set<Integer>> assignedPartitions) {
+
+ if (jobState == TaskState.TIMING_OUT && currState == TaskPartitionState.INIT
+ && pendingMessage.getToState().equals(TaskPartitionState.RUNNING.name())) {
Review comment:
This part is basically similar to the previous version of the code.
From the comments and the code: If there is a pending message for this task and this pending message is INIT->RUNNING, if the job goes to TIMING_OUT state, we need to cancel this state transition. Hence we would set the task back to INIT state in this case. Then if Helix can cancel the ST (if enabled) this ST will be cancelled by controller.
Also without this change we are in the same situation: We decided to send INIT->RUNNING message for this task (hence in previous pipeline we set previousAssignment to RUNNING and send INIT-> RUNNING message), in next pipelines while the message has not been processed yet, job goes to TIMING_OUT state, hence this message is not needed we can cancel it (if enabled). In this case we set the paMap back to INIT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org