You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/11/22 07:36:26 UTC

[GitHub] [druid] AmatyaAvadhanula opened a new pull request, #13411: Fix needless task shutdown on leader switch

AmatyaAvadhanula opened a new pull request, #13411:
URL: https://github.com/apache/druid/pull/13411

   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
   
   
   <!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
   
   <!-- If you are a committer, follow the PR action item checklist for committers:
   https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
   
   ### Description
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   The first run of a supervisor tries to resume all actively reading tasks and shuts down any tasks which fail to resume successfully in #13223.
   
   However it included pending / waiting tasks in the set of active tasks and killed them as they failed to resume.
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   
   The fix is to check if the task is also running using the TaskRunner before attempting to resume and fail tasks.
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   #### Release note
   <!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users. 
   
   If your change doesn't have end user impact, you can skip this section.
   
   -->
   
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `SeekableStreamSupervisor`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   
   - [x] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [x] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029036721


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.

Review Comment:
   ```
             leaderLifecycle.addManagedInstance(taskRunner);
             leaderLifecycle.addManagedInstance(taskQueue);
             leaderLifecycle.addManagedInstance(supervisorManager);
             leaderLifecycle.addManagedInstance(overlordHelperManager);
   ```
   
   Handlers are added while trying to become the leader and are also processed in this order.
   TaskRunner#start seems to sync its state for both RTR and HRTR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #13411:
URL: https://github.com/apache/druid/pull/13411#issuecomment-1332715805

   The CI failures are a flaky test and code coverage checks. @AmatyaAvadhanula is this patch unit-testable? If not feasible to do a UT fo rit then I think it's OK to bypass the coverage checks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1031463430


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Makes sense.
   
   Have we seen a specific drawback of killing off PENDING/WAITING tasks or a race condition that may have adverse effects, or is this PR just a safety measure to avoid unnecessary operations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #13411:
URL: https://github.com/apache/druid/pull/13411#issuecomment-1333082784

   > However KafkaSupervisorTest / KinesisSupervisorTest don't help with coverage of SeekableStreamSupervisor
   
   OK, in that case I suggest we should bypass the coverage check, since it's getting coverage through test cases that are "invisible" to the coverage checker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on PR #13411:
URL: https://github.com/apache/druid/pull/13411#issuecomment-1333068060

   @gianm @kfaraz thanks for the review.
   I've modified the existing UT to test the case where the task hasn't begun running. 
   However KafkaSupervisorTest / KinesisSupervisorTest don't help with coverage of SeekableStreamSupervisor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1031469357


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Killing pending tasks of a supervisor could lead to increased lag as it must re-create these tasks in the next run.
   The failure could also be misleading as one cannot resume a task that hasn't begun running



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029068426


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Yes. Paused is an unrelated internal task state and the task runner considers such tasks to be running as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029024840


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.

Review Comment:
   Are we guaranteed at this point that the task runner has initialized its state? (For RTR, has it synced with ZK yet? For HRTR, has it heard from each worker yet?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula merged pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula merged PR #13411:
URL: https://github.com/apache/druid/pull/13411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029080002


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Just catching up here, isn't the issue that we kill off RUNNING tasks which are actively processing data/publishing segments? Or were we killing off WAITING tasks?
   
   IIUC, the existing flow wouldn't affect PENDING or WAITING tasks anyway, as they haven't started execution yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029080002


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Just catching up here, isn't the issue that we kill off RUNNING tasks which are actively processing data/publishing segments? Or were we killing off WAITING tasks?
   
   IIUC, the existing flow wouldn't affect PENDING or WAITING tasks anyway, as they haven't started execution yet (unless they just started execution and we don't have the synced state yet).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029116418


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   The code introduced in #13223 tried to resume any task in the set of `activelyReadingTaskGroups` irrespective of the task runner worker status and killed the task if it failed to respond with a successful status.
   
   The issue was that there were PENDING / WAITING tasks in the set which couldn't respond to this request as they hadn't even begun RUNNING, and were killed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13411: Fix needless task shutdown on leader switch

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13411:
URL: https://github.com/apache/druid/pull/13411#discussion_r1029062580


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -2014,9 +2014,24 @@ private void resumeAllActivelyReadingTasks()
     }
 
     Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+    if (activelyReadingTaskGroups.isEmpty()) {
+      return;
+    }
+    // Resume only running tasks and not pending / waiting ones.
+    if (!taskMaster.getTaskRunner().isPresent()) {
+      return;
+    }
+    Set<String> runningTaskIds = taskMaster.getTaskRunner()
+                                           .get()
+                                           .getRunningTasks()

Review Comment:
   Does this also return paused tasks? According to the javadoc of this method, this method is meant to resume paused tasks (and try to resume running tasks, which should be a no-op).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org