You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/10/05 20:02:58 UTC

[GitHub] [druid] AmatyaAvadhanula opened a new pull request, #13172: Fix Overlord leader election when task lock re-acquisition fails

AmatyaAvadhanula opened a new pull request, #13172:
URL: https://github.com/apache/druid/pull/13172

   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
   
   Fixes Overlord leader election when task lock re-acquisition fails
   
   <!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
   
   <!-- If you are a committer, follow the PR action item checklist for committers:
   https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
   
   ### Description
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   https://github.com/apache/druid/issues/11653 describes an issue where Overlord leader election fails due lock re-acquisition issues
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   This PR aims to solve the issue by failing such tasks and clearing all their locks when re-acquisition doesn't succeed, so that the Overlord leader election is not blocked
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `TaskLockbox`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   - [x] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996627510


##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");

Review Comment:
   Done



##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");
+    final Task badTask1 = NoopTask.withGroupId("BadTask");
+    final Task goodTask0 = NoopTask.withGroupId("GoodTask");
+    taskStorage.insert(badTask0, TaskStatus.running(badTask0.getId()));
+    taskStorage.insert(badTask1, TaskStatus.running(badTask1.getId()));
+    taskStorage.insert(goodTask0, TaskStatus.running(goodTask0.getId()));
+
+    TaskLockbox testLockbox = new NullLockPosseTaskLockbox(taskStorage, metadataStorageCoordinator);
+    testLockbox.add(badTask0);
+    testLockbox.add(badTask1);
+    testLockbox.add(goodTask0);
+
+    testLockbox.tryLock(badTask0, new TimeChunkLockRequest(TaskLockType.EXCLUSIVE,

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996438613


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -173,6 +174,12 @@ public void start()
       Preconditions.checkState(!active, "queue must be stopped");
       active = true;
       syncFromStorage();
+      // Mark these tasks as failed as they could not reacuire the lock
+      // Clean up needs to happen after tasks have been synced from storage
+      Set<Task> tasksToFail = taskLockbox.syncFromStorage().getTasksToFail();
+      for (Task task : tasksToFail) {
+        shutdown(task.getId(), "Failed to reacquire lock.");

Review Comment:
   Sure, that works too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz merged pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz merged PR #13172:
URL: https://github.com/apache/druid/pull/13172


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995377690


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -770,6 +776,17 @@ private void syncFromStorage()
           addTaskInternal(task);
         }
 
+        for (Task task : tasksToFail) {

Review Comment:
   Thanks. I've moved this



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +184,33 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s.",

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995375065


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskMaster.java:
##########
@@ -113,7 +113,7 @@ public void becomeLeader()
         log.info("By the power of Grayskull, I have the power!");
 
         try {
-          taskLockbox.syncFromStorage();
+          SyncResult syncResult = taskLockbox.syncFromStorage();

Review Comment:
   Thanks! I have moved both TaskLockbox's sync and the processing to TaskQueue's start



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r985719442


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -197,17 +205,26 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
           activeTasks.size(),
           storedLocks.size() - taskLockCount
       );
+
+      for (Task task : failedToReacquireLockTasks) {
+        for (TaskLock lock : taskStorage.getLocks(task.getId())) {
+          taskStorage.removeLock(task.getId(), lock);
+        }
+        taskStorage.setStatus(TaskStatus.failure(task.getId(), "Failed to reacquire lock"));

Review Comment:
   This doesn't seem the right thing to do in `TaskLockbox`. If you notice, the `taskMaster` is used in this class to manipulate the locks and that's it. Lockbox doesn't modify any state other than the locks. We could maybe return the list of tasks for which we could not acquire locks to the caller. As you do this, it will be better to return a `Result` object where we can add more info later if needed. what do you think? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995377907


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -858,6 +873,26 @@ public void unlockAll(Task task)
     }
   }
 
+  public void unlockUnacquiredLocks(Task task)

Review Comment:
   Ah sorry. Removed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996249293


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -228,6 +235,13 @@ public ScheduledExecutors.Signal call()
           }
       );
       requestManagement();
+      // Remove any unacquired locks from storage

Review Comment:
   shutdown() only clears tasklocks for which a TaskLockPosse is present. The error here is that a lock could not be reacquired and a TaskLockPosse doesn't exist for the conflicting task, which is why these "unacquired" entries must be removed in this manner



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on PR #13172:
URL: https://github.com/apache/druid/pull/13172#issuecomment-1265515562

   @abhishekagarwal87 thank you for the review!
   I think there were a few other gaps such as cleaning up the task's existing TaskLockPosse and shutting it down if it's running, which I've fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995401178


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -108,17 +108,20 @@ public TaskLockbox(
 
   /**
    * Wipe out our current in-memory state and resync it from our bundled {@link TaskStorage}.
+   *
+   * @return SyncResult which needs to be processed by the caller
    */
-  public void syncFromStorage()
+  public SyncResult syncFromStorage()
   {
     giant.lock();
 
     try {
       // Load stuff from taskStorage first. If this fails, we don't want to lose all our locks.
-      final Set<String> storedActiveTasks = new HashSet<>();

Review Comment:
   Thanks for pointing this out. I've reverted the change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995311317


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -108,17 +108,20 @@ public TaskLockbox(
 
   /**
    * Wipe out our current in-memory state and resync it from our bundled {@link TaskStorage}.
+   *
+   * @return SyncResult which needs to be processed by the caller
    */
-  public void syncFromStorage()
+  public SyncResult syncFromStorage()
   {
     giant.lock();
 
     try {
       // Load stuff from taskStorage first. If this fails, we don't want to lose all our locks.
-      final Set<String> storedActiveTasks = new HashSet<>();

Review Comment:
   The previous approach is the preferred one, as it seems from the preceding comment.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -858,6 +873,26 @@ public void unlockAll(Task task)
     }
   }
 
+  public void unlockUnacquiredLocks(Task task)

Review Comment:
   Doesn't seem to be used anywhere.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskMaster.java:
##########
@@ -113,7 +113,7 @@ public void becomeLeader()
         log.info("By the power of Grayskull, I have the power!");
 
         try {
-          taskLockbox.syncFromStorage();
+          SyncResult syncResult = taskLockbox.syncFromStorage();

Review Comment:
   This is interesting. `TaskLockbox` is synced with storage just once when we become leader, whereas the `TaskQueue` is synced periodically?
   
   I wonder if we shouldn't just move the invocation of `taskLockbox.syncFromStorage()` to `TaskQueue.start()`. It would help `TaskQueue` ensure that `TaskLockbox` is actually synced without having to rely on something done before it was constructed.
   This would also help us avoid passing the `SyncResult` around.
   
   The only difference I see is if `taskLockbox.syncFromStorage()` throws an exception, then currently we would fail early, while becoming leader. Whereas, if we move the sync inside `TaskQueue`, it would fail on the lifecycle start, which is effectively the same.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -770,6 +776,17 @@ private void syncFromStorage()
           addTaskInternal(task);
         }
 
+        for (Task task : tasksToFail) {

Review Comment:
   I don't think this should happen in `syncFromStorage`. Put this in a separate method and call it after `syncFromStorage` in the lifecycle `start` method as it needs to be done only once.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +184,33 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s.",

Review Comment:
   Maybe include the group name in this log line too.
   
   And add another info log line which prints all the identified bad group names together, i.e. the contents of `failedToReacquireTaskLockGroups` and also the task ids that we have identified to cancel.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -770,6 +776,17 @@ private void syncFromStorage()
           addTaskInternal(task);
         }
 
+        for (Task task : tasksToFail) {
+          try {
+            tasks.putIfAbsent(task.getId(), task);

Review Comment:
   Why is this required? We are already synced with storage at this point. If the task id does not exist in the `tasks` map, it doesn't exist in the metadata store and thus doesn't hold any lock either. So we need not take further action on it, correct?



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/SyncResult.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Set;
+
+/**
+ * Result of TaskLockbox#syncFromStorage()
+ * Contains tasks which need to be forcefully failed to let the overlord become the leader
+ */
+class SyncResult

Review Comment:
   Maybe rename to `TaskLockboxSyncResult`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on PR #13172:
URL: https://github.com/apache/druid/pull/13172#issuecomment-1279652749

   @kfaraz, the changes have been tested on a druid cluster by introducing a bad entry in the metadata store to simulate this error and I monitored the changes for a while, and have ticked the corresponding box in the checklist.
   
   Thank you for the additional feedback. I'll be sure to test the changes on a cluster after addressing these comments as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996439170


##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");

Review Comment:
   The testing approach itself is fine but the names can be made a little more self-explanatory,
   like "TaskWithFailingLockAcquisition" or something. Name the Task instances also accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996569411


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -770,6 +776,17 @@ private void syncFromStorage()
           addTaskInternal(task);
         }
 
+        for (Task task : tasksToFail) {
+          try {
+            tasks.putIfAbsent(task.getId(), task);

Review Comment:
   With the previous changes, it was. It has been fixed while addressing your comments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996247487


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +187,39 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s from group %s.",
               savedTaskLockWithPriority.getInterval(),
               savedTaskLockWithPriority.getVersion(),
-              task.getId()
+              task.getId(),
+              task.getGroupId()
           );
+          continue;

Review Comment:
   Nit: probably not needed as we are already at the end of the loop.



##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");
+    final Task badTask1 = NoopTask.withGroupId("BadTask");
+    final Task goodTask0 = NoopTask.withGroupId("GoodTask");
+    taskStorage.insert(badTask0, TaskStatus.running(badTask0.getId()));
+    taskStorage.insert(badTask1, TaskStatus.running(badTask1.getId()));
+    taskStorage.insert(goodTask0, TaskStatus.running(goodTask0.getId()));
+
+    TaskLockbox testLockbox = new NullLockPosseTaskLockbox(taskStorage, metadataStorageCoordinator);
+    testLockbox.add(badTask0);
+    testLockbox.add(badTask1);
+    testLockbox.add(goodTask0);
+
+    testLockbox.tryLock(badTask0, new TimeChunkLockRequest(TaskLockType.EXCLUSIVE,

Review Comment:
   Nit: Style: Put each arg on a separate line (when necessary) for consistency with the rest of Druid code.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -207,7 +230,8 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
    * This method is called only in {@link #syncFromStorage()} and verifies the given task and the taskLock have the same
    * groupId, dataSource, and priority.
    */
-  private TaskLockPosse verifyAndCreateOrFindLockPosse(Task task, TaskLock taskLock)
+  @VisibleForTesting

Review Comment:
   Nit: is there a way to avoid this and still be able to test it? (without too much hassle)



##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/http/OverlordTest.java:
##########
@@ -270,19 +300,20 @@ public void testOverlordRun() throws Exception
     Assert.assertEquals(TASK_LOCATION, taskResponseObject.getLocation());
 
     // Simulate completion of task_1
-    taskCompletionCountDownLatches[Integer.parseInt(taskId_1)].countDown();
+    taskCompletionCountDownLatches.get(taskId_1).countDown();
     // Wait for taskQueue to handle success status of task_1
     waitForTaskStatus(taskId_1, TaskState.SUCCESS);
 
     // should return number of tasks which are not in running state
     response = overlordResource.getCompleteTasks(null, req);
-    Assert.assertEquals(2, (((List) response.getEntity()).size()));
+    Assert.assertEquals(4, (((List) response.getEntity()).size()));

Review Comment:
   Why did we need to change an existing test?
   I would advise adding a separate test for verifying the behaviour of such tasks where we failed to reacquire locks.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +187,39 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s from group %s.",
               savedTaskLockWithPriority.getInterval(),
               savedTaskLockWithPriority.getVersion(),
-              task.getId()
+              task.getId(),
+              task.getGroupId()
           );
+          continue;
+        }
+      }
+
+      Set<Task> tasksToFail = new HashSet<>();
+      for (Task task : taskStorage.getActiveTasks()) {

Review Comment:
   We don't want to make another call to the storage. Use an iterator over the `activeTasks` or `storedActiveTasks`. Either should be fine.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -173,6 +174,12 @@ public void start()
       Preconditions.checkState(!active, "queue must be stopped");
       active = true;
       syncFromStorage();
+      // Mark these tasks as failed as they could not reacuire the lock
+      // Clean up needs to happen after tasks have been synced from storage
+      Set<Task> tasksToFail = taskLockbox.syncFromStorage().getTasksToFail();
+      for (Task task : tasksToFail) {
+        shutdown(task.getId(), "Failed to reacquire lock.");

Review Comment:
   ```suggestion
           shutdown(task.getId(), "Shutting down forcefully as failed to reacquire lock after becoming leader.");
   ```
   It's a little verbose but paints a clearer picture of what happened.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +187,39 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s from group %s.",
               savedTaskLockWithPriority.getInterval(),
               savedTaskLockWithPriority.getVersion(),
-              task.getId()
+              task.getId(),
+              task.getGroupId()
           );
+          continue;
+        }
+      }
+
+      Set<Task> tasksToFail = new HashSet<>();
+      for (Task task : taskStorage.getActiveTasks()) {
+        if (failedToReacquireLockTaskGroups.contains(task.getGroupId())) {
+          tasksToFail.add(task);
+          activeTasks.remove(task.getId());

Review Comment:
   Nit: Style: You could choose to remove all of them in one go, thus retaining the sense of atomic update to `activeTasks`.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -228,6 +235,13 @@ public ScheduledExecutors.Signal call()
           }
       );
       requestManagement();
+      // Remove any unacquired locks from storage

Review Comment:
   Thanks for adding the comment!
   I am a little unclear on why there would be unacquired locks left behind. I would imagine that `shutdown()` would take care of this. If not, please rephrase the comments to clarify that point.



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -139,6 +141,8 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
       running.clear();
       activeTasks.clear();
       activeTasks.addAll(storedActiveTasks);
+      // Set of task groups in which at least one task failed to re-acquire a lock

Review Comment:
   Thanks for the comments!



##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");

Review Comment:
   Probably a better name explaining why it's bad or good?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r985838562


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -197,17 +205,26 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
           activeTasks.size(),
           storedLocks.size() - taskLockCount
       );
+
+      for (Task task : failedToReacquireLockTasks) {
+        for (TaskLock lock : taskStorage.getLocks(task.getId())) {
+          taskStorage.removeLock(task.getId(), lock);
+        }
+        taskStorage.setStatus(TaskStatus.failure(task.getId(), "Failed to reacquire lock"));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r986029406


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -197,7 +206,19 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
           activeTasks.size(),
           storedLocks.size() - taskLockCount
       );
+
+      // unlock all TaskLockPosse created, and also remove lock entries from storage
+      for (Task task : failedToReacquireLockTasks) {
+        unlockAll(task);
+        for (TaskLock lock : taskStorage.getLocks(task.getId())) {
+          taskStorage.removeLock(task.getId(), lock);
+        }
+      }
+      return new SyncResult(
+          ImmutableList.copyOf(failedToReacquireLockTasks)

Review Comment:
   I think wrapping inside ImmutableList is unnecessary. 



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -197,7 +206,19 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
           activeTasks.size(),
           storedLocks.size() - taskLockCount
       );
+
+      // unlock all TaskLockPosse created, and also remove lock entries from storage
+      for (Task task : failedToReacquireLockTasks) {
+        unlockAll(task);
+        for (TaskLock lock : taskStorage.getLocks(task.getId())) {
+          taskStorage.removeLock(task.getId(), lock);
+        }

Review Comment:
   should this all be in a block where the exceptions can be ignored? 



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/SyncResult.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.List;
+
+/**
+ * Result of TaskLockbox#syncFromStorage()

Review Comment:
   nit-pick - This description could be improved. 



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskMaster.java:
##########
@@ -416,4 +418,22 @@ public Map<String, Long> getBlacklistedTaskSlotCount()
       return null;
     }
   }
+
+  /**
+   * Process the results of synchronization from storage
+   *
+   * @param syncResult to be processed
+   * @param taskStorage stores tasks
+   * @param taskRunner runs tasks and holds state
+   */
+  private void processSyncResult(SyncResult syncResult, TaskStorage taskStorage, TaskRunner taskRunner)
+  {
+    String taskReacquisitionFailure = "Failed to reacquire lock";
+    for (Task task : syncResult.getTasksToFail()) {
+      // Shutdown task if it is running
+      taskRunner.shutdown(task.getId(), taskReacquisitionFailure);
+      // Mark as failed
+      taskStorage.setStatus(TaskStatus.failure(task.getId(), taskReacquisitionFailure));

Review Comment:
   should we ignore any exception in this block? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996627161


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -183,20 +187,39 @@ public int compare(Pair<Task, TaskLock> left, Pair<Task, TaskLock> right)
             );
           }
         } else {
-          throw new ISE(
-              "Could not reacquire lock on interval[%s] version[%s] for task: %s",
+          failedToReacquireLockTaskGroups.add(task.getGroupId());
+          log.error(
+              "Could not reacquire lock on interval[%s] version[%s] for task: %s from group %s.",
               savedTaskLockWithPriority.getInterval(),
               savedTaskLockWithPriority.getVersion(),
-              task.getId()
+              task.getId(),
+              task.getGroupId()
           );
+          continue;
+        }
+      }
+
+      Set<Task> tasksToFail = new HashSet<>();
+      for (Task task : taskStorage.getActiveTasks()) {

Review Comment:
   Fixed



##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -173,6 +174,12 @@ public void start()
       Preconditions.checkState(!active, "queue must be stopped");
       active = true;
       syncFromStorage();
+      // Mark these tasks as failed as they could not reacuire the lock
+      // Clean up needs to happen after tasks have been synced from storage
+      Set<Task> tasksToFail = taskLockbox.syncFromStorage().getTasksToFail();
+      for (Task task : tasksToFail) {
+        shutdown(task.getId(), "Failed to reacquire lock.");

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996627672


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -228,6 +235,13 @@ public ScheduledExecutors.Signal call()
           }
       );
       requestManagement();
+      // Remove any unacquired locks from storage

Review Comment:
   Included



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r995374764


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/SyncResult.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Set;
+
+/**
+ * Result of TaskLockbox#syncFromStorage()
+ * Contains tasks which need to be forcefully failed to let the overlord become the leader
+ */
+class SyncResult

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on PR #13172:
URL: https://github.com/apache/druid/pull/13172#issuecomment-1278532808

   @kfaraz thank you for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kfaraz commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996438652


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -228,6 +235,13 @@ public ScheduledExecutors.Signal call()
           }
       );
       requestManagement();
+      // Remove any unacquired locks from storage

Review Comment:
   Okay, please include this info in the comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996249588


##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -173,6 +174,12 @@ public void start()
       Preconditions.checkState(!active, "queue must be stopped");
       active = true;
       syncFromStorage();
+      // Mark these tasks as failed as they could not reacuire the lock
+      // Clean up needs to happen after tasks have been synced from storage
+      Set<Task> tasksToFail = taskLockbox.syncFromStorage().getTasksToFail();
+      for (Task task : tasksToFail) {
+        shutdown(task.getId(), "Failed to reacquire lock.");

Review Comment:
   Thanks for the suggestion. Shouldn't it be "while becoming leader"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13172: Fix Overlord leader election when task lock re-acquisition fails

Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13172:
URL: https://github.com/apache/druid/pull/13172#discussion_r996249897


##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskLockboxTest.java:
##########
@@ -1258,6 +1259,39 @@ public void testGetLockedIntervalsForRevokedLocks() throws Exception
     );
   }
 
+  @Test
+  public void testFailedToReacquireTaskLock() throws Exception
+  {
+    final Task badTask0 = NoopTask.withGroupId("BadTask");

Review Comment:
   I added a TestLockbox class which returns null for tasks whose group contains "BadTask". I'm not sure it's the best approach to test these changes.
   I'm hoping you could take a look and suggest a better approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org