You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "georgew5656 (via GitHub)" <gi...@apache.org> on 2023/04/05 01:42:20 UTC

[GitHub] [druid] georgew5656 opened a new pull request, #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

georgew5656 opened a new pull request, #14030:
URL: https://github.com/apache/druid/pull/14030

   ### Description
   Discovered a race condition in the shutdown code while doing some additional testing around https://github.com/apache/druid/commit/f60f377e5f67221f8a4a34163b257ecffece60d7
   
   If a task is shutdown, has its k8s job deleted and then subsequently removed from the tasks map on https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java#L218, 
   it is possible for the pod to still be running in a terminating state for a little while after. this is because k8s deletes the controller (the job) and then lets the pod clean itself up.
   
   If getKnownTasks (https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java#L338) is called by the TaskQueue main loop between when the tasks table is cleaned up and the peon pod finishes deleting, another run() will be called by getKnownTasks.
   
   Normally this sees the existing future in the tasks table and returns it, but since the task id will already have been deleted from the tasks table, the task runner will actually start another job with the same name as the original one that was deleted.
   
   This job will quickly be deleted again (because getKnownTasks will return it, the taskQueue will see the task that the task runner returned is not in TaskStorage and submit another shutdown request to delete the job), but this is a poor user experience.
   
   Additionally, there may be DuplicateKeyError's thrown because there are multiple pods with the same job name running, this can cause the ingestion to be briefly nonresponsive.
   
   It is also possible to get these DuplicateKeyErrors if a pod (not a job) is manually deleted in K8s. Normally, since we set backoffLimit to 0, if a pod fails, the job doesn't try to retry creating the pod and just fails out. But it seems like k8s treats manual termination differently from a pod failing, so when a pod is manually terminated, the job actually starts another pod.
   
   
   IMO it makes more sense for getKnownTasks to try to rehydrate state from Kubernetes via the list of jobs, rather than the list of Pods. This solves both of the above problems because we can be sure the K8s job has been deleted before deleting the task future from the tasks map, and there will never be duplicate jobs with the same name.
   #### Release note
   
   Bugfixes to the kubernetes overlord extension
   
   ##### Key changed/added classes in this PR
   - The change here to have toTask take in the Job object rather than the Pod object and then use this change to have getKnownTasks and getRunningTasks list jobs rather than pods when trying to read the k8s state.
   
   
   I explored some other options for fixing this issue, such as excluding terminating pods from getKNownTasks and getRunningTasks or having the main run loop wait for pods to delete if the job has been deleted, but this seemed like the cleanest solution.
   
   This PR has:
   
   - [ X been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [X] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [X] been tested in a test Druid cluster.
   I have tested more thoroughly with the PodTemplateTaskAdapter and done some smoke tests with the MultiContainerTaskAdapter. I'm currently doing some additional testing in my local cluster to try to catch any additional issues.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] georgew5656 commented on a diff in pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on code in PR #14030:
URL: https://github.com/apache/druid/pull/14030#discussion_r1158762517


##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -425,7 +425,9 @@ public void registerListener(TaskRunnerListener listener, Executor executor)
   public Collection<TaskRunnerWorkItem> getRunningTasks()
   {
     List<TaskRunnerWorkItem> result = new ArrayList<>();
-    for (Pod existingTask : client.listPeonPods(Sets.newHashSet(PeonPhase.RUNNING))) {
+    for (Job existingTask : client.listAllPeonJobs().stream()
+        .filter(job -> job.getStatus() != null && job.getStatus().getActive() != null && job.getStatus().getActive() > 0).collect(Collectors.toSet())

Review Comment:
   i added a class to do this, I think it may change slightly once https://github.com/apache/druid/pull/14028#pullrequestreview-1373136808 is merged so I'll have to publish a new version once thats merged



##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -338,7 +338,7 @@ public Map<String, Long> getTotalTaskSlotCount()
   public Collection<? extends TaskRunnerWorkItem> getKnownTasks()
   {
     List<TaskRunnerWorkItem> result = new ArrayList<>();
-    for (Pod existingTask : client.listPeonPods()) {
+    for (Job existingTask : client.listAllPeonJobs()) {

Review Comment:
   this is done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nlippis commented on pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "nlippis (via GitHub)" <gi...@apache.org>.
nlippis commented on PR #14030:
URL: https://github.com/apache/druid/pull/14030#issuecomment-1497869740

   > Looks good, let's merge after #14028 is in.
   
   #14028  has been merged, please pull in those changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14030:
URL: https://github.com/apache/druid/pull/14030#discussion_r1158002223


##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -425,7 +425,9 @@ public void registerListener(TaskRunnerListener listener, Executor executor)
   public Collection<TaskRunnerWorkItem> getRunningTasks()
   {
     List<TaskRunnerWorkItem> result = new ArrayList<>();
-    for (Pod existingTask : client.listPeonPods(Sets.newHashSet(PeonPhase.RUNNING))) {
+    for (Job existingTask : client.listAllPeonJobs().stream()
+        .filter(job -> job.getStatus() != null && job.getStatus().getActive() != null && job.getStatus().getActive() > 0).collect(Collectors.toSet())

Review Comment:
   Good idea. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nlippis commented on a diff in pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "nlippis (via GitHub)" <gi...@apache.org>.
nlippis commented on code in PR #14030:
URL: https://github.com/apache/druid/pull/14030#discussion_r1157936637


##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -425,7 +425,9 @@ public void registerListener(TaskRunnerListener listener, Executor executor)
   public Collection<TaskRunnerWorkItem> getRunningTasks()
   {
     List<TaskRunnerWorkItem> result = new ArrayList<>();
-    for (Pod existingTask : client.listPeonPods(Sets.newHashSet(PeonPhase.RUNNING))) {
+    for (Job existingTask : client.listAllPeonJobs().stream()
+        .filter(job -> job.getStatus() != null && job.getStatus().getActive() != null && job.getStatus().getActive() > 0).collect(Collectors.toSet())

Review Comment:
   Instead of defining logic for whether the job is active, succeeded or failed in various places.  Can you create a class with a method for each state.  I.E
   
   ```
   JobStatus.isActive(Job job)
   JobStatus.isSucceeded(Job job)
   JobStatus.isFailed(Job job)
   ```



##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -338,7 +338,7 @@ public Map<String, Long> getTotalTaskSlotCount()
   public Collection<? extends TaskRunnerWorkItem> getKnownTasks()
   {
     List<TaskRunnerWorkItem> result = new ArrayList<>();
-    for (Pod existingTask : client.listPeonPods()) {
+    for (Job existingTask : client.listAllPeonJobs()) {

Review Comment:
   The `listPeonPods()` and `listPeonPods(Set<PeonPhase> phases)` methods will be unused if you make this change right?  If so please remove them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "github-code-scanning[bot] (via GitHub)" <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #14030:
URL: https://github.com/apache/druid/pull/14030#discussion_r1159270399


##########
extensions-contrib/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/K8sTaskAdapterTest.java:
##########
@@ -163,22 +162,19 @@
         task,
         new PeonCommandContext(new ArrayList<>(), new ArrayList<>(), new File("/tmp/"))
     );
-
-    // cant launch jobs with test server, we have to hack around this.
-    Pod pod = K8sTestUtils.createPodFromJob(jobFromSpec);
-    client.pods().inNamespace("test").create(pod);
-    PodList podList = client.pods().inNamespace("test").list();
-    assertEquals(1, podList.getItems().size());
+    client.batch().v1().jobs().inNamespace("test").create(jobFromSpec);

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [ItemWritableOperation.create](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4731)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] georgew5656 commented on pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on PR #14030:
URL: https://github.com/apache/druid/pull/14030#issuecomment-1498030759

   > Looks good, let's merge after #14028 is in.
   
   just resolved merge conflicts, gonna do some more smoke testing once I can figure out how to get the changes from #14028 deployed to my environment but this should be good to go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis merged pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #14030:
URL: https://github.com/apache/druid/pull/14030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] georgew5656 commented on pull request #14030: Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks

Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on PR #14030:
URL: https://github.com/apache/druid/pull/14030#issuecomment-1500777164

   > > Looks good, let's merge after #14028 is in.
   > 
   > just resolved merge conflicts, gonna do some more smoke testing with the new fabric8 clients once I can figure out how to get the changes from #14028 deployed to my environment but this should be good to go
   
   @nlippis i got this deployed into my testing environment and it looks good to go with the new fabric8 client, other than the missing dependency from https://github.com/apache/druid/pull/14052


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org