You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/11/03 18:19:08 UTC

[GitHub] [druid] churromorales opened a new pull request, #13305: Fixing the K8s task runner to work with MSQ

churromorales opened a new pull request, #13305:
URL: https://github.com/apache/druid/pull/13305

   MSQ can work with tasks that have not yet launched, there was a bug in the k8s task runner where you could return a null host because the pod was still not up.  Fixed this to return a `TaskLocation.unknown()` if this happens to msq works.  Tested this on our internal clusters and it seemed to fix the problem with msq + k8s peon jobs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] churromorales commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
churromorales commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1303806747

   @abhishekagarwal87 it has even more, in our local fork i didn't write a test ;).  Seriously though, here was the original issue: 
   
   https://github.com/apache/druid/pull/13156#issuecomment-1291145564
   
   this fixes that issue, i tested and tasks index and i can query them, before it would not even start. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] churromorales commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
churromorales commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1305868317

   > @churromorales - does this PR has all the changes pushed?
   
   @abhishekagarwal87 I am so sorry, you are absolutely right, I missed the core logic here that was causing the issue.  I forgot to push up the last commit.  Now it should be good.  I am very sorry about that. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] churromorales commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
churromorales commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1305868922

   > @churromorales Thanks for the PR.
   > 
   > I went through it and it seems we are not setting `TaskLocation.unknown()` anywhere. Maybe I am missing something.
   
   I forgot to push up a commit that was local.  I apologize. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] churromorales commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
churromorales commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1302534201

   @gianm you were right, the bug was due to the worker task's pod being scheduled but it wasn't yet launched (so it had no podIP yet) thus, the task location was partially set.  The fix is pretty small, I tested this on my local cluster and MSQ now works.  Let me know what you think.  Thanks for helping look into this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 merged pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 merged PR #13305:
URL: https://github.com/apache/druid/pull/13305


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1303101999

   @churromorales - does this PR has all the changes pushed? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on PR #13305:
URL: https://github.com/apache/druid/pull/13305#issuecomment-1306873279

   Thank you @churromorales 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13305:
URL: https://github.com/apache/druid/pull/13305#discussion_r1014605161


##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -494,20 +505,28 @@ public RunnerTaskState getRunnerTaskState(String taskId)
     }
   }
 
-  public class K8sWorkItem extends TaskRunnerWorkItem
+  public static class K8sWorkItem extends TaskRunnerWorkItem
   {
     private final Task task;
+    private KubernetesPeonClient client;
 
-    private K8sWorkItem(Task task, ListenableFuture<TaskStatus> statusFuture)
+    public K8sWorkItem(KubernetesPeonClient client, Task task, ListenableFuture<TaskStatus> statusFuture)
     {
       super(task.getId(), statusFuture);
       this.task = task;
+      this.client = client;
     }
 
-    private K8sWorkItem(Task task, ListenableFuture<TaskStatus> statusFuture, DateTime createdTime)
+    public K8sWorkItem(
+        KubernetesPeonClient client,
+        Task task,
+        ListenableFuture<TaskStatus> statusFuture,
+        DateTime createdTime
+    )
     {
       super(task.getId(), statusFuture, createdTime, createdTime);
       this.task = task;
+      this.client = client;

Review Comment:
   where does it get accessed? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] churromorales commented on a diff in pull request #13305: Fixing the K8s task runner to work with MSQ

Posted by GitBox <gi...@apache.org>.
churromorales commented on code in PR #13305:
URL: https://github.com/apache/druid/pull/13305#discussion_r1014645283


##########
extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:
##########
@@ -494,20 +505,28 @@ public RunnerTaskState getRunnerTaskState(String taskId)
     }
   }
 
-  public class K8sWorkItem extends TaskRunnerWorkItem
+  public static class K8sWorkItem extends TaskRunnerWorkItem
   {
     private final Task task;
+    private KubernetesPeonClient client;
 
-    private K8sWorkItem(Task task, ListenableFuture<TaskStatus> statusFuture)
+    public K8sWorkItem(KubernetesPeonClient client, Task task, ListenableFuture<TaskStatus> statusFuture)
     {
       super(task.getId(), statusFuture);
       this.task = task;
+      this.client = client;
     }
 
-    private K8sWorkItem(Task task, ListenableFuture<TaskStatus> statusFuture, DateTime createdTime)
+    public K8sWorkItem(
+        KubernetesPeonClient client,
+        Task task,
+        ListenableFuture<TaskStatus> statusFuture,
+        DateTime createdTime
+    )
     {
       super(task.getId(), statusFuture, createdTime, createdTime);
       this.task = task;
+      this.client = client;

Review Comment:
   Used for testing.  I needed to inject the client to mock out 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org