[GitHub] jihoonson opened a new issue #6019: Race in TaskQueue when updating status of complete tasks

[GitBox <gi...@apache.org>]

jihoonson opened a new issue #6019: Race in TaskQueue when updating status of complete tasks
URL: https://github.com/apache/incubator-druid/issues/6019
 
 
   In [TaskQueue.notifyStatus()](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java#L380), the overlord updates the status of a complete task. To do so, it first calls [taskRunner.shutdown(task.getId())](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java#L396) to remove the complete task from the taskRunner, and then calls [taskStorage.setStatus(taskStatus)](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java#L424) to update the status of the task in metastore. So, between them, the complete task is not in taskRunner, but its status in metastore is `RUNNING`.
   
   This might break the overlord APIs returning the task status. Since we don't have a good system to track all task status changes yet, the overlord tries to find waiting tasks by `(all not-complete tasks - tasks in taskRunner)`. But, because of the above race, a complete task might not be in taskRunner, but its status in metastore is still `RUNNING`, and the overlord might return a wrong task status as `WAITING`.
   
   The short and simple solution is to fix this race, but I would say that it would be better to make a system to track all task status changes which can record the waiting task status correctly.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org