[GitHub] jihoonson opened a new issue #6042: KafkaSupervisor should wait for all running tasks to stop on shutdown

[GitBox <gi...@apache.org>]

jihoonson opened a new issue #6042: KafkaSupervisor should wait for all running tasks to stop on shutdown
URL: https://github.com/apache/incubator-druid/issues/6042
 
 
   When KafkaSupervisor is asked to shut down, it currently tries to shut down all running tasks, but simply gives up if shutdown requests fail or time out (1 min by default in `RemoteTaskRunner`). As a result, tasks can remain running even though the supervisor is stopped and this is really not good for users because they need to kill them all by hand.
   
   I suggest to change KafkaSupervisor should wait until all running tasks stop. This also requires a change for the `/druid/indexer/v1/supervisor/{supervisor_id}/shutdown` to be async because it can take a long time to finish. During waiting for tasks to stop, users should be able to know the supervisor is in the _SHUTTING_DOWN_ state and get the shutdown progress (how many tasks it's waiting, ...). The supervisor also needs to retry on shutdown request failures.
   
   Side issue1: `RemoteTaskRunner` uses HTTP only for sending shutdown requests. I wonder why it does even though every other requests is sent via ZooKeeper (except streaming task logs).
   
   Side issue2: Once the supervisor stops a running task on shutdown, the task is finished as `FAILED`. We need to add a new taskState something like `STOPPED`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org