[GitHub] [incubator-druid] jihoonson opened a new issue #8718: Fix graceful shutdown for tasks

[GitBox <gi...@apache.org>]

jihoonson opened a new issue #8718: Fix graceful shutdown for tasks
URL: https://github.com/apache/incubator-druid/issues/8718
 
 
   ### Affected Version
   
   0.14, 0.15, 0.16
   
   ### Description
   
   Since https://github.com/apache/incubator-druid/pull/6828, [task shutdown is always the graceful shutdown](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/SingleTaskBackgroundRunner.java#L184-L186). This was for some task types to clean up their resources. For example, Hadoop task kills its Hadoop job when it's stopped (https://github.com/apache/incubator-druid/pull/6828). The parallel indexing task kills its all running sub tasks (https://github.com/apache/incubator-druid/pull/7041). 
   
   However, this is different from how the stream ingestion task had been using graceful shutdown. On graceful shutdown, they immediately start persisting all segments in memory onto disk and publishing segments before they stop. This is to create a checkpoint when they stop, not for resource cleanup. As a result, https://github.com/apache/incubator-druid/pull/6828 unexpectedly changed the behavior of Kafka/Kinesis indexing service when the supervisor kills tasks.
   
   I think we need to distinguish immediate stop and graceful stop again. Immediate stop always involves resource cleanup. Graceful stop may involve extra work in addition to necessary resource cleanup.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org