[GitHub] [incubator-druid] pdeva opened a new issue #8175: allow decommissioning middle manage nodes

[GitBox <gi...@apache.org>]

pdeva opened a new issue #8175: allow decommissioning middle manage nodes
URL: https://github.com/apache/incubator-druid/issues/8175
 
 
   This proposal is written with Kafka Indexing Tasks in mind only.
   Please add your thoughts about other indexing tasks.
   
   ### Description
   
   Currently removing and adding a new MM node causes temporary phase of data blindness, essentially as good as 'downtime' for users.
   
   Allow decommissioning of Middle Manager nodes, so that new nodes can be added and tasks can be rescheduled on those instantly and without any temporary loss of data.
   
   ### Motivation
   
   Currently if you want to remove and add a new MM node, here is the experience. 
   
   Assuming 2 existing nodes A & B, with datasource DS with 2 replicas of KIS tasks.
   
   1. Add a new MM node C
   1. Shut down MM node B.
   1. DS KIS tasks on node B remain dormant for about 15 minutes
   1. after 15 mins the task is set to 'failed' state and rescheduled on node C.
   1. since task has to read from begining, the brokers temporary see no data for the last 15 mins or so. thus essentially causing a 'downtime'.
   1. after a few minutes node C has fully ingested incoming data and things go back to normal
   
   Ideal scenarios:
   
   1. add new node C
   2. decommission node B
   3. tasks on node B immediately die with status 'success' and new ones are spun up on Node C
   4. no temporary date loss is seen by the brokers
   
   its similar to how currently if you change a supervisor task definition, it starts a new task transparently without any temporary data loss/blindess to the brokers.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org