[GitHub] [druid] sthetland commented on issue #9253: Web console: make supervisor reset really scary in the UI

[GitBox <gi...@apache.org>]

sthetland commented on issue #9253: Web console: make supervisor reset really scary in the UI
URL: https://github.com/apache/druid/pull/9253#issuecomment-578983511
 
 
   I took a quick look at doc impact. The only existing doc I could find where this concept is touched upon is https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors (and equivalent section in the Kinesis page). 
   
   That section can throttle up the fear factor a bit, but it would be good to put the explanation in docs that are set in the context of UI doc (rather than this operation endpoint doc linked). This would be a larger effort though, I as don't see an existing logical spot for that doc. Something for me to work on. 
   
   Meanwhile, taking a stab at how to reword the existing section to induce more fear, for one thing, by moving possible ill effects up :
   
   ```
   The `POST /druid/indexer/v1/supervisor/<supervisorId>/reset` operation clears stored 
   offsets, causing the supervisor to start reading offsets from either the earliest or latest 
   offsets in Kafka (depending on the value of useEarliestOffset). After clearing stored 
   offsets, the supervisor kills and recreates any active tasks, so that tasks begin reading 
   from valid offsets. 
   
   Use care when using this operation! Resetting the supervisor may cause Kafka messages 
   to be skipped or read twice, resulting in missing or duplicate data. 
   
   The reason for using this operation is to recover from a state in which the supervisor 
   ceases operating due to missing offsets. The indexing service keeps track of the latest 
   persisted Kafka offsets in order to provide exactly-once ingestion guarantees across 
   tasks. Subsequent tasks must start reading from where the previous task completed in 
   order for the generated segments to be accepted. If the messages at the expected 
   starting offsets are no longer available in Kafka (typically because the message retention 
   period has elapsed or the topic was removed and re-created) the supervisor will refuse 
   to start and in flight tasks will fail.
   
   This operation enables you to recover from this condition. Note that the supervisor 
   must be running for this endpoint to be available.
   ```
   
   (Change applies to https://github.com/apache/druid/blob/master/docs/development/extensions-core/kafka-ingestion.md)
   
   Let me know if my rewrite is accurate (and addresses the goal). If so, I’ll file a doc ticket/PR (or add it to this one?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org