[GitHub] [incubator-druid] sascha-coenen commented on issue #8841: add readiness endpoints to processes having initialization delays

[GitBox <gi...@apache.org>]

sascha-coenen commented on issue #8841: add readiness endpoints to processes having initialization delays
URL: https://github.com/apache/incubator-druid/pull/8841#issuecomment-551945335
 
 
   BTW: readiness endpoints alone will not suffice to integrate Druid well with Kubernetes.
   Reason being that Kubernetes will only allow incoming requests to hit a service (e.g. a Historical ) after its readiness check passed. However, Druid uses zookeeper to discover if a new historical has joined the cluster. So as soon an a historical reports its existence to zookeeper (or to the coordinator if http announcement is configured), then the brokers think that the historical is ready for use and will start sending queries to it. However, those queries will not arrive at those historicals unless their readiness healthcheck has also passed already. So there is this intermittent timespan during which a broker cannot reach a historical and during this time all queries will fail because the broker will keep contacting the same historical, even if query-retries are configured. On retry, the same historical gets picked.
   
   I was wandering how to improve this behaviour during rolling updates.
   
   * One improvement would be to have the broker graylist a failing historical during retries.
   
   * Another improvement would be to add a configurable delay "druid.server.http.announcePropagationDelay" such that after a zookeeper announcement of a newly joined service (e.g. historical), there would be some delay until other services (e.g. brokers) would start to interact with the joined service. This would be similar to the existing property "druid.server.http.unannouncePropagationDelay"
   The purpose of this would be that if Kubernetes is set up to execute a readiness check every 30 seconds, one could set the announcePropagationDelay to a larger time like 45 seconds, so that there would be no situation in which one Druid service tries to talk to another one at a time when Kubernetes is not (yet) allowing incoming traffic to the latter service.
   
   I just meant to mention it in this place because when I saw the title of this PR, especially the "having initialization delays" I hoped that the PR would also be about adding this internal delay.
   
   I have so far not found another workaround in kubernetes yet. It is certainly foremost a weakness in kubernetes that causes this misbehaviour because the readiness concept violates the single-responsibilty principle. A readiness check is used for two things:
   a) to decide whether incoming network connections are allowed for a given pod
   b) to decide whether a rolling update can already continue updating the next pod
   
   Unfortunately there seems to be no way to have separate timings for these two unrelated aspects in k8s. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org