[GitHub] [incubator-pinot] mcvsubbu commented on issue #6532: Server Rack Metadata Retrieval and Persistence on Azure Environment

[GitBox <gi...@apache.org>]

mcvsubbu commented on issue #6532:
URL: https://github.com/apache/incubator-pinot/issues/6532#issuecomment-798771506


   @pabrahamusa , @fx19880617 , the idea here is to make it cloud neutral. The goal is to assign instances in such a way that each instance for a table is in an "island". The definition of an "island" is cloud-specific. The requirement is that the provider gives a guarantee that should there be some planned outage, no two "island"s will be taken down at the same time. For Azure, this happens to be called Fault Domain. For a data center providing native hardware, this turns out to be "rack". If there are APIs to recognize the "island" to which an instance belongs (or, APIs to ask for N instances in N -- or even M  different islands), then we can potentially use those APIs to arrive at an optimal instance assignment for a table.
   
   The approach followed is that we mark in a single place in zookeeper (at cluster level) the "ID" of the provider (e.g. "Azure" or "XYZ"). We then have specific paths in ZK where the characteristics, the API, perhaps the name of the plugin to be loaded, etc. are specified for the "ID" in question. 
   
   If the ID is not specified, an installation will not see any modifications to Znodes or any new znodes. @xulinjintu  is driving the design for this, so please provide input in the design document.
   
   cc: @kishoreg 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org