[GitHub] sijie opened a new pull request #1392: [table service] a zk based storage container controller

[GitBox <gi...@apache.org>]

sijie opened a new pull request #1392: [table service] a zk based storage container controller 
URL: https://github.com/apache/bookkeeper/pull/1392
 
 
   Descriptions of the changes in this PR:
   
   *Motivation*
   
   The original storage container "controller" was written in helix. The helix controller's placement is kind of not assuming any properties of shared storage. however the table service is leverage bookkeeper's as its segment/log store. the table service is built in a more `stateless` way, where a storage container can be moved between servers in a much lightweight way.
   
   Also helix codebase is a bit large, if we eventually going to eliminate zookeeper, the cost of switching it off helix will be much expensive. so this PR is introducing a simple zk based controller for assigning storage containers to servers.
   
   *Solution*
   
   In this zk base solution, it is comprised of 3 parts: server registration, cluster & storage controller, storage container manager.
   
   **registration**
   
   this solution leverages existing registration client/manager interfaces. so each storage server registers itself under `/stream/servers/available`.
   
   **cluster & storage controller**
   
   A storage server is elected as a the leader, which runs a ClusterController. The cluster controller reads cluster metadata from zookeeper (the cluster metadata includes like how many containers available in the cluster), and use a storage controller to compute an assignment plan as the ideal state for container assignment mapping, and then update the assignment mapping to zookeeper under `/stream/assignment`.
   
   **storage container manager**
   
   Each storage server runs a storage container manager, which watches the assignment plan in zookeeper. The cluster controller updates the assignment plan. Once it receives watches/updates from the assignment plan that computed by cluster controller, the storage container manager will align itself to the ideal assignment map by starting/stopping storage containers.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services