[GitHub] [incubator-pinot] Jackie-Jiang opened a new issue #4239: Instance Assignment for Multi-tenancy Clusters

[GitBox <gi...@apache.org>]

Jackie-Jiang opened a new issue #4239: Instance Assignment for Multi-tenancy Clusters
URL: https://github.com/apache/incubator-pinot/issues/4239
 
 
   **Motivation**
   
   Currently, when we get a new segment and try to assign it to some instances, we don’t have a step of selecting instances to serve the table, but purely rely on the tenant name for the table. The tenant name for each instance is derived from the Helix tag, which means all the instances with the same Helix tag will be selected to serve the table.
   This approach is okay for single-tenant cluster because all the instances are allocated to serve the table. But when it goes to the multi-tenant cluster with a large number of machines, it is not as scalable.
   
   For multi-tenant cluster, segments from a table will spread over all the machines without any constraints, which can cause the following issues:
   1. During deployment, when performing a rolling restart, any two servers will likely serve the same segment (as 2 replicas), so there is no way to restart 2 servers in parallel while only bringing 1 replica down for tables. In order to perform a restart without downtime, we can only bring {min replicas for all tables in the cluster - 1} servers down at the same time. For a cluster with more than 100 servers, deployment will take days to finish.
   2. Each table is served by a large number of servers, which means broker will have a large fan-out for the queries. This will add extra work on both broker side (scatter-gather to/from more servers) and server side (more requests from brokers).
   3. Because each table is served by more servers, each server will also serve more tables. Whenever the current states changed for the machine (restart, ZK disconnect etc), more tables get affected, thus controller need to gather more information and update more external views.
   
   Limiting the number of instances in a single cluster can alleviate these problems, but will add extra unnecessary limitations and management cost. We should shoot for a solution to improve the scalability of the cluster instead of limiting the scale.
   
   In order to solve these problems, we propose to split the segment assignment into 2 steps:
   1. Instance Assignment: when creating the table, assign instances (servers) to serve it based on the instance assignment strategy. All the segments for this table will only be served by the assigned instances.
   2. Segment Assignment: when adding a new segment, pick some instances from the assigned instances for the table to assign the new segment based on the segment assignment strategy.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org