[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #4211: StreamPartitionAssignmentStrategy interface to plug in various partition assignment strategies (uniform, replica group based)

[GitBox <gi...@apache.org>]

mcvsubbu commented on a change in pull request #4211: StreamPartitionAssignmentStrategy interface to plug in various partition assignment strategies (uniform, replica group based)
URL: https://github.com/apache/incubator-pinot/pull/4211#discussion_r284814502
 
 

 ##########
 File path: pinot-common/src/main/java/org/apache/pinot/common/partition/StreamPartitionAssignmentStrategy.java
 ##########
 @@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.partition;
+
+import java.util.List;
+import javax.annotation.Nonnull;
+import org.apache.helix.HelixManager;
+import org.apache.pinot.common.exception.InvalidConfigException;
+
+
+/**
+ * Creates a partition assignment for the partitions of a realtime table
+ */
+// TODO: Unify the interfaces for {@link org.apache.pinot.controller.helix.core.sharding.SegmentAssignmentStrategy} and {@link StreamPartitionAssignmentStrategy}
 
 Review comment:
   While it is nice to combine the interfaces, we need to be clear whether it will allow us any re-use. It will be a pity if we have code  like "if (partition assignment) then do this else do that" inside. Here is my reasoning:
   
   Segment assignment strategy is given 1 or more segments, and returns the servers on which they should be hosted. It is invoked each time new segments shows up. It does NOT change any assignment of existing segments. If the segment is a CONSUMING segment (a new one comes up each time previous one in a stream partition completes), then our goal should be to assign it to a set of servers that have previously hosted consuming segments from the table. We currently do the simple strategy of assigning it to the same server that the previous segment completed. So, essentially:
   
   segmentAssignmentStrategy.assign(Collection segments) returns a collection of servers.
   
   On the other hand, (stream) partition assignment strategy should be invoked at the table creation time, each time we get new partitions (not very often). Furthermore, the partition assignment strategy MAY change the existing assignment for partitions previously known. Therefore:
   
   partitionAssignmentStrategy.assign(numPartitions) returns a map of partition to a collection of servers.
   
   @kishoreg  and @Jackie-Jiang let me know if I am off here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org