[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7743: Always use SMALLEST offset criteria for new detected partitions

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on a change in pull request #7743:
URL: https://github.com/apache/pinot/pull/7743#discussion_r747876433



##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
##########
@@ -872,8 +872,14 @@ public void ensureAllPartitionsConsuming(TableConfig tableConfig, PartitionLevel
       if (idealState.isEnabled()) {
         List<PartitionGroupConsumptionStatus> currentPartitionGroupConsumptionStatusList =
             getPartitionGroupConsumptionStatusList(idealState, streamConfig);
+
+        // Read the smallest offset when a new partition is detected
+        OffsetCriteria originalOffsetCriteria = streamConfig.getOffsetCriteria();
+        streamConfig.setOffsetCriteria(OffsetCriteria.SMALLEST_OFFSET_CRITERIA);

Review comment:
       Good question. When creating the `StreamConfig`, the map from the table config is copied, and maintained separately, so there is no race condition on that.
   
   I actually started with the suggested approach by adding OffsetCriteria override to the method, then found that I need to carry the extra parameter all the way to the `StreamMetadataProvider` interface, and could pollute the interface and potentially cause unexpected behavior for the implementations of the interface. Then I decided to go with the changing stream config approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org