[GitHub] [pinot] npawar commented on a change in pull request #8067: Wire EmptySegmentPruner to routing config

[GitBox <gi...@apache.org>]

npawar commented on a change in pull request #8067:
URL: https://github.com/apache/pinot/pull/8067#discussion_r792174926



##########
File path: pinot-broker/pom.xml
##########
@@ -145,5 +145,11 @@
       <artifactId>mockito-core</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.apache.pinot</groupId>
+      <artifactId>pinot-kinesis</artifactId>
+      <version>0.10.0-SNAPSHOT</version>
+      <scope>compile</scope>

Review comment:
       remove hardcoded version

##########
File path: pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java
##########
@@ -188,36 +178,31 @@ public void testSegmentPrunerFactoryForTimeRangePruner() {
 
     // Routing config is missing
     List<SegmentPruner> segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
-    assertEquals(segmentPruners.size(), 1);
-    assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+    assertEquals(segmentPruners.size(), 0);
 
     // Segment pruner type is not configured
     RoutingConfig routingConfig = mock(RoutingConfig.class);
     when(tableConfig.getRoutingConfig()).thenReturn(routingConfig);
     segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
-    assertEquals(segmentPruners.size(), 1);
-    assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+    assertEquals(segmentPruners.size(), 0);
 
     // Validation config is missing
     when(routingConfig.getSegmentPrunerTypes()).thenReturn(
         Collections.singletonList(RoutingConfig.TIME_SEGMENT_PRUNER_TYPE));
     segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
-    assertEquals(segmentPruners.size(), 1);
-    assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+    assertEquals(segmentPruners.size(), 0);
 
     // Time column is missing
     SegmentsValidationAndRetentionConfig validationConfig = mock(SegmentsValidationAndRetentionConfig.class);
     when(tableConfig.getValidationConfig()).thenReturn(validationConfig);
     segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
-    assertEquals(segmentPruners.size(), 1);
-    assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+    assertEquals(segmentPruners.size(), 0);
 
     // Time range pruner should be returned
     when(validationConfig.getTimeColumnName()).thenReturn(TIME_COLUMN);
     segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
-    assertEquals(segmentPruners.size(), 2);
-    assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
-    assertTrue(segmentPruners.get(1) instanceof TimeSegmentPruner);
+    assertEquals(segmentPruners.size(), 1);
+    assertTrue(segmentPruners.get(0) instanceof TimeSegmentPruner);
   }

Review comment:
       some test to check that EmptySegmentPruner is added as expected if 1. already in table config 2. because streaming table with isEmptySegment?

##########
File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerFactory.java
##########
@@ -45,30 +50,41 @@ private SegmentPrunerFactory() {
 
   public static List<SegmentPruner> getSegmentPruners(TableConfig tableConfig,
       ZkHelixPropertyStore<ZNRecord> propertyStore) {
-    RoutingConfig routingConfig = tableConfig.getRoutingConfig();
     List<SegmentPruner> segmentPruners = new ArrayList<>();
-    // Always prune out empty segments first
-    segmentPruners.add(new EmptySegmentPruner(tableConfig, propertyStore));
+    boolean isKinesisEnabled = isKinesisEnabled(tableConfig);

Review comment:
       this is leaking immensely specific information into the broker about Kinesis and the behavior of empty segments. Plus, adding kinesis plugin dependency in pinot-broker is not the best.. 
   How about one of these options:
   1. Add validations to TableConfigUtils.validate, to check that a kinesis stream table has this pruner added (or if there's any logic in that path which decorates the table config)
   2. Move this mthod `isKinesisEnabled` to `TableConfigUtils` ad rename it as `needsEmptySegmentPruner`. Part of that, check if routingTypes already has EmptySegmentPruner, if not check if kinesis. Possibly even add "needsEmptySegmetPruner" to StreamConfig
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org