[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #4714: Remove the segment missing error message when acquiring segments

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on a change in pull request #4714: Remove the segment missing error message when acquiring segments
URL: https://github.com/apache/incubator-pinot/pull/4714#discussion_r336120908
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
 ##########
 @@ -126,8 +126,25 @@ public DataTable processQuery(ServerQueryRequest queryRequest, ExecutorService e
 
     TableDataManager tableDataManager = _instanceDataManager.getTableDataManager(tableNameWithType);
     Preconditions.checkState(tableDataManager != null, "Failed to find data manager for table: " + tableNameWithType);
-    List<SegmentDataManager> segmentDataManagers = tableDataManager.acquireSegments(queryRequest.getSegmentsToQuery());
-    int numSegmentsQueried = segmentDataManagers.size();
+
+    List<String> segmentsToQuery = queryRequest.getSegmentsToQuery();
+    List<SegmentDataManager> segmentDataManagers = tableDataManager.acquireSegments(segmentsToQuery);
+
+    // When segment is removed from the IdealState:
+    // 1. Controller schedules a state transition to server to turn segment OFFLINE
+    // 2. Server gets the state transition, removes the segment data manager and update its CurrentState
+    // 3. Controller gathers the CurrentState and update the ExternalView
+    // 4. Broker watches ExternalView change and updates the routing table to stop querying the segment
+    //
+    // After step 2 but before step 4, segment will be missing on server side
+    // TODO: Change broker to watch both IdealState and ExternalView to not query the removed segments
+    int numSegmentsQueried = segmentsToQuery.size();
+    int numSegmentsAcquired = segmentDataManagers.size();
+    if (numSegmentsQueried > numSegmentsAcquired) {
+      _serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS,
 
 Review comment:
   This is per query basis error message, which can flood the log

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org