You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Copilot (via GitHub)" <gi...@apache.org> on 2026/03/18 20:55:39 UTC

Re: [PR] Add time-range segment reload API [pinot]

Copilot commented on code in PR #17627:
URL: https://github.com/apache/pinot/pull/17627#discussion_r2956207712


##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerReloadJobTest.java:
##########
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.helix.core.controllerjob.ControllerJobTypes;
+import org.apache.pinot.spi.controller.ControllerJobType;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+public class PinotHelixResourceManagerReloadJobTest {
+
+  private static class CapturingHelixResourceManager extends PinotHelixResourceManager {
+    private String _jobId;
+    private Map<String, String> _jobMetadata;
+    private ControllerJobType _jobType;
+
+    CapturingHelixResourceManager() {
+      super(new ControllerConf());
+    }
+
+    @Override
+    public boolean addControllerJobToZK(String jobId, Map<String, String> jobMetadata, ControllerJobType jobType) {
+      _jobId = jobId;
+      _jobMetadata = new HashMap<>(jobMetadata);
+      _jobType = jobType;
+      return true;
+    }
+  }
+
+  @Test
+  public void testAddNewReloadSegmentJobCapturesMetadataWithInstance() {
+    CapturingHelixResourceManager manager = new CapturingHelixResourceManager();
+
+    boolean result =
+        manager.addNewReloadSegmentJob("table_OFFLINE", "seg_1,seg_2", "server1", "job-1", 123L, 4);
+
+    assertTrue(result);
+    assertEquals(manager._jobId, "job-1");
+    assertEquals(manager._jobType, ControllerJobTypes.RELOAD_SEGMENT);
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.JOB_ID), "job-1");
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.TABLE_NAME_WITH_TYPE), "table_OFFLINE");
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.JOB_TYPE),
+        ControllerJobTypes.RELOAD_SEGMENT.name());
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS), "123");
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.MESSAGE_COUNT), "4");
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.SEGMENT_RELOAD_JOB_SEGMENT_NAME),
+        "seg_1,seg_2");
+    assertEquals(manager._jobMetadata.get(CommonConstants.ControllerJob.SEGMENT_RELOAD_JOB_INSTANCE_NAME), "server1");

Review Comment:
   In this test, the segment list string is comma-separated ("seg_1,seg_2"), but controller reload job metadata and status parsing split multiple segments using SegmentNameUtils.SEGMENT_NAME_SEPARATOR (currently '|'). Using ',' here can encode an invalid/unsupported format and makes the test inconsistent with the rest of the reload flow. Prefer constructing the segmentNames string using SegmentNameUtils.SEGMENT_NAME_SEPARATOR (and updating the expected metadata accordingly).



##########
pinot-controller/src/main/java/org/apache/pinot/controller/services/PinotTableReloadStatusReporter.java:
##########
@@ -268,14 +268,28 @@ Map<String, List<String>> getServerToSegments(String tableNameWithType, @Nullabl
     if (instanceName != null) {
       return Map.of(instanceName, segmentNames);
     }
-    // If instance is null, then either one or all segments are being reloaded via current segment reload restful APIs.
-    // And the if-check at the beginning of this method has handled the case of reloading all segments. So here we
-    // expect only one segment name.
-    checkState(segmentNames.size() == 1, "Only one segment is expected but got: %s", segmentNames);
+    if (segmentNames.size() == 1) {
+      Map<String, List<String>> serverToSegments = new HashMap<>();
+      Set<String> servers = _pinotHelixResourceManager.getServers(tableNameWithType, segmentNamesString);
+      for (String server : servers) {
+        serverToSegments.put(server, Collections.singletonList(segmentNamesString));
+      }
+      return serverToSegments;
+    }
+    Set<String> targetSegments = new HashSet<>(segmentNames);
     Map<String, List<String>> serverToSegments = new HashMap<>();
-    Set<String> servers = _pinotHelixResourceManager.getServers(tableNameWithType, segmentNamesString);
-    for (String server : servers) {
-      serverToSegments.put(server, Collections.singletonList(segmentNamesString));
+    Map<String, List<String>> serverToAllSegments =
+        _pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType, null, true);
+    for (Map.Entry<String, List<String>> entry : serverToAllSegments.entrySet()) {
+      List<String> filteredSegments = new ArrayList<>();
+      for (String segmentName : entry.getValue()) {
+        if (targetSegments.contains(segmentName)) {
+          filteredSegments.add(segmentName);
+        }
+      }
+      if (!filteredSegments.isEmpty()) {
+        serverToSegments.put(entry.getKey(), filteredSegments);
+      }

Review Comment:
   When segmentNamesString contains multiple segments and instanceName is null, this implementation calls getServerToSegmentsMap(...) for the whole table and then filters. That requires iterating the entire ideal state (all segments) even if only a small subset of segments is being reloaded, which can be expensive for large tables. Consider adding/using a HelixResourceManager helper that maps servers for a provided segment set (single ideal-state fetch + only iterate target segments), or otherwise avoid building the full server-to-all-segments map here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org