[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8828: Add controller API for reload segment task status

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on code in PR #8828:
URL: https://github.com/apache/pinot/pull/8828#discussion_r898590984


##########
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java:
##########
@@ -110,6 +115,10 @@ public static String constructPropertyStorePathForInstancePartitions(String inst
     return StringUtil.join("/", PROPERTYSTORE_INSTANCE_PARTITIONS_PREFIX, instancePartitionsName);
   }
 
+  public static String constructPropertyStorePathForTaskResource(String resourceName) {

Review Comment:
   ```suggestion
     public static String constructPropertyStorePathForTask(String resourceName) {
   ```



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -563,16 +646,26 @@ public SuccessResponse reloadAllSegments(
     if (forceDownload && (tableTypeFromTableName == null && tableTypeFromRequest == null)) {
       tableTypeFromRequest = TableType.OFFLINE;
     }
-    List<String> tableNamesWithType = ResourceUtils
-        .getExistingTableNamesWithType(_pinotHelixResourceManager, tableName, tableTypeFromRequest, LOGGER);
-    Map<String, Integer> numMessagesSentPerTable = new LinkedHashMap<>();
+    List<String> tableNamesWithType =
+        ResourceUtils.getExistingTableNamesWithType(_pinotHelixResourceManager, tableName, tableTypeFromRequest,
+            LOGGER);
+    Map<String, Pair<Integer, String>> perTableMsgData = new LinkedHashMap<>();
     for (String tableNameWithType : tableNamesWithType) {
-      int numMsgSent = _pinotHelixResourceManager.reloadAllSegments(tableNameWithType, forceDownload);
-      numMessagesSentPerTable.put(tableNameWithType, numMsgSent);
+      Pair<Integer, String> msgInfo = _pinotHelixResourceManager.reloadAllSegments(tableNameWithType, forceDownload);
+      perTableMsgData.put(tableNameWithType, msgInfo);
+      // Store in ZK
+      try {
+        _pinotHelixResourceManager.addNewReloadAllSegmentsTask(tableNameWithType, msgInfo.getRight(),
+            msgInfo.getLeft());
+      } catch (Exception e) {
+        LOGGER.error("Failed to store task meta in zookepper ", e);
+      }
     }
-    return new SuccessResponse("Sent " + numMessagesSentPerTable + " reload messages");
+    return new SuccessResponse("Sent " + perTableMsgData + " reload messages");
   }
 
+

Review Comment:
   (nit) revert the empty lines



##########
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java:
##########
@@ -98,6 +99,10 @@ public static InstanceZKMetadata getInstanceZKMetadata(ZkHelixPropertyStore<ZNRe
     return new InstanceZKMetadata(znRecord);
   }
 
+  public static String constructPropertyStorePathForTask(String resourceName, String taskId) {

Review Comment:
   Suggest not keeping per-task level ZK metadata. It might create too many ZNode, and can be easily left over if not handled properly



##########
pinot-common/src/main/java/org/apache/pinot/common/messages/SegmentReloadMessage.java:
##########
@@ -35,6 +35,9 @@ public class SegmentReloadMessage extends Message {
 
   private static final String FORCE_DOWNLOAD_KEY = "forceDownload";
 
+  // todo (saurabh) : getMsgId() returns different id on the server side than the one being set at controller. Check

Review Comment:
   Any update on this?



##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentReloadTaskStatusCache.java:
##########
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.server.starter.helix;
+
+import com.google.common.cache.CacheBuilder;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.codehaus.jackson.annotate.JsonIgnore;
+
+
+public class SegmentReloadTaskStatusCache {
+
+  public static class SegmentReloadStatusValue {
+    private final long _totalSegmentCount;
+    private final AtomicLong _successCount;
+
+    SegmentReloadStatusValue(long totalSegmentCount) {
+      _totalSegmentCount = totalSegmentCount;
+      _successCount = new AtomicLong(0);
+    }
+
+    @JsonIgnore
+    public void incrementSuccess() {
+      _successCount.addAndGet(1);
+    }
+
+    public long getTotalSegmentCount() {
+      return _totalSegmentCount;
+    }
+
+    public long getSuccessCount() {
+      return _successCount.get();
+    }
+  }
+
+  private SegmentReloadTaskStatusCache() {
+  }
+
+  private static final ConcurrentMap<String, SegmentReloadStatusValue> SEGMENT_RELOAD_STATUS_MAP =
+      CacheBuilder.newBuilder()

Review Comment:
   Will this create a LRU cache?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org