Re: [PR] making nonLeaderForTables exhaustive [pinot]

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #12345:
URL: https://github.com/apache/pinot/pull/12345#discussion_r1474102380


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java:
##########
@@ -63,29 +69,25 @@ protected final void runTask(Properties periodicTaskProperties) {
       // Check if we have a specific table against which this task needs to be run.
       String propTableNameWithType = (String) periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME);
       // Process the tables that are managed by this controller
-      List<String> tablesToProcess = new ArrayList<>();
-      List<String> nonLeaderForTables = new ArrayList<>();
-      if (propTableNameWithType == null) {
-        // Table name is not available, so task should run on all tables for which this controller is the lead.
-        for (String tableNameWithType : _pinotHelixResourceManager.getAllTables()) {
-          if (_leadControllerManager.isLeaderForTable(tableNameWithType)) {
-            tablesToProcess.add(tableNameWithType);
-          } else {
-            nonLeaderForTables.add(tableNameWithType);
-          }
-        }
-      } else {
-        // Table name is available, so task should run only on the specified table.
-        if (_leadControllerManager.isLeaderForTable(propTableNameWithType)) {
-          tablesToProcess.add(propTableNameWithType);
-        }
-      }
+      Set<String> allTables = propTableNameWithType == null
+          ? new HashSet<>(_pinotHelixResourceManager.getAllTables())
+          : Collections.singleton(propTableNameWithType);
+
+      Set<String> currentLeaderOfTables = allTables.stream()
+          .filter(_leadControllerManager::isLeaderForTable)
+          .collect(Collectors.toSet());
 
-      if (!tablesToProcess.isEmpty()) {
-        processTables(tablesToProcess, periodicTaskProperties);
+      if (!currentLeaderOfTables.isEmpty()) {
+        processTables(new ArrayList<>(currentLeaderOfTables), periodicTaskProperties);
       }
+
+      Sets.SetView<String> allKnownTables = Sets.union(_prevLeaderOfTables, allTables);
+      Set<String> nonLeaderForTables = Sets.difference(allKnownTables, currentLeaderOfTables);

Review Comment:
   Javadoc of `nonLeaderCleanup` says: `Can be overridden to perform cleanups for tables that the current controller isn't the leader.`
   
   Where the key part is `tables that the current controller isn't the leader`. That includes tables the current controller is not leader and wasn't the leader in the previous iteration. The expression `_prevLeaderOfTables - currentLeaderOfTables` does only return the tables we ended up not being leader since previous iteration.
   
   I guess it is a subtle difference and probably the current code only cares about the tables we are no longer leader of. I can apply that change, but I wanted to be formal on the semantics in case there are third party extensions that are actually based on the javadoc contract.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org