You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "abhishekbafna (via GitHub)" <gi...@apache.org> on 2026/03/09 16:46:22 UTC

[PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

abhishekbafna opened a new pull request, #17842:
URL: https://github.com/apache/pinot/pull/17842

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2929957652


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java:
##########
@@ -535,23 +537,43 @@ public static Set<InstanceConfig> getBrokerInstanceConfigsForTenant(List<Instanc
 
   public static Set<String> getTablesForBrokerTag(HelixManager helixManager, String brokerTag) {
     Set<String> tablesForBrokerTag = new HashSet<>();
-    List<TableConfig> tableConfigs = ZKMetadataProvider.getAllTableConfigs(helixManager.getHelixPropertyStore());
+    ZkHelixPropertyStore<ZNRecord> propertyStore = helixManager.getHelixPropertyStore();
+    List<TableConfig> tableConfigs = ZKMetadataProvider.getAllTableConfigs(propertyStore);
     for (TableConfig tableConfig : tableConfigs) {
       if (TagNameUtils.getBrokerTagForTenant(tableConfig.getTenantConfig().getBroker()).equals(brokerTag)) {
         tablesForBrokerTag.add(tableConfig.getTableName());
       }
     }
+    // Include logical tables that use this broker tenant (broker resource partition name = logical table name)
+    for (LogicalTableConfig logicalTableConfig : ZKMetadataProvider.getAllLogicalTableConfigs(propertyStore)) {
+      String logicalBrokerTag =
+          TagNameUtils.getBrokerTagForTenant(logicalTableConfig.getBrokerTenant() != null

Review Comment:
   No. It would not return null.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2929525392


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   Why? Running on the non-leader would get the update from all the controllers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2941918050


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager

Review Comment:
   Check this out: https://docs.google.com/document/d/1iS0wtG_V2-W9sQsKkziMjTljT5qKAH_hhsksXr1Tvys/edit?tab=t.0#bookmark=id.iek6q8y68huw
   
   In the broker resource we are keeping a mapping of table to broker instance, same for the physical tables. I believe, query engine uses the broker instance from the table name key in the broker resources.
   
   I can not pin point this in the code at the moment, let me know if that is required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2909316057


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, logicalTableConfig.getBrokerTenant());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);

Review Comment:
   PR description says `processTable` differentiates physical vs logical partitions using `TableNameBuilder.isTableResource(...)`, but the implementation actually differentiates by config lookup (`getTableConfig(...) != null` else `getLogicalTableConfig(...)`). Please align the PR description with the implemented behavior (or update the code if the name-based check is required) to avoid confusion for future maintainers.



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java:
##########
@@ -422,6 +422,49 @@ public void testBrokerResourceValidationManager() {
     }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager");
   }
 
+  /**
+   * Verifies that BrokerResourceValidationManager also repairs broker resource for a logical table
+   * when a new broker is added (Issue #15751).
+   */
+  @Test
+  public void testBrokerResourceValidationManagerRepairsLogicalTable()
+      throws IOException {
+    // Add logical table (same broker tenant as physical table)
+    Schema logicalTableSchema = createSchema();
+    logicalTableSchema.setSchemaName(getLogicalTableName());
+    addSchema(logicalTableSchema);
+    createLogicalTable();
+
+    String helixClusterName = getHelixClusterName();
+    String logicalTableName = getLogicalTableName();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName), "Broker resource should have logical table");
+
+    // Add a new broker so logical table partition is out of sync
+    String brokerId = "Broker_localhost_5678";
+    InstanceConfig instanceConfig = InstanceConfig.toInstanceConfig(brokerId);
+    instanceConfig.addTag(TagNameUtils.getBrokerTagForTenant(TENANT_NAME));
+    _helixAdmin.addInstance(helixClusterName, instanceConfig);
+    Set<String> brokersAfterAdd = _helixResourceManager.getAllInstancesForBrokerTenant(TENANT_NAME);
+    assertTrue(brokersAfterAdd.contains(brokerId));
+
+    // Wait for BrokerResourceValidationManager to repair both physical and logical table partitions
+    String tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());
+    TestUtils.waitForCondition(aVoid -> {
+      IdealState is = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+      if (is == null) {
+        return false;
+      }
+      return is.getInstanceSet(tableNameWithType).equals(brokersAfterAdd)
+          && is.getInstanceSet(logicalTableName).equals(brokersAfterAdd);
+    }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager to repair logical table partition");
+
+    // Cleanup: drop broker and logical table
+    _helixAdmin.dropInstance(helixClusterName, instanceConfig);
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+  }

Review Comment:
   Test adds a logical-table schema via `addSchema(logicalTableSchema)` but cleanup only drops the broker instance + logical table config. To keep the integration test cluster clean (and consistent with other tests in this class that call `deleteSchema(...)`), also delete the logical table schema during cleanup; otherwise the schema ZNode persists and can accumulate across runs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2946134531


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager

Review Comment:
   no issues. thanks. this helps for context. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2909300891


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +49,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());
+    return combined;
+  }
+
   @Override

Review Comment:
   The logical table should have leader selection and it should be processed by multiple controller at the same time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2929529201


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager

Review Comment:
   The brokerTenant in the logical table is used to identity the broker node which should be used for the querying. For a logical table, the broker tenant of the physical table can be different and it would not be used, as much I remember. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2941820878


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   The original comment sounded more like a suggestion than a info question.
   
   We override `getTablesToProcess()` to include the logical table. The `runTask()` method in the `ControllerPeriodicTask` filter the tables by checking leader controller before starting the task processing.
   
   ```
   protected boolean shouldProcessTable(String tableNameWithType) {
       return _leadControllerManager.isLeaderForTable(tableNameWithType);
     }
   ```
   
   This is part of the PR only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2946197722


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   It is just a hash function on the table name. There is not leader assignment or anything. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2938171887


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   What do you mean ? 
   
   My question was - Is it safe to run this task on non-leaders for logical tables ? For physical tables it only runs on the table's leader. How are we ensuring for logical tables also it runs on leaders only ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2916160166


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   Can / Should this task run on non-leaders for logical tables ? 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -748,6 +748,29 @@ public List<String> getAllTables() {
     return getAllResources().stream().filter(TableNameBuilder::isTableResource).collect(Collectors.toList());
   }
 
+  /**
+   * Returns partition names in broker resource ideal state that are logical tables (have logical table config
+   * and no physical table config). Used by broker resource validation to repair logical table broker assignments.
+   * Identifies by config presence rather than name to avoid misclassifying logical tables named with
+   * _OFFLINE/_REALTIME.
+   */
+  public List<String> getBrokerResourceLogicalTablePartitions() {

Review Comment:
   This looks sub optimal and probably edge case prone
   1. We are getting table list from ideal state which might not be updated and is the problem the BrokerResourceValidationManager is trying to fix
   2. Doing a ZK read for every table config is also not ideal
   
   Can use getAllLogicalTableNames() instead ? 



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java:
##########
@@ -373,6 +448,56 @@ public void testRebuildBrokerResourceFromHelixTags()
     resetBrokerTags();
   }
 
+  /**
+   * Verifies that rebuildBrokerResourceFromHelixTags succeeds for a logical table name and
+   * updates the broker resource ideal state correctly (Issue #15751).
+   */
+  @Test
+  public void testRebuildBrokerResourceFromHelixTagsForLogicalTable()
+      throws Exception {
+    // Setup: physical tables (offline + realtime) via manager with test tenants, then logical table
+    addDummySchema(RAW_TABLE_NAME);
+    TableConfig offlineTableConfig =
+        new TableConfigBuilder(TableType.OFFLINE).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME).build();
+    waitForEVToDisappear(offlineTableConfig.getTableName());
+    _helixResourceManager.addTable(offlineTableConfig);
+    waitForEVToDisappear(REALTIME_TABLE_NAME);
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+    List<String> physicalTableNamesWithType = List.of(OFFLINE_TABLE_NAME, REALTIME_TABLE_NAME);
+    String logicalTableName = "test_logical_table_rebuild";
+    addDummySchema(logicalTableName);
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType, BROKER_TENANT_NAME);
+    addLogicalTableConfig(logicalTableConfig);
+
+    IdealState idealState = _helixAdmin.getResourceIdealState(_clusterName, Helix.BROKER_RESOURCE_INSTANCE);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName));
+    int initialBrokerCount = idealState.getInstanceSet(logicalTableName).size();
+
+    // Rebuild for logical table name should succeed (not return 400)
+    PinotResourceManagerResponse response =

Review Comment:
   Here too, the rebuild is not doing anything and the test will pass even if rebuild is not called right ? Because, there is no failure in the above path, and that's why the initial broker count remains the same in the assertion in line 490. 
   
   Either we should inject a failure when a new broker is added. Assert that initially the ideal state is not updated. And then call rebuild and then validate that the ideal state is updated. 



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java:
##########
@@ -326,13 +327,87 @@ public void testUpdateBrokerResource()
     resetBrokerTags();
   }
 
+  /**
+   * Verifies that when a new broker is added with the same tenant as a logical table,
+   * the broker resource ideal state is updated for the logical table partition (Issue #15751).
+   */
+  @Test
+  public void testUpdateBrokerResourceWithLogicalTable()
+      throws Exception {
+    untagBrokers();
+    Tenant brokerTenant = new Tenant(TenantRole.BROKER, BROKER_TENANT_NAME, 2, 0, 0);
+    _helixResourceManager.createBrokerTenant(brokerTenant);
+
+    String brokerTag = TagNameUtils.getBrokerTagForTenant(BROKER_TENANT_NAME);
+    List<InstanceConfig> instanceConfigs = HelixHelper.getInstanceConfigs(_helixManager);
+    List<String> taggedBrokers = HelixHelper.getInstancesWithTag(instanceConfigs, brokerTag);
+    assertEquals(taggedBrokers.size(), 2);
+
+    // Add physical tables (offline + realtime) via manager with test tenants, then logical table
+    addDummySchema(RAW_TABLE_NAME);
+    TableConfig offlineTableConfig =
+        new TableConfigBuilder(TableType.OFFLINE).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME).build();
+    waitForEVToDisappear(offlineTableConfig.getTableName());
+    _helixResourceManager.addTable(offlineTableConfig);
+    waitForEVToDisappear(REALTIME_TABLE_NAME);
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+    List<String> physicalTableNamesWithType = List.of(OFFLINE_TABLE_NAME, REALTIME_TABLE_NAME);
+    String logicalTableName = "test_logical_table";
+    addDummySchema(logicalTableName);
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType, BROKER_TENANT_NAME);
+    _helixResourceManager.addLogicalTableConfig(logicalTableConfig);
+
+    IdealState brokerResource = HelixHelper.getBrokerIdealStates(_helixAdmin, _clusterName);
+    assertTrue(brokerResource.getPartitionSet().contains(logicalTableName));
+    checkBrokerResourceForPartition(logicalTableName, taggedBrokers);
+
+    // Add a new broker instance with same tenant
+    Instance newBrokerInstance =
+        new Instance("localhost", 3, InstanceType.BROKER, Collections.singletonList(brokerTag), null, 0, 0, 0, 0,
+            false);
+    assertTrue(_helixResourceManager.addInstance(newBrokerInstance, true).isSuccessful());
+    String newBrokerId = InstanceUtils.getHelixInstanceId(newBrokerInstance);
+    List<String> taggedBrokersAfterAdd = new ArrayList<>(taggedBrokers);
+    taggedBrokersAfterAdd.add(newBrokerId);
+
+    // Logical table partition should include the new broker. If broker add path did not update it yet,
+    // rebuildBrokerResourceFromHelixTags (e.g. from BrokerResourceValidationManager) repairs it.
+    PinotResourceManagerResponse rebuildResponse =
+        _helixResourceManager.rebuildBrokerResourceFromHelixTags(logicalTableName);

Review Comment:
   We shouldn't be calling rebuild here, right? Rebuild is used only if there is some failure in the add broker path. 
   
   In this particular happy path test, we should not rebuild but verify that adding a new broker sets the resource correctly without requiring an explicit rebuild. 



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java:
##########
@@ -422,6 +422,50 @@ public void testBrokerResourceValidationManager() {
     }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager");
   }
 
+  /**
+   * Verifies that BrokerResourceValidationManager also repairs broker resource for a logical table
+   * when a new broker is added (Issue #15751).
+   */
+  @Test
+  public void testBrokerResourceValidationManagerRepairsLogicalTable()
+      throws IOException {
+    // Add logical table (same broker tenant as physical table)
+    Schema logicalTableSchema = createSchema();
+    logicalTableSchema.setSchemaName(getLogicalTableName());
+    addSchema(logicalTableSchema);
+    createLogicalTable();
+
+    String helixClusterName = getHelixClusterName();
+    String logicalTableName = getLogicalTableName();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName), "Broker resource should have logical table");
+
+    // Add a new broker so logical table partition is out of sync
+    String brokerId = "Broker_localhost_5678";
+    InstanceConfig instanceConfig = InstanceConfig.toInstanceConfig(brokerId);
+    instanceConfig.addTag(TagNameUtils.getBrokerTagForTenant(TENANT_NAME));
+    _helixAdmin.addInstance(helixClusterName, instanceConfig);
+    Set<String> brokersAfterAdd = _helixResourceManager.getAllInstancesForBrokerTenant(TENANT_NAME);
+    assertTrue(brokersAfterAdd.contains(brokerId));

Review Comment:
   We should assert that right now the idealState doesn't contain the new broker. 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1187,42 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      assert idealState != null;
+      for (String partitionName : idealState.getPartitionSet()) {
+        String brokerTag = resolveBrokerTagForBrokerResourcePartition(partitionName);
+        if (brokerTag == null) {
+          continue;
+        }
+        if (brokerTag.equals(brokerTenantTag)) {
+          idealState.setPartitionState(partitionName, instanceName, BrokerResourceStateModel.ONLINE);
+        }
       }
+      return idealState;
+    }, DEFAULT_RETRY_POLICY);
+  }
+
+  /**
+   * Resolves the broker tag for a partition in the broker resource. Tries physical table config first,
+   * then logical table config, so both are handled regardless of partition naming (e.g. a logical table
+   * name ending with _OFFLINE would otherwise be misclassified as physical and skipped).
+   *
+   * @param partitionName partition name in broker ideal state (physical table name with type or logical table name)
+   * @return broker tag for the partition, or null if the partition cannot be resolved (unknown or missing config)
+   */
+  private String resolveBrokerTagForBrokerResourcePartition(String partitionName) {
+    TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, partitionName);
+    if (tableConfig != null) {
+      return TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
     }
-    _helixAdmin.setResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE, tableIdealState);
+    LogicalTableConfig logicalTableConfig = ZKMetadataProvider.getLogicalTableConfig(_propertyStore, partitionName);
+    if (logicalTableConfig != null) {
+      return TagNameUtils.getBrokerTagForTenant(logicalTableConfig.getBrokerTenant());
+    }
+    LOGGER.debug("Skipping partition {} in broker resource: no table config or logical table config found",

Review Comment:
   This should be a warn rather than debug because its unexpected. previously we were throwing an exception so either we continue to throw an exception or update to warning



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager

Review Comment:
   Why does logical table config have a broker tenant? What happens if a physical table's broker tenant is different from a logical table's broker tenant? 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1187,42 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      assert idealState != null;

Review Comment:
   Instead of assert use Preconditions.checkNotNull as assert isn't production safe. 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1187,42 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      assert idealState != null;
+      for (String partitionName : idealState.getPartitionSet()) {
+        String brokerTag = resolveBrokerTagForBrokerResourcePartition(partitionName);
+        if (brokerTag == null) {
+          continue;
+        }
+        if (brokerTag.equals(brokerTenantTag)) {
+          idealState.setPartitionState(partitionName, instanceName, BrokerResourceStateModel.ONLINE);
+        }
       }
+      return idealState;
+    }, DEFAULT_RETRY_POLICY);
+  }
+
+  /**
+   * Resolves the broker tag for a partition in the broker resource. Tries physical table config first,
+   * then logical table config, so both are handled regardless of partition naming (e.g. a logical table
+   * name ending with _OFFLINE would otherwise be misclassified as physical and skipped).
+   *
+   * @param partitionName partition name in broker ideal state (physical table name with type or logical table name)
+   * @return broker tag for the partition, or null if the partition cannot be resolved (unknown or missing config)
+   */
+  private String resolveBrokerTagForBrokerResourcePartition(String partitionName) {

Review Comment:
   probably method name should be resolveBrokerTagForTable(String tableName)
   
   broker resource partition / partition name is too confusing of what the method is actually doing



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java:
##########
@@ -535,23 +537,43 @@ public static Set<InstanceConfig> getBrokerInstanceConfigsForTenant(List<Instanc
 
   public static Set<String> getTablesForBrokerTag(HelixManager helixManager, String brokerTag) {
     Set<String> tablesForBrokerTag = new HashSet<>();
-    List<TableConfig> tableConfigs = ZKMetadataProvider.getAllTableConfigs(helixManager.getHelixPropertyStore());
+    ZkHelixPropertyStore<ZNRecord> propertyStore = helixManager.getHelixPropertyStore();
+    List<TableConfig> tableConfigs = ZKMetadataProvider.getAllTableConfigs(propertyStore);
     for (TableConfig tableConfig : tableConfigs) {
       if (TagNameUtils.getBrokerTagForTenant(tableConfig.getTenantConfig().getBroker()).equals(brokerTag)) {
         tablesForBrokerTag.add(tableConfig.getTableName());
       }
     }
+    // Include logical tables that use this broker tenant (broker resource partition name = logical table name)
+    for (LogicalTableConfig logicalTableConfig : ZKMetadataProvider.getAllLogicalTableConfigs(propertyStore)) {
+      String logicalBrokerTag =
+          TagNameUtils.getBrokerTagForTenant(logicalTableConfig.getBrokerTenant() != null

Review Comment:
   can getBrokerTenant() return null ? If yes then other places in the PR that are calling getBrokerTenant() aren't handling null. 
   
   Ideally getBrokerTenant() should return DEFAULT if its null so that client code is simple. Can we make this change instead ? 



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java:
##########
@@ -535,23 +537,43 @@ public static Set<InstanceConfig> getBrokerInstanceConfigsForTenant(List<Instanc
 
   public static Set<String> getTablesForBrokerTag(HelixManager helixManager, String brokerTag) {

Review Comment:
   can this method call the below method with List.of(brokerTag). It will avoid duplicate code



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java:
##########
@@ -422,6 +422,50 @@ public void testBrokerResourceValidationManager() {
     }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager");
   }
 
+  /**
+   * Verifies that BrokerResourceValidationManager also repairs broker resource for a logical table
+   * when a new broker is added (Issue #15751).
+   */
+  @Test
+  public void testBrokerResourceValidationManagerRepairsLogicalTable()
+      throws IOException {
+    // Add logical table (same broker tenant as physical table)
+    Schema logicalTableSchema = createSchema();
+    logicalTableSchema.setSchemaName(getLogicalTableName());
+    addSchema(logicalTableSchema);
+    createLogicalTable();
+
+    String helixClusterName = getHelixClusterName();
+    String logicalTableName = getLogicalTableName();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName), "Broker resource should have logical table");
+
+    // Add a new broker so logical table partition is out of sync
+    String brokerId = "Broker_localhost_5678";
+    InstanceConfig instanceConfig = InstanceConfig.toInstanceConfig(brokerId);
+    instanceConfig.addTag(TagNameUtils.getBrokerTagForTenant(TENANT_NAME));
+    _helixAdmin.addInstance(helixClusterName, instanceConfig);
+    Set<String> brokersAfterAdd = _helixResourceManager.getAllInstancesForBrokerTenant(TENANT_NAME);
+    assertTrue(brokersAfterAdd.contains(brokerId));
+
+    // Wait for BrokerResourceValidationManager to repair both physical and logical table partitions
+    String tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());
+    TestUtils.waitForCondition(aVoid -> {
+      IdealState is = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+      if (is == null) {
+        return false;
+      }
+      return is.getInstanceSet(tableNameWithType).equals(brokersAfterAdd)
+          && is.getInstanceSet(logicalTableName).equals(brokersAfterAdd);
+    }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager to repair logical table partition");

Review Comment:
   Can we reduce the timeout from 10 minutes to 1 minute or something ? How frequently is the task running? Can we increase the frequency of the task in this test? And reduce the timeout? 



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java:
##########
@@ -373,6 +448,56 @@ public void testRebuildBrokerResourceFromHelixTags()
     resetBrokerTags();
   }
 
+  /**
+   * Verifies that rebuildBrokerResourceFromHelixTags succeeds for a logical table name and
+   * updates the broker resource ideal state correctly (Issue #15751).
+   */
+  @Test
+  public void testRebuildBrokerResourceFromHelixTagsForLogicalTable()
+      throws Exception {
+    // Setup: physical tables (offline + realtime) via manager with test tenants, then logical table
+    addDummySchema(RAW_TABLE_NAME);
+    TableConfig offlineTableConfig =
+        new TableConfigBuilder(TableType.OFFLINE).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME).build();
+    waitForEVToDisappear(offlineTableConfig.getTableName());
+    _helixResourceManager.addTable(offlineTableConfig);
+    waitForEVToDisappear(REALTIME_TABLE_NAME);
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(RAW_TABLE_NAME).setBrokerTenant(BROKER_TENANT_NAME)
+            .setServerTenant(SERVER_TENANT_NAME)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+    List<String> physicalTableNamesWithType = List.of(OFFLINE_TABLE_NAME, REALTIME_TABLE_NAME);
+    String logicalTableName = "test_logical_table_rebuild";
+    addDummySchema(logicalTableName);
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType, BROKER_TENANT_NAME);
+    addLogicalTableConfig(logicalTableConfig);
+
+    IdealState idealState = _helixAdmin.getResourceIdealState(_clusterName, Helix.BROKER_RESOURCE_INSTANCE);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName));
+    int initialBrokerCount = idealState.getInstanceSet(logicalTableName).size();
+
+    // Rebuild for logical table name should succeed (not return 400)
+    PinotResourceManagerResponse response =

Review Comment:
   I see testRebuildBrokerResourceWhenBrokerAddedForLogicalTable() is testing the rebuild properly. So we can remove this particular test right ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17842:
URL: https://github.com/apache/pinot/pull/17842#issuecomment-4025898411

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17842?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `16 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.58%. Comparing base ([`23c9a59`](https://app.codecov.io/gh/apache/pinot/commit/23c9a594747a8f7f27f757deacb754936a460a3b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ea66138`](https://app.codecov.io/gh/apache/pinot/commit/ea66138d22915a6b2673c79070d7af201ae7490f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17842?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...g/apache/pinot/common/utils/helix/HelixHelper.java](https://app.codecov.io/gh/apache/pinot/pull/17842?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2Fhelix%2FHelixHelper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaGVsaXgvSGVsaXhIZWxwZXIuamF2YQ==) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17842?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (23c9a59) and HEAD (ea66138). Click for more details.
   > 
   > <details><summary>HEAD has 28 uploads less than BASE</summary>
   >
   >| Flag | BASE (23c9a59) | HEAD (ea66138) |
   >|------|------|------|
   >|java-21|4|1|
   >|unittests|4|2|
   >|temurin|9|2|
   >|java-11|5|1|
   >|unittests2|2|0|
   >|integration|5|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   >|integration2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #17842      +/-   ##
   ============================================
   - Coverage     63.24%   55.58%   -7.66%     
   + Complexity     1466      721     -745     
   ============================================
     Files          3190     2494     -696     
     Lines        191963   141233   -50730     
     Branches      29410    22535    -6875     
   ============================================
   - Hits         121400    78501   -42899     
   + Misses        61059    56077    -4982     
   + Partials       9504     6655    -2849     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.54% <0.00%> (-7.67%)` | :arrow_down: |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.53% <0.00%> (-7.68%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.58% <0.00%> (-7.66%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.58% <0.00%> (-7.66%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.58% <0.00%> (+<0.01%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17842/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17842?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2930898348


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1100,23 +1108,25 @@ private PinotResourceManagerResponse scaleUpBroker(Tenant tenant, String brokerT
     return PinotResourceManagerResponse.SUCCESS;
   }
 
-  public PinotResourceManagerResponse rebuildBrokerResourceFromHelixTags(String tableNameWithType)
-      throws Exception {
-    TableConfig tableConfig;
+  public PinotResourceManagerResponse rebuildBrokerResourceFromHelixTags(String tableNameWithType) {
+    Set<String> brokerInstances;
     try {
-      tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+      if (tableConfig != null) {
+        brokerInstances = getAllInstancesForBrokerTenant(tableConfig.getTenantConfig().getBroker());
+      } else {
+        LogicalTableConfig logicalTableConfig =
+            ZKMetadataProvider.getLogicalTableConfig(_propertyStore, tableNameWithType);
+        Preconditions.checkNotNull(logicalTableConfig, "No table config or logical table config found for %s",
+            tableNameWithType);
+        brokerInstances = getAllInstancesForBrokerTenant(logicalTableConfig.getBrokerTenant());
+      }
     } catch (Exception e) {
-      LOGGER.warn("Caught exception while getting table config for table {}", tableNameWithType, e);
+      LOGGER.warn("Caught exception while getting config for table {}", tableNameWithType, e);
       throw new InvalidTableConfigException(
-          "Failed to fetch broker tag for table " + tableNameWithType + " due to exception: " + e.getMessage());
+          "Failed to fetch broker config for table " + tableNameWithType + " due to exception: " + e.getMessage());
     }
-    if (tableConfig == null) {
-      LOGGER.warn("Table {} does not exist", tableNameWithType);
-      throw new InvalidConfigException(
-          "Invalid table configuration for table " + tableNameWithType + ". Table does not exist");
-    }
-    return rebuildBrokerResource(tableNameWithType,
-        getAllInstancesForBrokerTenant(tableConfig.getTenantConfig().getBroker()));
+    return rebuildBrokerResource(tableNameWithType, brokerInstances);
   }

Review Comment:
   For a non-existent `tableNameWithType`, the old behavior threw a targeted “table does not exist” style error; this new flow triggers `Preconditions.checkNotNull(...)`, which is caught and rethrown as `InvalidTableConfigException` with a generic “Failed to fetch broker config...” message. Consider explicitly handling the “both configs are null” case outside the catch to preserve clearer, stable error semantics (and avoid treating “not found” as an exception path).



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTables());
+    return combined;
+  }

Review Comment:
   The combined list can contain duplicates (e.g., if the logical list contains entries already present, or if the underlying methods return duplicates). Duplicates can cause repeated rebuild calls and noisy logs/metrics. Consider de-duplicating while preserving deterministic order (e.g., via a `LinkedHashSet` or `stream().distinct()`).



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java:
##########
@@ -64,18 +64,32 @@ public ControllerPeriodicTask(String taskName, long runFrequencyInSeconds, long
     _controllerMetrics = controllerMetrics;
   }
 
+  /**
+   * Returns the list of table names (with type) to consider for this task. Subclasses may override to add
+   * more names (e.g. logical table partitions). Default: single table from property or all physical tables.
+   */
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    String propTableNameWithType = (String) periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME);

Review Comment:
   `Properties` is string-keyed and typically accessed via `getProperty(...)`. Casting `periodicTaskProperties.get(...)` to `String` can throw `ClassCastException` if a non-String value is ever set (including in tests/utilities). Prefer `periodicTaskProperties.getProperty(PeriodicTask.PROPERTY_KEY_TABLE_NAME)` for safer, idiomatic access.
   



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1160,37 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      Preconditions.checkNotNull(idealState, "Broker ideal state must not be null");
+      for (String partitionName : idealState.getPartitionSet()) {
+        String brokerTag = resolveBrokerTagForTable(partitionName);
+        if (brokerTag.equals(brokerTenantTag)) {
+          idealState.setPartitionState(partitionName, instanceName, BrokerResourceStateModel.ONLINE);
+        }
       }
+      return idealState;
+    }, DEFAULT_RETRY_POLICY);

Review Comment:
   `resolveBrokerTagForTable(partitionName)` does ZK reads per partition, which can be expensive for large broker resources and adds load during broker add. Also, if one partition can’t be resolved, the exception will abort the entire IdealState update. Consider (1) caching `partitionName -> brokerTag` within the update lambda to avoid repeated ZK lookups, and (2) handling resolution failures per-partition (warn + continue) so one stale/orphaned partition doesn’t prevent updating the rest.



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java:
##########
@@ -422,6 +422,56 @@ public void testBrokerResourceValidationManager() {
     }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager");
   }
 
+  /**
+   * Verifies that BrokerResourceValidationManager also repairs broker resource for a logical table
+   * when a new broker is added (Issue #15751).
+   */
+  @Test
+  public void testBrokerResourceValidationManagerRepairsLogicalTable()
+      throws IOException {
+    // Add logical table (same broker tenant as physical table)
+    Schema logicalTableSchema = createSchema();
+    logicalTableSchema.setSchemaName(getLogicalTableName());
+    addSchema(logicalTableSchema);
+    createLogicalTable();
+
+    String helixClusterName = getHelixClusterName();
+    String logicalTableName = getLogicalTableName();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName), "Broker resource should have logical table");
+
+    // Add a new broker so logical table partition is out of sync
+    String brokerId = "Broker_localhost_5678";
+    InstanceConfig instanceConfig = InstanceConfig.toInstanceConfig(brokerId);
+    instanceConfig.addTag(TagNameUtils.getBrokerTagForTenant(TENANT_NAME));
+    _helixAdmin.addInstance(helixClusterName, instanceConfig);
+    Set<String> brokersAfterAdd = _helixResourceManager.getAllInstancesForBrokerTenant(TENANT_NAME);
+    assertTrue(brokersAfterAdd.contains(brokerId));
+
+    // Assert logical table partition does not yet contain the new broker (periodic task will repair it)
+    idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertFalse(idealState.getInstanceSet(logicalTableName).contains(brokerId),
+        "Logical table partition should not yet include the new broker before periodic task runs");
+
+    // Wait for BrokerResourceValidationManager to repair both physical and logical table partitions
+    String tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());
+    TestUtils.waitForCondition(aVoid -> {
+      IdealState is = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+      if (is == null) {
+        return false;
+      }
+      return is.getInstanceSet(tableNameWithType).equals(brokersAfterAdd)
+          && is.getInstanceSet(logicalTableName).equals(brokersAfterAdd);
+    }, 60_000L, "Timeout when waiting for BrokerResourceValidationManager to repair logical table partition");
+
+    // Cleanup: drop broker, logical table config, and logical table schema
+    _helixAdmin.dropInstance(helixClusterName, instanceConfig);
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+    deleteSchema(logicalTableName);

Review Comment:
   Cleanup isn’t in a `finally` block. If any assertion or the wait times out, the test can leak the broker instance and/or logical table config into subsequent tests, making the suite flaky. Wrap the broker/config creation portion in `try` and move cleanup to `finally` to guarantee cluster state is restored.
   



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java:
##########
@@ -422,6 +422,56 @@ public void testBrokerResourceValidationManager() {
     }, 600_000L, "Timeout when waiting for BrokerResourceValidationManager");
   }
 
+  /**
+   * Verifies that BrokerResourceValidationManager also repairs broker resource for a logical table
+   * when a new broker is added (Issue #15751).
+   */
+  @Test
+  public void testBrokerResourceValidationManagerRepairsLogicalTable()
+      throws IOException {
+    // Add logical table (same broker tenant as physical table)
+    Schema logicalTableSchema = createSchema();
+    logicalTableSchema.setSchemaName(getLogicalTableName());
+    addSchema(logicalTableSchema);
+    createLogicalTable();
+
+    String helixClusterName = getHelixClusterName();
+    String logicalTableName = getLogicalTableName();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertTrue(idealState.getPartitionSet().contains(logicalTableName), "Broker resource should have logical table");
+
+    // Add a new broker so logical table partition is out of sync
+    String brokerId = "Broker_localhost_5678";
+    InstanceConfig instanceConfig = InstanceConfig.toInstanceConfig(brokerId);
+    instanceConfig.addTag(TagNameUtils.getBrokerTagForTenant(TENANT_NAME));
+    _helixAdmin.addInstance(helixClusterName, instanceConfig);
+    Set<String> brokersAfterAdd = _helixResourceManager.getAllInstancesForBrokerTenant(TENANT_NAME);
+    assertTrue(brokersAfterAdd.contains(brokerId));
+
+    // Assert logical table partition does not yet contain the new broker (periodic task will repair it)
+    idealState = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+    assertNotNull(idealState);
+    assertFalse(idealState.getInstanceSet(logicalTableName).contains(brokerId),
+        "Logical table partition should not yet include the new broker before periodic task runs");
+
+    // Wait for BrokerResourceValidationManager to repair both physical and logical table partitions
+    String tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());
+    TestUtils.waitForCondition(aVoid -> {
+      IdealState is = HelixHelper.getBrokerIdealStates(_helixAdmin, helixClusterName);
+      if (is == null) {
+        return false;
+      }
+      return is.getInstanceSet(tableNameWithType).equals(brokersAfterAdd)
+          && is.getInstanceSet(logicalTableName).equals(brokersAfterAdd);
+    }, 60_000L, "Timeout when waiting for BrokerResourceValidationManager to repair logical table partition");
+
+    // Cleanup: drop broker, logical table config, and logical table schema
+    _helixAdmin.dropInstance(helixClusterName, instanceConfig);
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+    deleteSchema(logicalTableName);

Review Comment:
   Cleanup isn’t in a `finally` block. If any assertion or the wait times out, the test can leak the broker instance and/or logical table config into subsequent tests, making the suite flaky. Wrap the broker/config creation portion in `try` and move cleanup to `finally` to guarantee cluster state is restored.
   



##########
pinot-controller/src/test/java/org/apache/pinot/controller/validation/ValidationManagerStatelessTest.java:
##########
@@ -100,6 +104,62 @@ public void testRebuildBrokerResourceWhenBrokerAdded()
         .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
   }
 
+  /**
+   * Verifies that rebuildBrokerResourceFromHelixTags works for a logical table partition when a new broker
+   * is added manually and the ideal state is out of sync (Issue #15751).
+   */
+  @Test
+  public void testRebuildBrokerResourceWhenBrokerAddedForLogicalTable()
+      throws Exception {
+    // Add realtime table so we can create a logical table with both offline and realtime
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(TEST_TABLE_NAME).setNumReplicas(2)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+
+    String logicalTableName = "test_logical_rebuild";
+    addDummySchema(logicalTableName);
+    List<String> physicalTableNamesWithType =
+        List.of(TableNameBuilder.OFFLINE.tableNameWithType(TEST_TABLE_NAME),
+            TableNameBuilder.REALTIME.tableNameWithType(TEST_TABLE_NAME));
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType,
+            TagNameUtils.DEFAULT_TENANT_NAME);
+    addLogicalTableConfig(logicalTableConfig);
+
+    HelixAdmin helixAdmin = _helixManager.getClusterManagmentTool();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getPartitionSet().contains(logicalTableName));
+
+    // Add a new broker manually so the logical table partition is missing it
+    final String newBrokerId = "Broker_localhost_3";
+    InstanceConfig instanceConfig = new InstanceConfig(newBrokerId);
+    instanceConfig.setInstanceEnabled(true);
+    instanceConfig.setHostName("Broker_localhost");
+    instanceConfig.setPort("3");
+    helixAdmin.addInstance(getHelixClusterName(), instanceConfig);
+    helixAdmin.addInstanceTag(getHelixClusterName(), instanceConfig.getInstanceName(),
+        TagNameUtils.getBrokerTagForTenant(TagNameUtils.DEFAULT_TENANT_NAME));
+
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertFalse(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    _helixResourceManager.rebuildBrokerResourceFromHelixTags(logicalTableName);
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    // Cleanup
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+    _helixResourceManager.deleteRealtimeTable(TEST_TABLE_NAME);
+    // Remove the manually added broker instance so that subsequent tests see a clean cluster state
+    helixAdmin.removeInstanceTag(getHelixClusterName(), instanceConfig.getInstanceName(),
+        TagNameUtils.getBrokerTagForTenant(TagNameUtils.DEFAULT_TENANT_NAME));
+    instanceConfig.setInstanceEnabled(false);
+    helixAdmin.dropInstance(getHelixClusterName(), instanceConfig);

Review Comment:
   Similar to the integration test, cleanup is not guaranteed if the test fails mid-way. Since this test mutates shared Helix cluster state, it should use `try/finally` to always remove the instance tag / drop the instance / delete configs to avoid cross-test interference.
   



##########
pinot-controller/src/test/java/org/apache/pinot/controller/validation/ValidationManagerStatelessTest.java:
##########
@@ -100,6 +104,62 @@ public void testRebuildBrokerResourceWhenBrokerAdded()
         .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
   }
 
+  /**
+   * Verifies that rebuildBrokerResourceFromHelixTags works for a logical table partition when a new broker
+   * is added manually and the ideal state is out of sync (Issue #15751).
+   */
+  @Test
+  public void testRebuildBrokerResourceWhenBrokerAddedForLogicalTable()
+      throws Exception {
+    // Add realtime table so we can create a logical table with both offline and realtime
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(TEST_TABLE_NAME).setNumReplicas(2)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+
+    String logicalTableName = "test_logical_rebuild";
+    addDummySchema(logicalTableName);
+    List<String> physicalTableNamesWithType =
+        List.of(TableNameBuilder.OFFLINE.tableNameWithType(TEST_TABLE_NAME),
+            TableNameBuilder.REALTIME.tableNameWithType(TEST_TABLE_NAME));
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType,
+            TagNameUtils.DEFAULT_TENANT_NAME);
+    addLogicalTableConfig(logicalTableConfig);
+
+    HelixAdmin helixAdmin = _helixManager.getClusterManagmentTool();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getPartitionSet().contains(logicalTableName));
+
+    // Add a new broker manually so the logical table partition is missing it
+    final String newBrokerId = "Broker_localhost_3";
+    InstanceConfig instanceConfig = new InstanceConfig(newBrokerId);
+    instanceConfig.setInstanceEnabled(true);
+    instanceConfig.setHostName("Broker_localhost");
+    instanceConfig.setPort("3");
+    helixAdmin.addInstance(getHelixClusterName(), instanceConfig);
+    helixAdmin.addInstanceTag(getHelixClusterName(), instanceConfig.getInstanceName(),
+        TagNameUtils.getBrokerTagForTenant(TagNameUtils.DEFAULT_TENANT_NAME));
+
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertFalse(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    _helixResourceManager.rebuildBrokerResourceFromHelixTags(logicalTableName);
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    // Cleanup
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+    _helixResourceManager.deleteRealtimeTable(TEST_TABLE_NAME);
+    // Remove the manually added broker instance so that subsequent tests see a clean cluster state
+    helixAdmin.removeInstanceTag(getHelixClusterName(), instanceConfig.getInstanceName(),
+        TagNameUtils.getBrokerTagForTenant(TagNameUtils.DEFAULT_TENANT_NAME));
+    instanceConfig.setInstanceEnabled(false);
+    helixAdmin.dropInstance(getHelixClusterName(), instanceConfig);

Review Comment:
   Similar to the integration test, cleanup is not guaranteed if the test fails mid-way. Since this test mutates shared Helix cluster state, it should use `try/finally` to always remove the instance tag / drop the instance / delete configs to avoid cross-test interference.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2938178775


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -55,15 +69,21 @@ protected Context preprocess(Properties periodicTaskProperties) {
   @Override
   protected void processTable(String tableNameWithType, Context context) {
     TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
+    if (tableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+      _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
       return;
     }
-
-    // Rebuild broker resource
-    Set<String> brokerInstances = _pinotHelixResourceManager
-        .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
-    _pinotHelixResourceManager.rebuildBrokerResource(tableNameWithType, brokerInstances);
+    LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+    if (logicalTableConfig != null) {
+      Set<String> brokerInstances = _pinotHelixResourceManager

Review Comment:
   Can we confirm this pls. What will be the behaviour if physical table's broker is different ? 
   
   If it will be unused, then we should add a validation / prevent physical tables broker to be different. Otherwise it will lead to confusion during debugging. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2946133467


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +48,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());

Review Comment:
   thanks. I was wondering how / where a leader is created for logical table. Seems like isLeaderForTable() is a dynamic function which doesn't really create a leader for a table. So should work for logical tables too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2906718144


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -54,15 +69,26 @@ protected Context preprocess(Properties periodicTaskProperties) {
 
   @Override
   protected void processTable(String tableNameWithType, Context context) {
-    TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
-    if (tableConfig == null) {
-      LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation", tableNameWithType);
-      return;
+    Set<String> brokerInstances;
+    if (TableNameBuilder.isTableResource(tableNameWithType)) {
+      TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
+      if (tableConfig == null) {
+        LOGGER.warn("Failed to find table config for table: {}, skipping broker resource validation",
+            tableNameWithType);
+        return;
+      }
+      brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, tableConfig.getTenantConfig().getBroker());
+    } else {
+      LogicalTableConfig logicalTableConfig = _pinotHelixResourceManager.getLogicalTableConfig(tableNameWithType);
+      if (logicalTableConfig == null) {
+        LOGGER.warn("Failed to find logical table config for: {}, skipping broker resource validation",
+            tableNameWithType);
+        return;
+      }
+      brokerInstances = _pinotHelixResourceManager
+          .getAllInstancesForBrokerTenant(context._instanceConfigs, logicalTableConfig.getBrokerTenant());

Review Comment:
   Using `TableNameBuilder.isTableResource(...)` to decide physical vs logical can misclassify a logical table whose name ends with `_OFFLINE`/`_REALTIME`, causing it to be treated as a physical table and skipped (no table config found). Prefer determining the type by config presence (e.g., attempt table config lookup first; if absent, fall back to logical table config) rather than relying on name patterns.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -748,6 +748,25 @@ public List<String> getAllTables() {
     return getAllResources().stream().filter(TableNameBuilder::isTableResource).collect(Collectors.toList());
   }
 
+  /**
+   * Returns partition names in broker resource ideal state that are logical tables (not physical _OFFLINE/_REALTIME)
+   * and have a logical table config. Used by broker resource validation to repair logical table broker assignments.
+   */
+  public List<String> getBrokerResourceLogicalTablePartitions() {
+    IdealState brokerIdealState = HelixHelper.getBrokerIdealStates(_helixAdmin, _helixClusterName);
+    if (brokerIdealState == null) {
+      return Collections.emptyList();
+    }
+    List<String> logicalPartitions = new ArrayList<>();
+    for (String partition : brokerIdealState.getPartitionSet()) {
+      if (!TableNameBuilder.isTableResource(partition)
+          && ZKMetadataProvider.getLogicalTableConfig(_propertyStore, partition) != null) {
+        logicalPartitions.add(partition);
+      }
+    }

Review Comment:
   `getBrokerResourceLogicalTablePartitions()` currently excludes partitions that *look* like physical tables by name (`isTableResource`). This can drop valid logical tables whose names match the physical naming convention (e.g., end with `_OFFLINE`). Consider identifying logical partitions by checking configs instead (e.g., if `getTableConfig(...)` is null and `getLogicalTableConfig(...)` is non-null, treat as logical), rather than relying on name heuristics.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/BrokerResourceValidationManager.java:
##########
@@ -45,6 +49,17 @@ public BrokerResourceValidationManager(ControllerConf config, PinotHelixResource
         controllerMetrics);
   }
 
+  @Override
+  protected List<String> getTablesToProcess(Properties periodicTaskProperties) {
+    List<String> tables = super.getTablesToProcess(periodicTaskProperties);
+    if (periodicTaskProperties.get(PeriodicTask.PROPERTY_KEY_TABLE_NAME) != null) {
+      return tables;
+    }
+    List<String> combined = new ArrayList<>(tables);
+    combined.addAll(_pinotHelixResourceManager.getBrokerResourceLogicalTablePartitions());
+    return combined;
+  }
+
   @Override

Review Comment:
   This adds logical-table partitions to the work list, but `ControllerPeriodicTask` will still filter via `shouldProcessTable()` (default: `_leadControllerManager.isLeaderForTable(tableNameWithType)`). If table leadership is only tracked for physical tables, logical partitions may be filtered out and never processed. Consider overriding `shouldProcessTable` in `BrokerResourceValidationManager` to ensure logical partitions are processed (e.g., return `true` for logical partitions, while retaining the leader check for physical tables).
   ```suggestion
     @Override
     protected boolean shouldProcessTable(String tableNameWithType) {
       // Always process logical-table partitions; retain leader-based filtering for physical tables.
       if (!TableNameBuilder.isTableResource(tableNameWithType)) {
         return true;
       }
       return super.shouldProcessTable(tableNameWithType);
     }
   
     @Override
   ```



##########
pinot-controller/src/test/java/org/apache/pinot/controller/validation/BrokerResourceValidationManagerTest.java:
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.validation;
+
+import java.lang.reflect.Method;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.LeadControllerManager;
+import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.core.periodictask.PeriodicTask;
+import org.apache.pinot.spi.metrics.PinotMetricUtils;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Unit tests for {@link BrokerResourceValidationManager}, including that getTablesToProcess
+ * returns both physical table names and logical table partition names (Issue #15751).
+ */
+public class BrokerResourceValidationManagerTest {
+
+  private static final String PHYSICAL_TABLE = "myTable_OFFLINE";
+  private static final String LOGICAL_TABLE_PARTITION = "my_logical_table";
+
+  private PinotHelixResourceManager _resourceManager;
+  private BrokerResourceValidationManager _validationManager;
+
+  @BeforeMethod
+  public void setUp() {
+    _resourceManager = mock(PinotHelixResourceManager.class);
+    when(_resourceManager.getAllTables()).thenReturn(List.of(PHYSICAL_TABLE));
+    when(_resourceManager.getBrokerResourceLogicalTablePartitions()).thenReturn(List.of(LOGICAL_TABLE_PARTITION));
+
+    ControllerConf config = new ControllerConf();
+    LeadControllerManager leadControllerManager = mock(LeadControllerManager.class);
+    ControllerMetrics controllerMetrics = new ControllerMetrics(PinotMetricUtils.getPinotMetricsRegistry());
+    _validationManager = new BrokerResourceValidationManager(config, _resourceManager, leadControllerManager,
+        controllerMetrics);
+  }
+
+  /**
+   * Verifies that getTablesToProcess returns both physical tables (from getAllTables) and
+   * logical table partitions (from getBrokerResourceLogicalTablePartitions) so that the
+   * periodic task validates and repairs broker resource for logical tables too.
+   */
+  @Test
+  public void testGetTablesToProcessIncludesLogicalTablePartitions()
+      throws Exception {
+    Method getTablesToProcess =
+        BrokerResourceValidationManager.class.getDeclaredMethod("getTablesToProcess", Properties.class);
+    getTablesToProcess.setAccessible(true);
+
+    @SuppressWarnings("unchecked")
+    List<String> tables = (List<String>) getTablesToProcess.invoke(_validationManager, new Properties());

Review Comment:
   The test uses reflection to invoke `getTablesToProcess`, but the method is `protected` and the test is in the same package (`org.apache.pinot.controller.validation`), so it can be called directly without reflection. Calling it directly will make the test less brittle (e.g., no dependence on method name strings or reflective access).



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1189,42 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      assert idealState != null;
+      for (String partitionName : idealState.getPartitionSet()) {
+        String brokerTag = resolveBrokerTagForBrokerResourcePartition(partitionName);
+        if (brokerTag == null) {
+          continue;
+        }
+        if (brokerTag.equals(brokerTenantTag)) {
+          idealState.setPartitionState(partitionName, instanceName, BrokerResourceStateModel.ONLINE);
+        }
       }
+      return idealState;
+    }, DEFAULT_RETRY_POLICY);
+  }
+
+  /**
+   * Resolves the broker tag for a partition in the broker resource. Tries physical table config first,
+   * then logical table config, so both are handled regardless of partition naming (e.g. a logical table
+   * name ending with _OFFLINE would otherwise be misclassified as physical and skipped).
+   *
+   * @param partitionName partition name in broker ideal state (physical table name with type or logical table name)
+   * @return broker tag for the partition, or null if the partition cannot be resolved (unknown or missing config)
+   */
+  private String resolveBrokerTagForBrokerResourcePartition(String partitionName) {
+    TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, partitionName);
+    if (tableConfig != null) {
+      return TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
+    }
+    LogicalTableConfig logicalTableConfig = ZKMetadataProvider.getLogicalTableConfig(_propertyStore, partitionName);
+    if (logicalTableConfig != null) {
+      return TagNameUtils.getBrokerTagForTenant(logicalTableConfig.getBrokerTenant());
     }
-    _helixAdmin.setResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE, tableIdealState);
+    LOGGER.warn("Skipping partition {} in broker resource: no table config or logical table config found",

Review Comment:
   Logging a WARN for every unresolved broker-resource partition during broker add can create noisy logs (especially if partitions transiently exist without configs during table drops/updates). Consider lowering this to DEBUG, or rate-limiting/aggregating (e.g., log once per call with a count), to avoid spamming controller logs.
   ```suggestion
       LOGGER.debug("Skipping partition {} in broker resource: no table config or logical table config found",
   ```



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1102,21 +1121,41 @@ private PinotResourceManagerResponse scaleUpBroker(Tenant tenant, String brokerT
 
   public PinotResourceManagerResponse rebuildBrokerResourceFromHelixTags(String tableNameWithType)
       throws Exception {
-    TableConfig tableConfig;
-    try {
-      tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-    } catch (Exception e) {
-      LOGGER.warn("Caught exception while getting table config for table {}", tableNameWithType, e);
-      throw new InvalidTableConfigException(
-          "Failed to fetch broker tag for table " + tableNameWithType + " due to exception: " + e.getMessage());
-    }
-    if (tableConfig == null) {
-      LOGGER.warn("Table {} does not exist", tableNameWithType);
-      throw new InvalidConfigException(
-          "Invalid table configuration for table " + tableNameWithType + ". Table does not exist");
+    Set<String> brokerInstances;
+    if (TableNameBuilder.isTableResource(tableNameWithType)) {
+      TableConfig tableConfig;
+      try {
+        tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+      } catch (Exception e) {
+        LOGGER.warn("Caught exception while getting table config for table {}", tableNameWithType, e);
+        throw new InvalidTableConfigException(
+            "Failed to fetch broker tag for table " + tableNameWithType + " due to exception: " + e.getMessage());
+      }
+      if (tableConfig == null) {
+        LOGGER.warn("Table {} does not exist", tableNameWithType);
+        throw new InvalidConfigException(
+            "Invalid table configuration for table " + tableNameWithType + ". Table does not exist");
+      }
+      brokerInstances = getAllInstancesForBrokerTenant(tableConfig.getTenantConfig().getBroker());
+    } else {
+      // Logical table
+      LogicalTableConfig logicalTableConfig;

Review Comment:
   `rebuildBrokerResourceFromHelixTags` uses `isTableResource(tableNameWithType)` to branch. This has the same misclassification problem as above: a logical table named with a physical suffix will go down the physical path and throw `InvalidConfigException` even though a logical table config exists. Consider branching by config existence (try physical `getTableConfig`; if null then try `getLogicalTableConfig`) rather than by name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "abhishekbafna (via GitHub)" <gi...@apache.org>.
abhishekbafna commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2929552967


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -748,6 +748,29 @@ public List<String> getAllTables() {
     return getAllResources().stream().filter(TableNameBuilder::isTableResource).collect(Collectors.toList());
   }
 
+  /**
+   * Returns partition names in broker resource ideal state that are logical tables (have logical table config
+   * and no physical table config). Used by broker resource validation to repair logical table broker assignments.
+   * Identifies by config presence rather than name to avoid misclassifying logical tables named with
+   * _OFFLINE/_REALTIME.
+   */
+  public List<String> getBrokerResourceLogicalTablePartitions() {

Review Comment:
   Yeah. we can do that. Earlier I created an table with `_offline` suffix and it was created successfully. Now I tried with `_OFFLINE` and it failed. So I think, we can use the simpler approach. This is not required. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enhance BrokerResourceValidationManager to support logical table partitions in resource validation [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17842:
URL: https://github.com/apache/pinot/pull/17842#discussion_r2930675768


##########
pinot-controller/src/test/java/org/apache/pinot/controller/validation/BrokerResourceValidationManagerTest.java:
##########
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.validation;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.LeadControllerManager;
+import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.core.periodictask.PeriodicTask;
+import org.apache.pinot.spi.metrics.PinotMetricUtils;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Unit tests for {@link BrokerResourceValidationManager}, including that getTablesToProcess
+ * returns both physical table names and logical table partition names (Issue #15751).
+ */
+public class BrokerResourceValidationManagerTest {
+
+  private static final String PHYSICAL_TABLE = "myTable_OFFLINE";
+  private static final String LOGICAL_TABLE_PARTITION = "my_logical_table";
+
+  private PinotHelixResourceManager _resourceManager;
+  private BrokerResourceValidationManager _validationManager;
+
+  @BeforeMethod
+  public void setUp() {
+    _resourceManager = mock(PinotHelixResourceManager.class);
+    when(_resourceManager.getAllTables()).thenReturn(List.of(PHYSICAL_TABLE));
+    when(_resourceManager.getBrokerResourceLogicalTables()).thenReturn(List.of(LOGICAL_TABLE_PARTITION));
+
+    ControllerConf config = new ControllerConf();
+    LeadControllerManager leadControllerManager = mock(LeadControllerManager.class);
+    ControllerMetrics controllerMetrics = new ControllerMetrics(PinotMetricUtils.getPinotMetricsRegistry());
+    _validationManager = new BrokerResourceValidationManager(config, _resourceManager, leadControllerManager,
+        controllerMetrics);
+  }
+
+  /**
+   * Verifies that getTablesToProcess returns both physical tables (from getAllTables) and
+   * logical table partitions (from getBrokerResourceLogicalTablePartitions) so that the

Review Comment:
   The test Javadoc mentions `getBrokerResourceLogicalTablePartitions`, but the production API used/mocked here is `getBrokerResourceLogicalTables()`. Updating the comment to match the actual method name will avoid confusion during future maintenance/refactors.
   



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1150,16 +1172,36 @@ public PinotResourceManagerResponse rebuildBrokerResource(String tableNameWithTy
   }
 
   private void addInstanceToBrokerIdealState(String brokerTenantTag, String instanceName) {
-    IdealState tableIdealState = _helixAdmin.getResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE);
-    for (String tableNameWithType : tableIdealState.getPartitionSet()) {
-      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
-      Preconditions.checkNotNull(tableConfig);
-      String brokerTag = TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
-      if (brokerTag.equals(brokerTenantTag)) {
-        tableIdealState.setPartitionState(tableNameWithType, instanceName, BrokerResourceStateModel.ONLINE);
+    // Use atomic read-modify-write so updates (including for logical tables) are persisted and not lost to races.
+    HelixHelper.updateIdealState(getHelixZkManager(), Helix.BROKER_RESOURCE_INSTANCE, idealState -> {
+      Preconditions.checkNotNull(idealState, "Broker ideal state must not be null");
+      for (String partitionName : idealState.getPartitionSet()) {
+        String brokerTag = resolveBrokerTagForTable(partitionName);
+        if (brokerTag.equals(brokerTenantTag)) {
+          idealState.setPartitionState(partitionName, instanceName, BrokerResourceStateModel.ONLINE);
+        }
       }
+      return idealState;
+    }, DEFAULT_RETRY_POLICY);
+  }
+
+  /**
+   * Resolves the broker tag for a table in the broker resource. Tries physical table config first,
+   * then logical table config.
+   *
+   * @param tableName table name in broker ideal state (physical table name with type or logical table name)
+   * @return broker tag for the partition, or null if the partition cannot be resolved (unknown or missing config)
+   */
+  private String resolveBrokerTagForTable(String tableName) {
+    TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableName);
+    if (tableConfig != null) {
+      return TagNameUtils.extractBrokerTag(tableConfig.getTenantConfig());
+    }
+    LogicalTableConfig logicalTableConfig = ZKMetadataProvider.getLogicalTableConfig(_propertyStore, tableName);
+    if (logicalTableConfig != null) {
+      return TagNameUtils.getBrokerTagForTenant(logicalTableConfig.getBrokerTenant());
     }
-    _helixAdmin.setResourceIdealState(_helixClusterName, Helix.BROKER_RESOURCE_INSTANCE, tableIdealState);
+    throw new InvalidTableConfigException("Failed to resolve broker tag for table " + tableName + " because no table config or logical table config found");

Review Comment:
   `resolveBrokerTagForTable` Javadoc says it "@return broker tag ... or null" when the partition cannot be resolved, but the implementation always throws `InvalidTableConfigException` when neither physical nor logical config is found. Please either update the Javadoc to reflect the exception behavior, or change the method to return `null` and have callers handle the unresolved-partition case gracefully.



##########
pinot-controller/src/test/java/org/apache/pinot/controller/validation/ValidationManagerStatelessTest.java:
##########
@@ -100,6 +104,57 @@ public void testRebuildBrokerResourceWhenBrokerAdded()
         .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
   }
 
+  /**
+   * Verifies that rebuildBrokerResourceFromHelixTags works for a logical table partition when a new broker
+   * is added manually and the ideal state is out of sync (Issue #15751).
+   */
+  @Test
+  public void testRebuildBrokerResourceWhenBrokerAddedForLogicalTable()
+      throws Exception {
+    // Add realtime table so we can create a logical table with both offline and realtime
+    TableConfig realtimeTableConfig =
+        new TableConfigBuilder(TableType.REALTIME).setTableName(TEST_TABLE_NAME).setNumReplicas(2)
+            .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap()).build();
+    _helixResourceManager.addTable(realtimeTableConfig);
+
+    String logicalTableName = "test_logical_rebuild";
+    addDummySchema(logicalTableName);
+    List<String> physicalTableNamesWithType =
+        List.of(TableNameBuilder.OFFLINE.tableNameWithType(TEST_TABLE_NAME),
+            TableNameBuilder.REALTIME.tableNameWithType(TEST_TABLE_NAME));
+    LogicalTableConfig logicalTableConfig =
+        ControllerTest.getDummyLogicalTableConfig(logicalTableName, physicalTableNamesWithType,
+            TagNameUtils.DEFAULT_TENANT_NAME);
+    addLogicalTableConfig(logicalTableConfig);
+
+    HelixAdmin helixAdmin = _helixManager.getClusterManagmentTool();
+    IdealState idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getPartitionSet().contains(logicalTableName));
+
+    // Add a new broker manually so the logical table partition is missing it
+    final String newBrokerId = "Broker_localhost_3";
+    InstanceConfig instanceConfig = new InstanceConfig(newBrokerId);
+    instanceConfig.setInstanceEnabled(true);
+    instanceConfig.setHostName("Broker_localhost");
+    instanceConfig.setPort("3");
+    helixAdmin.addInstance(getHelixClusterName(), instanceConfig);
+    helixAdmin.addInstanceTag(getHelixClusterName(), instanceConfig.getInstanceName(),
+        TagNameUtils.getBrokerTagForTenant(TagNameUtils.DEFAULT_TENANT_NAME));
+
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertFalse(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    _helixResourceManager.rebuildBrokerResourceFromHelixTags(logicalTableName);
+    idealState = HelixHelper.getBrokerIdealStates(helixAdmin, getHelixClusterName());
+    Assert.assertTrue(idealState.getInstanceSet(logicalTableName)
+        .equals(_helixResourceManager.getAllInstancesForBrokerTenant(TagNameUtils.DEFAULT_TENANT_NAME)));
+
+    // Cleanup
+    _helixResourceManager.deleteLogicalTableConfig(logicalTableName);
+    _helixResourceManager.deleteRealtimeTable(TEST_TABLE_NAME);

Review Comment:
   This new test adds a broker instance (`Broker_localhost_3`) but does not remove it in the cleanup. Because this test class shares a single Helix cluster across test methods and TestNG does not guarantee test execution order, leaving the extra broker can make other tests flaky (e.g., physical-table broker ideal state can become out-of-sync vs `getAllInstancesForBrokerTenant`). Consider dropping the instance (and/or removing its tag) in the cleanup block so each test leaves the cluster in a known state.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org